Make WordPress Core

Opened 4 weeks ago

Last modified 3 weeks ago

#64103 new defect (bug)

Run WP-CLI test suite in core repo

Reported by: swissspidy's profile swissspidy Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

Occasionally, a WordPress core change (small or big) can cause test failures over at WP-CLI, which is another official WordPress project.

The most recent example is #53323, but that's just one of many.

Whenever this happens, WP-CLI maintainers have to analyze test failures to see its origin and assess whether it's a regression in core that needs to be fixed, or whether WP-CLI tests simply need to be hardened.

To improve the feedback loop, we could add another CI workflow that runs Behat tests for all WP-CLI packages (as in https://github.com/wp-cli/automated-tests) for the current WordPress build.

It could work very similar to the pull-request-comments.yml workflow that runs only after the current commit (either in trunk or a PR) has a built ZIP file. Then, that ZIP file is used to run tests. Running them only for a single PHP version would already help a lot, so I don't think it would need to involve a complex matrix or triggering reusable workflows.

As mentioned, sometimes a test failure means WP-CLI tests simply need some hardening, so they shouldn't necessarily block merging. Perhaps we could find a good way to communicate that. Suggestions welcome.

There might also be some alternatives, like posting WP-CLI failures in the #core Slack channel or a combined testing dashboard that shows test status for core, Gutenberg, and WP-CLI.

Change History (1)

This ticket was mentioned in Slack in #core by amykamala. View the logs.


3 weeks ago

Note: See TracTickets for help on using tickets.