Opened 4 weeks ago
Last modified 3 weeks ago
#64103 new defect (bug)
Run WP-CLI test suite in core repo
| Reported by: |
|
Owned by: | |
|---|---|---|---|
| Milestone: | Awaiting Review | Priority: | normal |
| Severity: | normal | Version: | |
| Component: | Build/Test Tools | Keywords: | |
| Focuses: | Cc: |
Description
Occasionally, a WordPress core change (small or big) can cause test failures over at WP-CLI, which is another official WordPress project.
The most recent example is #53323, but that's just one of many.
Whenever this happens, WP-CLI maintainers have to analyze test failures to see its origin and assess whether it's a regression in core that needs to be fixed, or whether WP-CLI tests simply need to be hardened.
To improve the feedback loop, we could add another CI workflow that runs Behat tests for all WP-CLI packages (as in https://github.com/wp-cli/automated-tests) for the current WordPress build.
It could work very similar to the pull-request-comments.yml workflow that runs only after the current commit (either in trunk or a PR) has a built ZIP file. Then, that ZIP file is used to run tests. Running them only for a single PHP version would already help a lot, so I don't think it would need to involve a complex matrix or triggering reusable workflows.
As mentioned, sometimes a test failure means WP-CLI tests simply need some hardening, so they shouldn't necessarily block merging. Perhaps we could find a good way to communicate that. Suggestions welcome.
There might also be some alternatives, like posting WP-CLI failures in the #core Slack channel or a combined testing dashboard that shows test status for core, Gutenberg, and WP-CLI.