Make WordPress Core

Opened 12 months ago

Last modified 12 months ago

#64103 new defect (bug)

Run WP-CLI test suite in core repo

Reported by: swissspidy Owned by:
Priority: normal Milestone: Awaiting Review
Component: Build/Test Tools Version:
Severity: normal Keywords:
Cc: Focuses:

Description

Occasionally, a WordPress core change (small or big) can cause test failures over at WP-CLI, which is another official WordPress project.

The most recent example is #53323, but that's just one of many.

Whenever this happens, WP-CLI maintainers have to analyze test failures to see its origin and assess whether it's a regression in core that needs to be fixed, or whether WP-CLI tests simply need to be hardened.

To improve the feedback loop, we could add another CI workflow that runs Behat tests for all WP-CLI packages (as in ​https://github.com/wp-cli/automated-tests) for the current WordPress build.

It could work very similar to the pull-request-comments.yml workflow that runs only after the current commit (either in trunk or a PR) has a built ZIP file. Then, that ZIP file is used to run tests. Running them only for a single PHP version would already help a lot, so I don't think it would need to involve a complex matrix or triggering reusable workflows.

As mentioned, sometimes a test failure means WP-CLI tests simply need some hardening, so they shouldn't necessarily block merging. Perhaps we could find a good way to communicate that. Suggestions welcome.

There might also be some alternatives, like posting WP-CLI failures in the #core Slack channel or a combined testing dashboard that shows test status for core, Gutenberg, and WP-CLI.

Change History (1)

This ticket was mentioned in ​Slack in #core by amykamala. ​View the logs.


12 months ago

Note: See TracTickets for help on using tickets.