Consider adopting workflows for testing build packages

[costdev]

Currently, build package installations and upgrades are manually tested by WordPress.org and release party attendees. However, this testing doesn't cover a very wide range of PHP, MySQL and older WordPress versions, and some of these encounter issues that aren't discovered or reported until much later. We've previously had broken ZIP archives, polyfills used where they can't be used, database errors, and such.

I've been working on ​GitHub Actions workflows to help improve our build package coverage.

More details are in the comment below.

[costdev]

What do the workflows cover?

Installation - 32 tests

• PHP: 7.0 - 8.2
• MySQL: 5.7 and 8.0
• Single site and Multisite - Sub-directory

Upgrade - 620 tests

• WP: The latest minor release of the 4.1 - 6.x branches
• PHP: 7.0 - 8.2
• MySQL: 5.7 and 8.0
• Single site and Multisite - Sub-directory
• Caveat: WordPress 4.6 - 5.2 tests run on PHP 7.0 - 7.4 due to Fatal Errors because of uses of __autoload() and curly braces for array/string offsets.

All currently run on ubuntu-latest.

What could they cover later?

• I'd hope to see these extended to test windows-latest and mac-latest.
• SQLite and other databases.
• Earlier Beta / RC releases in a cycle.
• End-to-End tests for manual installs.

How do they work?

The workflows leverage WP-CLI to run installation and upgrade tests.

To ensure that matrices don't exceed 256 jobs for now, the workflows are split into:

• Installation Tests
• WP 4.x - Single site
• WP 4.x - Multisite - Sub-directory
• WP 5.x - Single site
• WP 5.x - Multisite - Sub-directory
• WP 6.x - Single site
• WP 6.x - Multisite - Sub-directory

Each workflow accepts one input:

• new_version - The version being installed/upgraded to. Any value that works with the --version parameter in WP-CLI will work for this, including 6.3, 6.3-RC3 and nightly.

How long do they take?

As my GitHub plan means I'm limited to 20 concurrent jobs, the workflows take longer than they would on WordPress' plan. I'd expect all 652 jobs to complete within 1-3 minutes if adopted by WordPress.

How could they be used?

For example:

• Workflows could be run on nightly builds, ensuring we catch issues in any of these scenarios within 24 hours of the commit that introduced them.
• We might later look at integrating these in the release party process to catch issues within hours of being introduced.
  • This would require a Make post to discuss how best to maximise the benefits of testing by release party attendees, which could focus only on items addressed during the latest build so that we might catch any issues early. If adopted, we must make sure we don't increase coverage while decreasing community engagement.

[costdev]

Pinging folks I spoke with regarding this in DM: @azaozz @desrosj @hellofromTonya @peterwilsoncc @afragen @ironprogrammer

[azaozz]

Thinking this is a great idea. "Live testing" at release parties is very nice but "test coverage" may be limited. Having the assurance the packages would work on all possible combinations of WordPress + PHP + MySQL/MariaDB would be very good.

Frankly I'm tempted to add this to the 6.3 milestone as it is for the build tools. Would have been nice to see it in action on release day. But maybe better to first try it on few beta and RC releases :)

[costdev]

I've previously run the workflows on 6.3-RC2, 6.3-RC3 and a nightly package.

In terms of a patch, the workflows download and install a copy of WordPress directly into the repo's root during each run. For simplicity, I'd suggest that the repo remains a standalone one rather than integrating it directly into the SVN repository.

I'd think that a transfer of ownership from my GitHub account to the WordPress organisation's account would do here, then if we're going to run them on nightly packages, the workflows would just need to be called with inputs.new_version being set to nightly.

[swissspidy]

Duplicate of #43395?

[costdev]

Rather than a duplicate, I would consider this ticket a partial implementation of @dd32's list here, with #43395 being an umbrella ticket.

Where #43395 mentions E2E testing, permissions, different filesystem abstractions and the like, this ticket could be considered a partial fulfilment of the overall list. Once this ticket is handled, we could then close it and move onto the remaining items on the list, rather than trying to accomplish everything in one ticket.

[ironprogrammer]

+1 on this initiative! Regular validation of install/upgrade is a great idea 👍🏻

Impact on Release Parties

@costdev makes an important point above regarding the need to continue to encourage release party engagement. Providing a list of items to test after the build is available would present additional guidance for testers to exercise the build beyond the installer process. (Individual/novel install testing would still be encouraged, but need not be the focus of post-build testing efforts.)

The community coordination required to come together for a release party is significant; this proposal can help ensure timely builds and would allow testers to focus on items pertinent to the build.

[desrosj]

Related to adding more broad testing for MySQL 8.0: #30462.

[desrosj]

Thanks for the great work here so far, @costdev! I'd love to get this merged and give it a try. Some thoughts:

• I'm not sure it should be removed, but we don't yet test MySQL 8 regularly in general. I've been working on #30462 to add this. I don't think that 8.0 needs to be removed from this in order to be merged, but there may be some strange incompatibilities that pop up (old versions were not fully compatible).
• I would very much prefer that we get a bit creative with matrix strategies instead of splitting it into several workflows for the upgrade testing. I think we can absolutely stay below the maximum number of jobs with certain groupings. I have some ideas around how we could do this.
• Let's move this repository to the WordPress organization in order to iterate. I've reached out to @costdev in Slack to facilitate this. It will be easier to suggest and test changes under the organization's enterprise account (which has 500 concurrent jobs).

[desrosj]

I've been working with @costdev quite a bit over on the ​WordPress/wordpress-upgrade-tester repository where we moved his POC.

I think this is in a great state. It currently:

• Tests updating all branches of WordPress that receive security updates to the version being tested.
• Tests updating with all currently supported PHP versions (where possible).
• Tests updating with all currently supported MySQL versions.

Let's get this in to see how it goes throughout the 6.4 beta/RC period leading up to final release.

For now, it will only run if manually triggered. I've checked that this is possible (and it is), but in the future when the WP.org server creates the release package for a specific version, it should make a request to GitHub with the version number being packaged to trigger this workflow automatically.

Here's a list of remaining tasks that are not blockers, but need to be explored further (not necessarily in this ticket):

• Perform some actual test assertions. Currently, the workflow only checks that WP-CLI does not encounter a failure when attempting to update. It would be good to also have some tests that perform some basic tasks to confirm things look good and actually work.
• When testing minor releases for older versions, we should only test up to that version. For example, if 5.0.10 is being released, we shouldn't test updating 5.1.x to 5.0.10. Only WP versions lower than the current version should be tested.
• When testing minor releases for older versions, we should test the versions of PHP that were supported for that version. For example, if we're releasing a ​4.9 minor version, we should test PHP 5.2-5.6, and 7.0-7.2. This also applies for database versions.
• Ideally, the workflow should also perform MariaDB testing. After [56439], MariaDB is tested by default for all commits.

One last call out is that the workflow takes a bit of time to report success. However, the jobs in the workflow itself run surprisingly fast. The time that it actually takes to run all of the associated jobs can be calculated by monitoring the GitHub Action Runner administration page when the workflow starts. The jobs are all completed within 5-8 minutes, but the delay seems to be with the product UI receiving and displaying job details from the 650+ runners, and takes double to triple that amount of time.

During the release process, this could be a pain, but that usually takes around that same amount of time. But this workflow is not meant to replace manual testing, but rather serve as an additional sanity check to catch some additional potential issues. I've reached out to GitHub about this, and we can continue to explore ways to speed this up, but I think it will need to be fixed on their end.

Trac ticket: https://core.trac.wordpress.org/ticket/58977

[desrosj]

In 56661:

Build/Test Tools: Introduce install and upgrade testing workflows.

This introduces GitHub Action workflows for testing the installation and upgrade process to ensure there are no unexpected issues. The goal of these workflows is to replicate some of the manual testing that happens during the release process. This does not replace that testing which is a fantastic, low barrier way for someone newer to contributing to get involved. Instead, it should serve as an additional sanity check to ensure no combinations are missed.

The installation workflow tests installing the latest or specified version of WordPress on all PHP and MySQL version combinations for both single and multisite.

The upgrade testing will test upgrading from each branch that still receives security updates to the latest or specified version on all PHP and MySQL version combinations for both single and multisite.

Both workflows confirm that no errors are encountered when using WP-CLI to install or upgrade WordPress. No test assertions of any type are currently performed, but adding some will be explored.

Props costdev, azaozz, ironprogrammer, desrosj.
See #58977.

Merged in https://core.trac.wordpress.org/changeset/56661. Thanks @costdev!