Consider adopting workflows for testing build packages

[costdev]

Currently, build package installations and upgrades are manually tested by WordPress.org and release party attendees. However, this testing doesn't cover a very wide range of PHP, MySQL and older WordPress versions, and some of these encounter issues that aren't discovered or reported until much later. We've previously had broken ZIP archives, polyfills used where they can't be used, database errors, and such.

I've been working on ​GitHub Actions workflows to help improve our build package coverage.

More details are in the comment below.

[costdev]

What do the workflows cover?

Installation - 32 tests

• PHP: 7.0 - 8.2
• MySQL: 5.7 and 8.0
• Single site and Multisite - Sub-directory

Upgrade - 620 tests

• WP: The latest minor release of the 4.1 - 6.x branches
• PHP: 7.0 - 8.2
• MySQL: 5.7 and 8.0
• Single site and Multisite - Sub-directory
• Caveat: WordPress 4.6 - 5.2 tests run on PHP 7.0 - 7.4 due to Fatal Errors because of uses of __autoload() and curly braces for array/string offsets.

All currently run on ubuntu-latest.

What could they cover later?

• I'd hope to see these extended to test windows-latest and mac-latest.
• SQLite and other databases.
• Earlier Beta / RC releases in a cycle.
• End-to-End tests for manual installs.

How do they work?

The workflows leverage WP-CLI to run installation and upgrade tests.

To ensure that matrices don't exceed 256 jobs for now, the workflows are split into:

• Installation Tests
• WP 4.x - Single site
• WP 4.x - Multisite - Sub-directory
• WP 5.x - Single site
• WP 5.x - Multisite - Sub-directory
• WP 6.x - Single site
• WP 6.x - Multisite - Sub-directory

Each workflow accepts one input:

• new_version - The version being installed/upgraded to. Any value that works with the --version parameter in WP-CLI will work for this, including 6.3, 6.3-RC3 and nightly.

How long do they take?

As my GitHub plan means I'm limited to 20 concurrent jobs, the workflows take longer than they would on WordPress' plan. I'd expect all 652 jobs to complete within 1-3 minutes if adopted by WordPress.

How could they be used?

For example:

• Workflows could be run on nightly builds, ensuring we catch issues in any of these scenarios within 24 hours of the commit that introduced them.
• We might later look at integrating these in the release party process to catch issues within hours of being introduced.
  • This would require a Make post to discuss how best to maximise the benefits of testing by release party attendees, which could focus only on items addressed during the latest build so that we might catch any issues early. If adopted, we must make sure we don't increase coverage while decreasing community engagement.

[costdev]

Pinging folks I spoke with regarding this in DM: @azaozz @desrosj @hellofromTonya @peterwilsoncc @afragen @ironprogrammer

[azaozz]

Thinking this is a great idea. "Live testing" at release parties is very nice but "test coverage" may be limited. Having the assurance the packages would work on all possible combinations of WordPress + PHP + MySQL/MariaDB would be very good.

Frankly I'm tempted to add this to the 6.3 milestone as it is for the build tools. Would have been nice to see it in action on release day. But maybe better to first try it on few beta and RC releases :)

[costdev]

I've previously run the workflows on 6.3-RC2, 6.3-RC3 and a nightly package.

In terms of a patch, the workflows download and install a copy of WordPress directly into the repo's root during each run. For simplicity, I'd suggest that the repo remains a standalone one rather than integrating it directly into the SVN repository.

I'd think that a transfer of ownership from my GitHub account to the WordPress organisation's repository, then if we're going to run them on nightly packages, the workflows would just need to be called with inputs.new_version being set to nightly.

[swissspidy]

Duplicate of #43395?

[costdev]

Rather than a duplicate, I would consider this ticket a partial implementation of @dd32's list here, with #43395 being an umbrella ticket.

Where #43395 mentions E2E testing, permissions, different filesystem abstractions and the like, this ticket could be considered a partial fulfilment of the overall list. Once this ticket is handled, we could then close it and move onto the remaining items on the list, rather than trying to accomplish everything in one ticket.

[ironprogrammer]

+1 on this initiative! Regular validation of install/upgrade is a great idea 👍🏻

Impact on Release Parties

@costdev makes an important point above regarding the need to continue to encourage release party engagement. Providing a list of items to test after the build is available would present additional guidance for testers to exercise the build beyond the installer process. (Individual/novel install testing would still be encouraged, but need not be the focus of post-build testing efforts.)

The community coordination required to come together for a release party is significant; this proposal can help ensure timely builds and would allow testers to focus on items pertinent to the build.

[desrosj]

Related to adding more broad testing for MySQL 8.0: #30462.

[desrosj]

Thanks for the great work here so far, @costdev! I'd love to get this merged and give it a try. Some thoughts:

• I'm not sure it should be removed, but we don't yet test MySQL 8 regularly in general. I've been working on #30462 to add this. I don't think that 8.0 needs to be removed from this in order to be merged, but there may be some strange incompatibilities that pop up (old versions were not fully compatible).
• I would very much prefer that we get a bit creative with matrix strategies instead of splitting it into several workflows for the upgrade testing. I think we can absolutely stay below the maximum number of jobs with certain groupings. I have some ideas around how we could do this.
• Let's move this repository to the WordPress organization in order to iterate. I've reached out to @costdev in Slack to facilitate this. It will be easier to suggest and test changes under the organization's enterprise account (which has 500 concurrent jobs).

[desrosj]

I've been working with @costdev quite a bit over on the ​WordPress/wordpress-upgrade-tester repository where we moved his POC.

I think this is in a great state. It currently:

• Tests updating all branches of WordPress that receive security updates to the version being tested.
• Tests updating with all currently supported PHP versions (where possible).
• Tests updating with all currently supported MySQL versions.

Let's get this in to see how it goes throughout the 6.4 beta/RC period leading up to final release.

For now, it will only run if manually triggered. I've checked that this is possible (and it is), but in the future when the WP.org server creates the release package for a specific version, it should make a request to GitHub with the version number being packaged to trigger this workflow automatically.

Here's a list of remaining tasks that are not blockers, but need to be explored further (not necessarily in this ticket):

• Perform some actual test assertions. Currently, the workflow only checks that WP-CLI does not encounter a failure when attempting to update. It would be good to also have some tests that perform some basic tasks to confirm things look good and actually work.
• When testing minor releases for older versions, we should only test up to that version. For example, if 5.0.10 is being released, we shouldn't test updating 5.1.x to 5.0.10. Only WP versions lower than the current version should be tested.
• When testing minor releases for older versions, we should test the versions of PHP that were supported for that version. For example, if we're releasing a ​4.9 minor version, we should test PHP 5.2-5.6, and 7.0-7.2. This also applies for database versions.
• Ideally, the workflow should also perform MariaDB testing. After [56439], MariaDB is tested by default for all commits.

One last call out is that the workflow takes a bit of time to report success. However, the jobs in the workflow itself run surprisingly fast. The time that it actually takes to run all of the associated jobs can be calculated by monitoring the GitHub Action Runner administration page when the workflow starts. The jobs are all completed within 5-8 minutes, but the delay seems to be with the product UI receiving and displaying job details from the 650+ runners, and takes double to triple that amount of time.

During the release process, this could be a pain, but that usually takes around that same amount of time. But this workflow is not meant to replace manual testing, but rather serve as an additional sanity check to catch some additional potential issues. I've reached out to GitHub about this, and we can continue to explore ways to speed this up, but I think it will need to be fixed on their end.

Trac ticket: https://core.trac.wordpress.org/ticket/58977

[desrosj]

In 56661:

Build/Test Tools: Introduce install and upgrade testing workflows.

This introduces GitHub Action workflows for testing the installation and upgrade process to ensure there are no unexpected issues. The goal of these workflows is to replicate some of the manual testing that happens during the release process. This does not replace that testing which is a fantastic, low barrier way for someone newer to contributing to get involved. Instead, it should serve as an additional sanity check to ensure no combinations are missed.

The installation workflow tests installing the latest or specified version of WordPress on all PHP and MySQL version combinations for both single and multisite.

The upgrade testing will test upgrading from each branch that still receives security updates to the latest or specified version on all PHP and MySQL version combinations for both single and multisite.

Both workflows confirm that no errors are encountered when using WP-CLI to install or upgrade WordPress. No test assertions of any type are currently performed, but adding some will be explored.

Props costdev, azaozz, ironprogrammer, desrosj.
See #58977.

Merged in https://core.trac.wordpress.org/changeset/56661. Thanks @costdev!

[desrosj]

In 56823:

Build/Test Tools: Correct variable typo.

This was causing the version of WordPress being tested to be listed as latest even when that is not the case.

Props davidbaumwald.
See #58977.

This makes the install testing workflow a bit more flexible in order to allow usage for any branch of WordPress currently receiving security updates.

Because you can't manually start workflows for a PR (that I'm aware of), I've run the workflow manually within my fork. Here are a few test workflows:

• ​WordPress 4.1.20
• ​WordPress 5.0.18
• ​WordPress 5.2.1
• ​WordPress 6.0.6

Remaining items:

• ​https://github.com/desrosj/wordpress-develop/actions/runs/6512389627/job/17689928003#step:3:15 The correct version of WP CLI is not [installing on PHP 5.3], but is for 5.4-5.5.
• [ ] The MySQL versions needs to be verified and confirmed.
• [ ] It's unlikely because of how WP versions and ships security updates, but Is there a more standard way to list version support?

Trac ticket: https://core.trac.wordpress.org/ticket/58977

Some updates here:

• I've added an exclusion for testing PHP 5.3 for now. While it would be nice to have this testing running, I don't think it should block this improvement.
• I'm still debugging why the MySQL 5.5 container won't start. I've also added this to the exclude list. We don't currently run tests against MySQL 5.5 and while it shouldn't be ignored (currently accounts for 15% of all sites), I also don't think this is a blocker.
• I've gone back and reviewed the MySQL version support. Historically, the changes to support have not been as well announced as PHP. New versions of MySQL usually just work with a few small compatibility issues that don't creep up immediately. Another review is appreciated, though.

Here are a few test workflow runs within my fork to confirm that manually triggered runs work:

• ​WP 4.2.16
• ​WP 5.1.1
• ​WP 5.8.3

[jorbin]

Moving to milestone 6.5 as WP 6.4 RC3 has been released.

This looks really useful. A couple of questions:

1. Why is it helpful to add a scheduled run on every Monday?
2. Is there an official documented page somewhere on WP.org that lists all the supported PHP/MySQL versions for each WP version that this can be cross-referenced against?

1. Why is it helpful to add a scheduled run on every Monday?

My thinking is that regular testing of the nightly build against the complete matrix of supported versions will ensure no bugs in the install process are introduced. It's likely that folks are not regularly creating new installs using the nightly package very often.

2. Is there an official documented page somewhere on WP.org that lists all the supported PHP/MySQL versions for each WP version that this can be cross-referenced against?

There is for ​PHP versions, but not currently for MySQL. I think it's worth adding.

However, in my research, new MySQL versions do not have clear "supported" milestones. For PHP, it's always been "the first release following or immediately preceding a PHP version release". But for MySQL, it seemed to be "it should just work, but there may be some edge cases and bugs that are discovered along the way."

Using MySQL 5.7 as an example, it was ​released on October 21, 2015. ​WordPress 4.4 was the next major version released on November 25, 2015. WP 4.4 did bump the _recommended_ version of MySQL from 5.5 to 5.6, but there are no tickets in the milestone specifically referencing MySQL 5.7. I looked through 4.9, but could not find any MySQL 5.7 compatibility issues (though Trac search is not great). I'll need to do some more historical research to try and find out if there's something I'm missing.

For sake of this PR, I used the MySQL version release dates and cross referenced them with the WP version release dates. I assumed that the first major version of WP after a MySQL release supported that version.

• WP 3.1 first release after MySQL 5.5
• WP 3.6 first release after MySQL 5.6
• WP 4.4 first release after MySQL 5.7
• WP 5.0 first release after MySQL 8.0

WP has also not historically trimmed support for MySQL versions. The current minimum required version sits at 5.0 😱 I plan to address this during the 6.5 cycle. Regardless, the ​oldest versions of MySQL available in Docker is 5.5.x, and that container fails to start in my testing.

[desrosj]

In 57218:

Build/Test Tools: Make the install testing workflow more flexible.

In [56661], a new GitHub Actions workflow was introduced that focused on running some minimal installation tests for a version of WordPress for every PHP and MySQL combination.

This workflow has tested well, but lacks flexibility and possesses one flaw: tests are only ever performed with currently supported versions, even if the version being tested had a different support policy.

This updates the workflow to be more flexible, allowing all versions of WordPress currently receiving security fixes (back through 4.1) to be tested under the correct support policy.

Additionally, the workflow can now run against the nightly build of WordPress. This replaces latest as the new default. This allows the tests to be run at any point during a release cycle without the need for an officially tagged version.

Props jorbin, joemcgill, costdev.
See #58977.

[desrosj]

In 57219:

Build/Test Tools: Add missing PHP versions for 6.5 and 6.4.

This adds 6.4 and 6.5 to the PHP version support JSON files used to build the test matrix in the Installation Testing workflow.

Also, WP 6.3 did not support PHP 8.3.

Follow up to [57218].

See #58977.

[desrosj]

In 57220:

Build/Test Tools: Run installation tests when data files are updated.

When the JSON data files that supply the PHP/MySQL version support to the installation test workflow are changed, the workflow should be run to confirm the changes are correct. This updates the path filtering rules to ensure this happens.

Follow up to [57218], [57219].

See #58977.

Merged in https://core.trac.wordpress.org/changeset/57218.

[desrosj]

In 57221:

Build/Test Tools: Add missing MySQL versions for WordPress 5.0.

Follow up to [57218], [57219], [57220].

See #58977.

[swissspidy]

@desrosj What's left to be done here?

[swissspidy]

Seems like this is fixed by now, no new changes needed.