Opened 16 months ago
Closed 9 months ago
#58977 closed task (blessed) (fixed)
Consider adopting workflows for testing build packages
Reported by: | costdev | Owned by: | costdev |
---|---|---|---|
Milestone: | 6.5 | Priority: | normal |
Severity: | normal | Version: | |
Component: | Build/Test Tools | Keywords: | has-patch commit |
Focuses: | Cc: |
Description
Currently, build package installations and upgrades are manually tested by WordPress.org and release party attendees. However, this testing doesn't cover a very wide range of PHP, MySQL and older WordPress versions, and some of these encounter issues that aren't discovered or reported until much later. We've previously had broken ZIP archives, polyfills used where they can't be used, database errors, and such.
I've been working on GitHub Actions workflows to help improve our build package coverage.
More details are in the comment below.
Change History (29)
#2
@
16 months ago
Pinging folks I spoke with regarding this in DM: @azaozz @desrosj @hellofromTonya @peterwilsoncc @afragen @ironprogrammer
#3
@
16 months ago
- Keywords needs-patch added; 2nd-opinion removed
- Milestone changed from Awaiting Review to 6.4
Thinking this is a great idea. "Live testing" at release parties is very nice but "test coverage" may be limited. Having the assurance the packages would work on all possible combinations of WordPress + PHP + MySQL/MariaDB would be very good.
Frankly I'm tempted to add this to the 6.3 milestone as it is for the build tools. Would have been nice to see it in action on release day. But maybe better to first try it on few beta and RC releases :)
#4
@
16 months ago
I've previously run the workflows on 6.3-RC2, 6.3-RC3 and a nightly package.
In terms of a patch, the workflows download and install a copy of WordPress directly into the repo's root during each run. For simplicity, I'd suggest that the repo remains a standalone one rather than integrating it directly into the SVN repository.
I'd think that a transfer of ownership from my GitHub account to the WordPress organisation's account would do here, then if we're going to run them on nightly packages, the workflows would just need to be called with inputs.new_version
being set to nightly
.
#6
@
16 months ago
Rather than a duplicate, I would consider this ticket a partial implementation of @dd32's list here, with #43395 being an umbrella ticket.
Where #43395 mentions E2E testing, permissions, different filesystem abstractions and the like, this ticket could be considered a partial fulfilment of the overall list. Once this ticket is handled, we could then close it and move onto the remaining items on the list, rather than trying to accomplish everything in one ticket.
#7
@
16 months ago
+1 on this initiative! Regular validation of install/upgrade is a great idea 👍🏻
Impact on Release Parties
@costdev makes an important point above regarding the need to continue to encourage release party engagement. Providing a list of items to test after the build is available would present additional guidance for testers to exercise the build beyond the installer process. (Individual/novel install testing would still be encouraged, but need not be the focus of post-build testing efforts.)
The community coordination required to come together for a release party is significant; this proposal can help ensure timely builds and would allow testers to focus on items pertinent to the build.
#9
@
16 months ago
Thanks for the great work here so far, @costdev! I'd love to get this merged and give it a try. Some thoughts:
- I'm not sure it should be removed, but we don't yet test MySQL 8 regularly in general. I've been working on #30462 to add this. I don't think that 8.0 needs to be removed from this in order to be merged, but there may be some strange incompatibilities that pop up (old versions were not fully compatible).
- I would very much prefer that we get a bit creative with matrix strategies instead of splitting it into several workflows for the upgrade testing. I think we can absolutely stay below the maximum number of jobs with certain groupings. I have some ideas around how we could do this.
- Let's move this repository to the
WordPress
organization in order to iterate. I've reached out to @costdev in Slack to facilitate this. It will be easier to suggest and test changes under the organization's enterprise account (which has 500 concurrent jobs).
#10
@
15 months ago
- Keywords has-patch commit added; needs-patch removed
I've been working with @costdev quite a bit over on the WordPress/wordpress-upgrade-tester repository where we moved his POC.
I think this is in a great state. It currently:
- Tests updating all branches of WordPress that receive security updates to the version being tested.
- Tests updating with all currently supported PHP versions (where possible).
- Tests updating with all currently supported MySQL versions.
Let's get this in to see how it goes throughout the 6.4 beta/RC period leading up to final release.
For now, it will only run if manually triggered. I've checked that this is possible (and it is), but in the future when the WP.org server creates the release package for a specific version, it should make a request to GitHub with the version number being packaged to trigger this workflow automatically.
Here's a list of remaining tasks that are not blockers, but need to be explored further (not necessarily in this ticket):
- Perform some actual test assertions. Currently, the workflow only checks that WP-CLI does not encounter a failure when attempting to update. It would be good to also have some tests that perform some basic tasks to confirm things look good and actually work.
- When testing minor releases for older versions, we should only test up to that version. For example, if 5.0.10 is being released, we shouldn't test updating 5.1.x to 5.0.10. Only WP versions lower than the current version should be tested.
- When testing minor releases for older versions, we should test the versions of PHP that were supported for that version. For example, if we're releasing a 4.9 minor version, we should test PHP 5.2-5.6, and 7.0-7.2. This also applies for database versions.
- Ideally, the workflow should also perform MariaDB testing. After [56439], MariaDB is tested by default for all commits.
One last call out is that the workflow takes a bit of time to report success. However, the jobs in the workflow itself run surprisingly fast. The time that it actually takes to run all of the associated jobs can be calculated by monitoring the GitHub Action Runner administration page when the workflow starts. The jobs are all completed within 5-8 minutes, but the delay seems to be with the product UI receiving and displaying job details from the 650+ runners, and takes double to triple that amount of time.
During the release process, this could be a pain, but that usually takes around that same amount of time. But this workflow is not meant to replace manual testing, but rather serve as an additional sanity check to catch some additional potential issues. I've reached out to GitHub about this, and we can continue to explore ways to speed this up, but I think it will need to be fixed on their end.
This ticket was mentioned in PR #5282 on WordPress/wordpress-develop by @desrosj.
15 months ago
#11
Trac ticket: https://core.trac.wordpress.org/ticket/58977
15 months ago
#13
Merged in https://core.trac.wordpress.org/changeset/56661. Thanks @costdev!
This ticket was mentioned in Slack in #core by costdev. View the logs.
15 months ago
This ticket was mentioned in PR #5482 on WordPress/wordpress-develop by @desrosj.
14 months ago
#16
This makes the install testing workflow a bit more flexible in order to allow usage for any branch of WordPress currently receiving security updates.
Because you can't manually start workflows for a PR (that I'm aware of), I've run the workflow manually within my fork. Here are a few test workflows:
Remaining items:
- https://github.com/desrosj/wordpress-develop/actions/runs/6512389627/job/17689928003#step:3:15 The correct version of WP CLI is not [installing on PHP 5.3], but is for 5.4-5.5.
- [ ] The MySQL versions needs to be verified and confirmed.
- [ ] It's unlikely because of how WP versions and ships security updates, but Is there a more standard way to list version support?
Trac ticket: https://core.trac.wordpress.org/ticket/58977
14 months ago
#17
Some updates here:
- I've added an exclusion for testing PHP 5.3 for now. While it would be nice to have this testing running, I don't think it should block this improvement.
- I'm still debugging why the MySQL 5.5 container won't start. I've also added this to the exclude list. We don't currently run tests against MySQL 5.5 and while it shouldn't be ignored (currently accounts for 15% of all sites), I also don't think this is a blocker.
- I've gone back and reviewed the MySQL version support. Historically, the changes to support have not been as well announced as PHP. New versions of MySQL usually just work with a few small compatibility issues that don't creep up immediately. Another review is appreciated, though.
14 months ago
#18
Here are a few test workflow runs within my fork to confirm that manually triggered runs work:
#19
@
13 months ago
- Milestone changed from 6.4 to 6.5
Moving to milestone 6.5 as WP 6.4 RC3 has been released.
@joemcgill commented on PR #5482:
13 months ago
#20
This looks really useful. A couple of questions:
- Why is it helpful to add a scheduled run on every Monday?
- Is there an official documented page somewhere on WP.org that lists all the supported PHP/MySQL versions for each WP version that this can be cross-referenced against?
13 months ago
#21
- Why is it helpful to add a scheduled run on every Monday?
My thinking is that regular testing of the nightly build against the complete matrix of supported versions will ensure no bugs in the install process are introduced. It's likely that folks are not regularly creating new installs using the nightly package very often.
- Is there an official documented page somewhere on WP.org that lists all the supported PHP/MySQL versions for each WP version that this can be cross-referenced against?
There is for PHP versions, but not currently for MySQL. I think it's worth adding.
However, in my research, new MySQL versions do not have clear "supported" milestones. For PHP, it's always been "the first release following or immediately preceding a PHP version release". But for MySQL, it seemed to be "it should just work, but there may be some edge cases and bugs that are discovered along the way."
Using MySQL 5.7 as an example, it was released on October 21, 2015. WordPress 4.4 was the next major version released on November 25, 2015. WP 4.4 did bump the _recommended_ version of MySQL from 5.5 to 5.6, but there are no tickets in the milestone specifically referencing MySQL 5.7. I looked through 4.9, but could not find any MySQL 5.7 compatibility issues (though Trac search is not great). I'll need to do some more historical research to try and find out if there's something I'm missing.
For sake of this PR, I used the MySQL version release dates and cross referenced them with the WP version release dates. I assumed that the first major version of WP after a MySQL release supported that version.
- WP 3.1 first release after MySQL 5.5
- WP 3.6 first release after MySQL 5.6
- WP 4.4 first release after MySQL 5.7
- WP 5.0 first release after MySQL 8.0
WP has also not historically trimmed support for MySQL versions. The current minimum required version sits at 5.0 😱 I plan to address this during the 6.5 cycle. Regardless, the oldest versions of MySQL available in Docker is 5.5.x, and that container fails to start in my testing.
What do the workflows cover?
Installation - 32 tests
Upgrade - 620 tests
__autoload()
and curly braces for array/string offsets.All currently run on
ubuntu-latest
.What could they cover later?
windows-latest
andmac-latest
.How do they work?
The workflows leverage WP-CLI to run installation and upgrade tests.
To ensure that matrices don't exceed 256 jobs for now, the workflows are split into:
Each workflow accepts one input:
new_version
- The version being installed/upgraded to. Any value that works with the--version
parameter in WP-CLI will work for this, including6.3
,6.3-RC3
andnightly
.How long do they take?
As my GitHub plan means I'm limited to 20 concurrent jobs, the workflows take longer than they would on WordPress' plan. I'd expect all 652 jobs to complete within 1-3 minutes if adopted by WordPress.
How could they be used?
For example: