Consider testing against MySQL "Innovation Releases"

[desrosj]

The ​MySQL project has announced a new version release cadence that it has begun following with version 8.1 (eleased on July 18, 2023), and has since been superseded with 8.2 (released on October 25, 2023).

I recommend reading their full post, but here are some important details I've extracted:

Support for Innovation releases is limited until the next minor release (Innovation or LTS). The current cadence goal is to have an Innovation release every quarter, incrementing the minor version number (eg. 8.2, 8.3, etc.). Patch releases within Innovation releases are possible but are less likely to happen.

About every 2 years a new Long Term Support version will be released (blue). An exception is the 8.x LTS release which will happen well before EOL of 8.0

Innovation releases will also be Generally Available and are recommended to be used in production environments. Bugfixes and Security patches will generally be included in the next Innovation or LTS release and not as part of a patch release within that Innovation release. When using Innovation releases, it is necessary to upgrade to the latest innovation releases regularly to keep up with the latest bugfixes and security patches.

Because innovation versions are considered production safe, we should consider testing against these new versions of MySQL as they are released. However, since innovation releases are replaced more frequently, it's not clear how long we should consider testing against them.

For example, the current Innovation release is 8.2, which will be replaced by 8.3 in (approximately) January 2024. Because of backwards compatibility concerns, should 8.1 and 8.2 continue to be tested? Or is testing against the latest innovation release, and any supported LTS versions enough?

Trac ticket: https://core.trac.wordpress.org/ticket/59779

[desrosj]

I've attached a PR that tests against both MySQL 8.1 and 8.2.

Since 8.1 was replaced by 8.2 on October 25th, I think it's reasonable to only include the most recent version. But I want to collect some more opinions.

[jorbin]

Perhaps we want to test Innovation and Innovation -1 to allow for a slightly lagging update like this PR does.

[desrosj]

Been chatting with @jorbin in Slack, and we have a new proposed approach:

• Test and support all innovation releases for a MySQL version range.
• When the LTS for that version range comes out, stop testing and supporting the innovation versions and only test the LTS.
• Test new innovation versions when they come out after that.

In today's situation this would tentatively mean the following for WP 6.5 and 6.6:

• Test against 8.1 and 8.2 for WP 6.5.
• If 8.3 is released prior to releasing WP 6.5, it will also be tested against.
• When 8.4 is released (assuming that becomes LTS as described in the post), 6.6 tests against that version. Support for 8.1 8.2, and 8.3 is removed in that release and only the LTS is officially supported. All changes in the innovation releases will be in the LTS in some form.
• Go back and remove innovation releases from the WP 6.5 test matrix.
• 9.0 comes out (first innovation release), 6.6 tests against it.

etc.

This allows site owners to try out innovation releases (which MySQL is encouraging and describing as production ready) without abandoning testing for them too soon. The landmark of an LTS version (which will be well publicized) is a sign to update off of the innovation versions for a release range.

If there's no objection to this approach, I'll plan to commit the PR attached here later this week and draft a proposal post along these lines for the Making Core blog.

We're at 500 GHA jobs with this configuration. Is that the limit?

Concurrently, yes. But this includes the Upgrade and Installation tests, which do not run on PRs or commits unless the corresponding workflow file is modified (or run manually). The installation test workflow accounts for 72 jobs, and the upgrade testing one has 384 in this run (some are not counted because the matrix was too large).

So for normal pushes and PRs, we're sitting around around 150-200.

It looks like the matrices in the upgrade testing workflow puts several of the jobs over the maximum allowed number of 255. I think I'll proceed with testing against these innovation releases in the PHPUnit workflow only for now.

[desrosj]

In 57132:

Build/Test Tools: Run unit tests using new MySQL versions.

The MySQL project has introduced a new versioning strategy with two types of releases: innovation and long-term support. While long-term support is a familiar concept, it’s not clear how innovation releases should be supported in various WordPress versions for a number of reasons.

Mainly, innovation releases are very short-lived. Each one is only supported until the next innovation version is released with a goal of one per quarter. Even though these versions will have a short lifespan they are described as producton-grade and generally available.

More discussion is needed to determine exactly how each WordPress release should support these new innovation releases. But while the parameters for proper support are determined, the PHPUnit tests can be run against these innovation releases to detect any potential problems or incompatibilities.

This change adds the 8.1 and 8.2 innovation releases to the testing matrix for the PHPUnit test worfklow.

Props johnbillion, jorbin.
See #59779.

Merged in https://core.trac.wordpress.org/changeset/57132. We can revisit the upgrade/install testing at a later time.

Trac ticket: https://core.trac.wordpress.org/ticket/59779

[desrosj]

In 57379:

Build/Test Tools: Test against MySQL 8.3

Version 8.3 is the latest short-term innovation release of MySQL.

See #59779.

Merged in https://core.trac.wordpress.org/changeset/5737.

[kirasong]

We reviewed this ticket in ​a triage session for 6.5 today.

Since this is a tests related ticket, there's no particular deadline.

However, in terms of moving the ticket forward, and since I noticed existing commits:

What's left here before it is finished/closed?

[swissspidy]

Sounds like this is fixed for 6.5, and a make/core proposal is planned to decide how to handle these innovation releases in the future.