Ensure Gutenberg Plugin e2e Tests only run on WP versions supported by Gutenberg

[jorbin]

Right now, the e2e tests to install and activate the Gutenberg Plugin run on 6.2+ however Gutenberg does not support 6.2 so the test fails on that branch ( ​https://github.com/WordPress/wordpress-develop/actions/runs/8617843489/job/23619026501 )

The tests for the Gutenberg plugin should only run on trunk and branches that currently are supported by Gutenberg.

Follow up to [54913] and #57197.

[johnbillion]

[57972] fixed this for the 6.2 branch.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that activating the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset compares the WordPress version of the current branch (found in package.json) to the Gutenberg plugin's required_wp field. If the WordPress version is lower than the requirement, the test is skipped.

### Testing instructions

• Verify that the test passes on trunk.
• For earlier branches, temporarily cherry-pick the commit from this PR to the relevant branch, and run npm run test:e2e -- tests/e2e/specs/gutenberg-plugin.test.js.
  • It should pass on 6.5 and 6.4.
  • On 6.2, it should be skipped.
  • Per ​https://wordpress.org/plugins/gutenberg/, 6.3 is still supported, so it will currently also pass there.

Trac ticket: https://core.trac.wordpress.org/ticket/60971

Hmm I don't like using semver for this. Can we do it another way?

Why is it even possible to _install_ an incompatible version in the first place?

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to _activate_ the plugin or not?

Hmm I don't like using semver for this. Can we do it another way?

We can try 🙂 (But FWIW, it's only used to compare versions; and AFAICT, the WP version found in package.json should be valid semver, as it e.g. includes the patch-level .0.)

Why is it even possible to _install_ an incompatible version in the first place?

The original error (on the 6.2 branch) is here: ​https://github.com/WordPress/wordpress-develop/actions/runs/8617843489/job/23619026501

This was still with Puppeteer, i.e. using the ​`installPlugin` helper.

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to _activate_ the plugin or not?

Possibly! I don't see it covered ​here. With Playwright, we now use ​this helper; we might get an HTTP error code back from that PUT request that we can evaluate.

I'll dig a bit.

The commit cannot be cherry-picked to 6.3 and prior as the test was refactored (from Puppeteer to Playwright) after 6.3. Per ​https://wordpress.org/plugins/gutenberg/, 6.3 is still supported, so it is currently expected to pass there. However, this is probably best solved by bumping the required_wp field, and deleting the test file from the 6.3 branch.

Gutenberg PR to bump the "Required WP Version" field to 6.4: ​https://github.com/WordPress/gutenberg/pull/60780

I'll dig a bit.

If I do a POST request to wp/v2/plugins?slug=gutenberg on a WP 6.2 install, I get

{
    "code": "incompatible_wp_required_version",
    "message": "The package could not be installed.",
    "data": {
        "status": 500
    },
    "additional_data": [
        "Your WordPress version is 6.2.5, however the uploaded plugin requires 6.3."
    ]
}

We should be able to work with that.

Nice!

Nice!

Thank you! I had to tweak it a bit since it didn't seem to throw upon plugin install, but only upon activation.

Remove tests/e2e/specs/gutenberg-plugin.test.js from the 6.3 branch.

See ​https://github.com/WordPress/wordpress-develop/pull/6396:

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

There's a Gutenberg counterpart PR (​https://github.com/WordPress/gutenberg/pull/60780) to bump the GB plugin's minimum required WP version to 6.4.

Trac ticket: https://core.trac.wordpress.org/ticket/60971

[Bernhard Reiter]

In 58046:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

Thank you @swissspidy!

Committed to Core in https://core.trac.wordpress.org/changeset/58046.
Will re-open the ticket for backporting to 6.5 and 6.4.

[Bernhard Reiter]

Re-opening to request signoff by a second Core Committer for backporting [58046] to the 6.5 and 6.4 branches.

We'll also want to delete the test file altogether from the 6.3 branch. See ​https://github.com/WordPress/wordpress-develop/pull/6397 for the suggested change and the rationale (deleting vs updating the test).

[Bernhard Reiter]

In 58047:

Tests: Remove Gutenberg plugin activation e2e test.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

Additionally, the test was migrated from Puppeteer to Playwright after WP 6.3, so it's not possible to simply backport [58046] (which would skip the test from running on outdated WP versions) to the 6.3 branch.

As a consequence, it makes most sense to remove the test from the 6.3 branch altogether, as has already been done for 6.2.

Follow-up to [57972].
Props jorbin, johnbillion, swissspidy.
See #60971.

Thank you @swissspidy!

Committed to Core's 6.3 branch in https://core.trac.wordpress.org/changeset/58047.

[jorbin]

In 58141:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

See #60971.

[jorbin]

[58046] looks good for backporting to 6.4 and 6.5 with the suggestion that it also include the comment I added in [58141] to make it a bit clearer why the skipping and what the error code means when someone encounters this block of code. Thanks @bernhard-reiter!

[Bernhard Reiter]

In 58151:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Reviewed by jorbin.
Merges [58046] to the 6.5 branch.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

[Bernhard Reiter]

In 58152:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

Reviewed by jorbin.
Merges [58141] to the 6.5 branch.

See #60971.

[Bernhard Reiter]

Reopening once more for backporting to the 6.4 branch.

[Bernhard Reiter]

In 58153:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Reviewed by jorbin.
Merges [58046] to the 6.4 branch.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

[Bernhard Reiter]

In 58154:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

Reviewed by jorbin.
Merges [58141] to the 6.4 branch.

See #60971.