Make WordPress Core

Opened 7 weeks ago

Closed 11 days ago

Last modified 11 days ago

#60971 closed defect (bug) (fixed)

Ensure Gutenberg Plugin e2e Tests only run on WP versions supported by Gutenberg

Reported by: jorbin's profile jorbin Owned by: bernhard-reiter's profile Bernhard Reiter
Milestone: 6.6 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch has-unit-tests fixed-major dev-reviewed
Focuses: Cc:

Description

Right now, the e2e tests to install and activate the Gutenberg Plugin run on 6.2+ however Gutenberg does not support 6.2 so the test fails on that branch ( https://github.com/WordPress/wordpress-develop/actions/runs/8617843489/job/23619026501 )

The tests for the Gutenberg plugin should only run on trunk and branches that currently are supported by Gutenberg.

Follow up to [54913] and #57197.

Change History (22)

#1 @johnbillion
7 weeks ago

[57972] fixed this for the 6.2 branch.

This ticket was mentioned in PR #6396 on WordPress/wordpress-develop by @Bernhard Reiter.


6 weeks ago
#2

  • Keywords has-patch has-unit-tests added

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that activating the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset compares the WordPress version of the current branch (found in package.json) to the Gutenberg plugin's required_wp field. If the WordPress version is lower than the requirement, the test is skipped.

### Testing instructions

  • Verify that the test passes on trunk.
  • For earlier branches, temporarily cherry-pick the commit from this PR to the relevant branch, and run npm run test:e2e -- tests/e2e/specs/gutenberg-plugin.test.js.

Trac ticket: https://core.trac.wordpress.org/ticket/60971

@swissspidy commented on PR #6396:


6 weeks ago
#3

Hmm I don't like using semver for this. Can we do it another way?

Why is it even possible to _install_ an incompatible version in the first place?

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to _activate_ the plugin or not?

@Bernhard Reiter commented on PR #6396:


6 weeks ago
#4

Hmm I don't like using semver for this. Can we do it another way?

We can try 🙂 (But FWIW, it's only used to compare versions; and AFAICT, the WP version found in package.json should be valid semver, as it e.g. includes the patch-level .0.)

Why is it even possible to _install_ an incompatible version in the first place?

The original error (on the 6.2 branch) is here: https://github.com/WordPress/wordpress-develop/actions/runs/8617843489/job/23619026501

This was still with Puppeteer, i.e. using the `installPlugin` helper.

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to _activate_ the plugin or not?

Possibly! I don't see it covered here. With Playwright, we now use this helper; we might get an HTTP error code back from that PUT request that we can evaluate.

I'll dig a bit.

@Bernhard Reiter commented on PR #6396:


6 weeks ago
#5

The commit cannot be cherry-picked to 6.3 and prior as the test was refactored (from Puppeteer to Playwright) after 6.3. Per https://wordpress.org/plugins/gutenberg/, 6.3 is still supported, so it is currently expected to pass there. However, this is probably best solved by bumping the required_wp field, and deleting the test file from the 6.3 branch.

Gutenberg PR to bump the "Required WP Version" field to 6.4: https://github.com/WordPress/gutenberg/pull/60780

@Bernhard Reiter commented on PR #6396:


6 weeks ago
#6

I'll dig a bit.

If I do a POST request to wp/v2/plugins?slug=gutenberg on a WP 6.2 install, I get

{
    "code": "incompatible_wp_required_version",
    "message": "The package could not be installed.",
    "data": {
        "status": 500
    },
    "additional_data": [
        "Your WordPress version is 6.2.5, however the uploaded plugin requires 6.3."
    ]
}

We should be able to work with that.

@swissspidy commented on PR #6396:


6 weeks ago
#7

Nice!

@Bernhard Reiter commented on PR #6396:


6 weeks ago
#8

Nice!

Thank you! I had to tweak it a bit since it didn't seem to throw upon plugin install, but only upon activation.

This ticket was mentioned in PR #6397 on WordPress/wordpress-develop by @Bernhard Reiter.


6 weeks ago
#9

Remove tests/e2e/specs/gutenberg-plugin.test.js from the 6.3 branch.

See https://github.com/WordPress/wordpress-develop/pull/6396:

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

There's a Gutenberg counterpart PR (https://github.com/WordPress/gutenberg/pull/60780) to bump the GB plugin's minimum required WP version to 6.4.

Trac ticket: https://core.trac.wordpress.org/ticket/60971

#10 @Bernhard Reiter
4 weeks ago

  • Owner set to Bernhard Reiter
  • Resolution set to fixed
  • Status changed from new to closed

In 58046:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

@Bernhard Reiter commented on PR #6396:


4 weeks ago
#13

Thank you @swissspidy!

Committed to Core in https://core.trac.wordpress.org/changeset/58046.
Will re-open the ticket for backporting to 6.5 and 6.4.

#14 @Bernhard Reiter
4 weeks ago

  • Keywords fixed-major dev-feedback added
  • Resolution fixed deleted
  • Status changed from closed to reopened

Re-opening to request signoff by a second Core Committer for backporting [58046] to the 6.5 and 6.4 branches.

We'll also want to delete the test file altogether from the 6.3 branch. See https://github.com/WordPress/wordpress-develop/pull/6397 for the suggested change and the rationale (deleting vs updating the test).

#15 @Bernhard Reiter
4 weeks ago

In 58047:

Tests: Remove Gutenberg plugin activation e2e test.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

Additionally, the test was migrated from Puppeteer to Playwright after WP 6.3, so it's not possible to simply backport [58046] (which would skip the test from running on outdated WP versions) to the 6.3 branch.

As a consequence, it makes most sense to remove the test from the 6.3 branch altogether, as has already been done for 6.2.

Follow-up to [57972].
Props jorbin, johnbillion, swissspidy.
See #60971.

@Bernhard Reiter commented on PR #6397:


4 weeks ago
#16

Thank you @swissspidy!

Committed to Core's 6.3 branch in https://core.trac.wordpress.org/changeset/58047.

This ticket was mentioned in Slack in #core by bernhardreiter. View the logs.


13 days ago

#18 @jorbin
12 days ago

In 58141:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

See #60971.

#19 @jorbin
12 days ago

  • Keywords dev-reviewed added; dev-feedback removed

[58046] looks good for backporting to 6.4 and 6.5 with the suggestion that it also include the comment I added in [58141] to make it a bit clearer why the skipping and what the error code means when someone encounters this block of code. Thanks @bernhard-reiter!

#20 @Bernhard Reiter
11 days ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 58151:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Reviewed by jorbin.
Merges [58046] to the 6.5 branch.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

#21 @Bernhard Reiter
11 days ago

In 58152:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

Reviewed by jorbin.
Merges [58141] to the 6.5 branch.

See #60971.

#22 @Bernhard Reiter
11 days ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening once more for backporting to the 6.4 branch.

#23 @Bernhard Reiter
11 days ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 58153:

Tests: Skip Gutenberg plugin activation test on older WP versions.

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test was introduced in [54913], i.e. it has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset checks the REST API response from the plugin activation request. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Reviewed by jorbin.
Merges [58046] to the 6.4 branch.

Props jorbin, johnbillion, swissspidy.
Fixes #60971.

#24 @Bernhard Reiter
11 days ago

In 58154:

Tests: Add comment to explain when e2e test for gutenberg is skipped

Follow-up to [58046].

Reviewed by jorbin.
Merges [58141] to the 6.4 branch.

See #60971.

Note: See TracTickets for help on using tickets.