Test with Gutenberg active

[dd32]

On the back of #60315, an issue that keeps occurring on some sites (​For example, wordpressfoundation.org earlier today) is that WordPress trunk ends up being incompatible with the latest stable release of Gutenberg. Usually this happens around PHP back-porting from Gutenberg to Core, but I guess could happen with some of the JS too (I just don't see it).

Generally, WordPress trunk has been treated in the past as "it should be stable", and as a result, is run on many WordPress.org properties as a way to ​dogfood and find out about bugs before it hits every other WordPress site in the world. Over the years, this has identified many issues that wouldn't have otherwise been noticed until after release. Often these sites consume the SVN/Git repo's directly, and as such the upgrade routines don't protect those sites.

I'd like to propose two changes to the unit tests in Core, as a way to hopefully notice these incompatibilities before it happens.

1. Run a unit test run with Gutenberg stable active (Similar to how we run multisite vs not-multisite, but we can probably slim it down to Gutenberg stable + Trunk + Single PHP version, just to reduce action time)
2. If it's a packages update, run all the tests with Gutenberg active (in addition to without)

For PRs such as the one that lead to the most recent issue, it'd have flagged that the commit would make trunk incompatible with the stable release, and either a) the core commit could've been delayed 12 hours, or b) the release of an updated Gutenberg could've been done first.

For other PRs/commits, it would highlight when a change in Core has caused Gutenberg to fatal or act oddly - Something that should be highly unlikely, but is plausible, and given how close Gutenberg and Core are, it just makes sense to me. This wouldn't replace the Gutenberg unit tests, but would act as a separate double-check.

[dd32]

Additionally; I believe that PHP within Gutenberg is supposed to be guarded against this with class_exists() and function_exists() checks to prevent such issues, but we're all human and it can be missed. Adding checks and standards on the Gutenberg side to enforce that doesn't prevent Core adding a unit test run that ensures it's caught.

This proposal is all about ensuring that trunk remains stable and can be deployed to WordPress.org sites without requiring someone to check all is good.

[ocean90]

There’s actually an e2e test since [54913] / #57197 but I guess it’s broken?

[peterwilsoncc]

Replying to ocean90:

There’s actually an e2e test since [54913] / #57197 but I guess it’s broken?

It does a quick test for activating and deactivating the plugin but doesn't do a full test suite run. I think this is about running the full test suite (phpunit, e2e, etc) with the plugin activated.

[dd32]

Replying to ocean90:

There’s actually an e2e test since [54913] / #57197 but I guess it’s broken?

I guess so.

Attempting to activate the at-the-time stable gutenberg via plugins.php should've hit Plugin could not be activated because it triggered a fatal error. (I just tested this)

If the outcome of this is fixing that e2e test, that'd be enough I guess.

[dd32]

It turns out that this didn't affect every PHP version (Only 7.0~7.3), which is likely why the e2e test (Which runs with PHP 8.2) didn't catch it.

See ​https://github.com/WordPress/gutenberg/issues/58467

Running an extra unit test run with Gutenberg active wouldn't have caught this, unless it was run on the entire PHP matrix (or at least the oldest supported PHP)

I'm inclined to close this as worksforme or invalid based on that, even though I still think something should've flagged it on the Core side.