Run Gutenberg e2e tests in core

[swissspidy]

Occasionally there are cases where a change in WordPress core can break the end-to-end tests in Gutenberg because of an uncaught bug. Right now, such bugs are only discovered once a change lands in WordPress trunk.

Ideally, we find a way to run the Gutenberg e2e tests against trunk or, even better, merge them into the core repository so that the tests run on PRs as well.

Previously this couldn't even be done because of the whole Puppeteer vs. Playwright difference, but now both projects use Playwright.

Footnotes is an example where this happened before and would be a valuable test to port over.

[desrosj]

I'm not too familiar with how they are set up on the Gutenberg repository side and I'm not strong with E2E tests. A few questions/thoughts:

• If there's an update to packages upstream that are not merged into trunk, would that cause failures?
• We will need a way to either disable or pin branches of WordPress to specific versions or states of the GB tests. Otherwise new tests will run in old branches and cause failures. Since Gutenberg only usually supports the last few versions, disabling in all branches except the currently supported one is probably fine and appropriate.
• Will the Core E2E tests fail with the presence of Gutenberg?

[swissspidy]

I haven't fully thought about the details upon creating the ticket, but there could be multiple ways to approach this. For instance:

1. Every time we update packages in core, we copy over all the e2e test files from Gutenberg as well so they run. This way we know all the functionality covered by the tests is in core and that the tests should pass. Downside is that it's confusing for contributors as the source of truth for the tests is in another repo.
2. We check out the Gutenberg repo on CI and run their tests, but not against their wp-env environment but the core environment. This requires activating Gutenberg and adding the same test plugins and themes. So it's testing core+Gutenberg and not core itself.
3. Something else?

[swissspidy]

Just stumbled upon tests/gutenberg/run.js — does this actually already exist??

From a commit message:

In core, you can run tests/gutenberg/run.js script to run the Gutenberg e2e tests

against Core with the plugin being disabled. This test ensures that the integration
of the block editor with Core is working properly.

So basically all we need is to run this on CI.