Migrate Puppeteer tests to Playwright

[swissspidy]

As announced in ​https://make.wordpress.org/core/2022/03/23/migrating-wordpress-e2e-tests-to-playwright, Gutenberg contributors have made great progress in updating the setup to a point where it has become very easy to migrate all browser-based tests to Playwright, and all existing tests in core can now be safely converted as well.

Migrating the tests to Playwright not only is in accordance with the plan outlined in that post, it also helps address the intermittent timeout failures we're constantly seeing in core.

Because of that, and since changes to tests and build tools can happen during beta periods as well, I am marking this with the 6.4 milestone.

As per ​https://make.wordpress.org/core/2022/03/23/migrating-wordpress-e2e-tests-to-playwright/, this migrates all Puppeteer usage in core to Playwright.

To-do

• [ ] Maybe set up ESLint/Prettier for formatting...? Odd that core doesn't have this yet 🤷

Trac ticket: https://core.trac.wordpress.org/ticket/59517

[swissspidy]

In 56926:

Build/Test Tools: Migrate Puppeteer tests to Playwright.

As per the migration plan shared last year, this migrates all browser-based tests in WordPress core to use Playwright.
This includes end-to-end, performance, and visual regression tests.

Props swissspidy, mamaduka, kevin940726, bartkalisz, desrosj, adamsilverstein.
Fixes #59517.

[swissspidy]

In 56927:

Build/Test Tools: Revert accidental change to Gruntfile.js

This is a follow-up to r56926.

See #59517.

Committed in https://core.trac.wordpress.org/changeset/56926 🎉

Thanks y'all!

[swissspidy]

In 56928:

Build/Test Tools: Fix file prefix handling in performance test results.

This is a follow-up to r56926.

See #59517.

[swissspidy]

In 56930:

Build/Test Tools: Fix path check when comparing performance test results.

This is a follow-up to r56926, r56927, r56928.

See #59517.

[swissspidy]

In 56934:

Build/Test Tools: Do not round percentages when comparing performance test results.

Props joemcgill.
See #59517.

[afercia]

Note:

after [56926], for all the ones who previously ran the Core visual regression tests locally, all the existing files in the tests/visual-regression/specs/__image_snapshots__/ directory will be reported by git as 'Untracked files'. This is because of the changes to the .gitignore file.

If you don't need those snapshot images any longer, you can safely delete them and after next tests run they will be generated in the new path tests/visual-regression/specs/__snapshots__/.

@swissspidy I'd suggest to consider to keep the old path in the .gitignore file, just to avoid unnecessary headache for contributors.

[swissspidy]

We discussed that on the PR but decided it's not worth it as it will only affect very few contributors and deleting files is quickly done. This change has also been called out in ​a make/core post.

[swissspidy]

In 56942:

Build/Test Tools: Ensure the failed workflow tasks run last for e2e tests.

This is a follow-up to [56926], where this logic has been reintroduced after it was originally removed in [56198].

Props desrosj.
See #59517.

[SergeyBiryukov]

It looks like [56926] removed the PUPPETEER_SKIP_DOWNLOAD env variable from a few workflow files:

.github/workflows/coding-standards.yml
.github/workflows/end-to-end-tests.yml
.github/workflows/performance.yml
.github/workflows/phpunit-tests-run.yml
.github/workflows/test-coverage.yml
.github/workflows/test-npm.yml

I noticed a ​recent test failure (from 12 hours ago) that seems to be related to this:

npm ERR! code 1
npm ERR! path /home/runner/work/wordpress-develop/wordpress-develop/node_modules/puppeteer
npm ERR! command failed
npm ERR! command sh -c -- node install.js
npm ERR! ERROR: Failed to set up Chromium r1108766! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
npm ERR! Error: read ECONNRESET
npm ERR!     at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
npm ERR!   errno: -104,
npm ERR!   code: 'ECONNRESET',
npm ERR!   syscall: 'read'
npm ERR! }

Looking at package-lock.json, it appears that puppeteer is still a dependency of some packages, specifically grunt-contrib-qunit.

Should the PUPPETEER_SKIP_DOWNLOAD env variable perhaps be reinstated for now, to avoid downloading Chromium in the workflows that don't need it?

[swissspidy]

Ugh, I didn't realize those tests use Puppeteer. I just opened #59644 for migrating them to something more maintainable.

In the meantime I will reinstate those env vars. Thanks!

[swissspidy]

In 56954:

Build/Test Tools: Reinstate PUPPETEER_SKIP_DOWNLOAD for most CI workflows.

The PUPPETEER_SKIP_DOWNLOAD environment variable is used to prevent Puppeteer from automatically downloading browser binaries.
It was removed in [56926] due to the migration to Playwright. However, because of the QUnit tests, Puppeteer is actually still a dependency.

Until those tests change, we have to keep this environment variable to prevent unnecessary downloads on CI.

Props SergeyBiryukov.
Fixes #59517.

[desrosj]

In 56973:

Build/Test Tools: Skip Puppeteer download in build workflow.

This adds the PUPPETEER_SKIP_DOWNLOAD environment variable to the Build WordPress workflow to skip downloading Puppeteer browser binaries unnecessarily.

Follow up to [56958].

See #59416, #59517, #58863.

[swissspidy]

In 56980:

Build/Test Tools: Remove now obsolete jest-image-snapshot dependency.

With the migration of visual regression tests to Playwright in [56926], this package is no longer needed and can be safely removed.

See #59517.