Make WordPress Core

Opened 7 months ago

Closed 6 months ago

Last modified 6 months ago

#59517 closed enhancement (fixed)

Migrate Puppeteer tests to Playwright

Reported by: swissspidy's profile swissspidy Owned by: swissspidy's profile swissspidy
Milestone: 6.4 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch has-unit-tests
Focuses: Cc:

Description

As announced in https://make.wordpress.org/core/2022/03/23/migrating-wordpress-e2e-tests-to-playwright, Gutenberg contributors have made great progress in updating the setup to a point where it has become very easy to migrate all browser-based tests to Playwright, and all existing tests in core can now be safely converted as well.

Migrating the tests to Playwright not only is in accordance with the plan outlined in that post, it also helps address the intermittent timeout failures we're constantly seeing in core.

Because of that, and since changes to tests and build tools can happen during beta periods as well, I am marking this with the 6.4 milestone.

Change History (18)

This ticket was mentioned in PR #5333 on WordPress/wordpress-develop by @swissspidy.


7 months ago
#1

  • Keywords has-patch has-unit-tests added

As per https://make.wordpress.org/core/2022/03/23/migrating-wordpress-e2e-tests-to-playwright/, this migrates all Puppeteer usage in core to Playwright.

To-do

  • [ ] Maybe set up ESLint/Prettier for formatting...? Odd that core doesn't have this yet 🤷

Trac ticket: https://core.trac.wordpress.org/ticket/59517

This ticket was mentioned in Slack in #core-test by brianhenryie. View the logs.


7 months ago

This ticket was mentioned in Slack in #core-performance by joemcgill. View the logs.


6 months ago

#4 @swissspidy
6 months ago

  • Owner set to swissspidy
  • Resolution set to fixed
  • Status changed from assigned to closed

In 56926:

Build/Test Tools: Migrate Puppeteer tests to Playwright.

As per the migration plan shared last year, this migrates all browser-based tests in WordPress core to use Playwright.
This includes end-to-end, performance, and visual regression tests.

Props swissspidy, mamaduka, kevin940726, bartkalisz, desrosj, adamsilverstein.
Fixes #59517.

#5 @swissspidy
6 months ago

In 56927:

Build/Test Tools: Revert accidental change to Gruntfile.js

This is a follow-up to r56926.

See #59517.

@swissspidy commented on PR #5333:


6 months ago
#6

Committed in https://core.trac.wordpress.org/changeset/56926 🎉

Thanks y'all!

#7 @swissspidy
6 months ago

In 56928:

Build/Test Tools: Fix file prefix handling in performance test results.

This is a follow-up to r56926.

See #59517.

#8 @swissspidy
6 months ago

In 56930:

Build/Test Tools: Fix path check when comparing performance test results.

This is a follow-up to r56926, r56927, r56928.

See #59517.

#9 @swissspidy
6 months ago

In 56934:

Build/Test Tools: Do not round percentages when comparing performance test results.

Props joemcgill.
See #59517.

#10 @afercia
6 months ago

Note:

after [56926], for all the ones who previously ran the Core visual regression tests locally, all the existing files in the tests/visual-regression/specs/__image_snapshots__/ directory will be reported by git as 'Untracked files'. This is because of the changes to the .gitignore file.

If you don't need those snapshot images any longer, you can safely delete them and after next tests run they will be generated in the new path tests/visual-regression/specs/__snapshots__/.

@swissspidy I'd suggest to consider to keep the old path in the .gitignore file, just to avoid unnecessary headache for contributors.

#11 @swissspidy
6 months ago

We discussed that on the PR but decided it's not worth it as it will only affect very few contributors and deleting files is quickly done. This change has also been called out in a make/core post.

#12 @swissspidy
6 months ago

In 56942:

Build/Test Tools: Ensure the failed workflow tasks run last for e2e tests.

This is a follow-up to [56926], where this logic has been reintroduced after it was originally removed in [56198].

Props desrosj.
See #59517.

#13 @SergeyBiryukov
6 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

It looks like [56926] removed the PUPPETEER_SKIP_DOWNLOAD env variable from a few workflow files:

.github/workflows/coding-standards.yml
.github/workflows/end-to-end-tests.yml
.github/workflows/performance.yml
.github/workflows/phpunit-tests-run.yml
.github/workflows/test-coverage.yml
.github/workflows/test-npm.yml

I noticed a recent test failure (from 12 hours ago) that seems to be related to this:

npm ERR! code 1
npm ERR! path /home/runner/work/wordpress-develop/wordpress-develop/node_modules/puppeteer
npm ERR! command failed
npm ERR! command sh -c -- node install.js
npm ERR! ERROR: Failed to set up Chromium r1108766! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
npm ERR! Error: read ECONNRESET
npm ERR!     at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
npm ERR!   errno: -104,
npm ERR!   code: 'ECONNRESET',
npm ERR!   syscall: 'read'
npm ERR! }

Looking at package-lock.json, it appears that puppeteer is still a dependency of some packages, specifically grunt-contrib-qunit.

Should the PUPPETEER_SKIP_DOWNLOAD env variable perhaps be reinstated for now, to avoid downloading Chromium in the workflows that don't need it?

#14 @swissspidy
6 months ago

Ugh, I didn't realize those tests use Puppeteer. I just opened #59644 for migrating them to something more maintainable.

In the meantime I will reinstate those env vars. Thanks!

#15 @swissspidy
6 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 56954:

Build/Test Tools: Reinstate PUPPETEER_SKIP_DOWNLOAD for most CI workflows.

The PUPPETEER_SKIP_DOWNLOAD environment variable is used to prevent Puppeteer from automatically downloading browser binaries.
It was removed in [56926] due to the migration to Playwright. However, because of the QUnit tests, Puppeteer is actually still a dependency.

Until those tests change, we have to keep this environment variable to prevent unnecessary downloads on CI.

Props SergeyBiryukov.
Fixes #59517.

This ticket was mentioned in Slack in #core-performance by joemcgill. View the logs.


6 months ago

#17 @desrosj
6 months ago

In 56973:

Build/Test Tools: Skip Puppeteer download in build workflow.

This adds the PUPPETEER_SKIP_DOWNLOAD environment variable to the Build WordPress workflow to skip downloading Puppeteer browser binaries unnecessarily.

Follow up to [56958].

See #59416, #59517, #58863.

#18 @swissspidy
6 months ago

In 56980:

Build/Test Tools: Remove now obsolete jest-image-snapshot dependency.

With the migration of visual regression tests to Playwright in [56926], this package is no longer needed and can be safely removed.

See #59517.

Note: See TracTickets for help on using tickets.