WordPress.org

Make WordPress Core

Opened 11 months ago

Last modified 18 hours ago

#49606 new enhancement

Add visual regression testing to core

Reported by: isabel_brison Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: ui, css Cc:

Description (last modified by SergeyBiryukov)

This idea came up during discussion about the upcoming CSS audit #49582. If we have a suite of visual regression tests in place, it will give us confidence to make changes to existing CSS without breaking anything.

What tools to use?

We're in the process of adding e2e tests to core: #49507 and these are using Puppeteer and Jest, so it would be nice for visual regression testing to follow along the same lines. There are a few options to look into:

Change History (25)

#1 @SergeyBiryukov
11 months ago

  • Description modified (diff)

This ticket was mentioned in Slack in #core-js by aduth. View the logs.


11 months ago

#3 @netweb
11 months ago

This comment is really about me subscribing to this ticket for notification to be honest, but I am keen to see the results of https://github.com/americanexpress/jest-image-snapshot testing

#4 @isabel_brison
11 months ago

Update: discussion in #core-js https://wordpress.slack.com/archives/C5UNMSU4R/p1583849715154400 revealed there's a PR in Gutenberg that explores adding visual regression testing with percy.io https://github.com/WordPress/gutenberg/pull/18797

Worth investigating if this is a viable option for us, given it's a commercial service. The main advantage would be not having to store the test snapshots in our repo, as they might be quite large.

#5 @netweb
11 months ago

Thanks @isabel_brison, I’ll take a read of that later, and sure repo size of storing visual snapshots might be an issue.

What I’m hoping for is that visual regression testing by way of Jest can be extended into @wordpress/scripts package so that developers using the existing integration and e2e already offered in that package can also benefit from this also.

I know a lot of the projects and devs I work with are all considering how best to add visual regression testing right now, so being able to do implement that without the use of a commercial service would be beneficial for the wider community

This ticket was mentioned in Slack in #core-css by isabelbrison. View the logs.


11 months ago

This ticket was mentioned in Slack in #design by notlaura. View the logs.


11 months ago

This ticket was mentioned in Slack in #accessibility by audrasjb. View the logs.


10 months ago

This ticket was mentioned in PR #209 on WordPress/wordpress-develop by tellthemachines.


10 months ago

<!--
Hi there! Thanks for contributing to WordPress!

Pull Requests in this GitHub repository must be linked to a ticket in the WordPress Core Trac instance (https://core.trac.wordpress.org), and are only used for code review. No pull requests will be merged on GitHub.

See the WordPress Handbook page on using PRs for Code Review more information: https://make.wordpress.org/core/handbook/contribute/git/github-pull-requests-for-code-review/

If this is your first time contributing, you may also find reviewing these guides first to be helpful:

-->

Trac ticket: [49606](https://core.trac.wordpress.org/ticket/49606)

Adding visual regression tests for wp-admin pages. Uses [jest-image-snapshot](https://github.com/americanexpress/jest-image-snapshot), which integrates seamlessly with our Jest/Puppeteer setup for e2e tests.

One thing to consider is that these tests are a little slow to run. If we find it's not useful to run them as part of the e2e tests every time, we might want to split them out so they can be run separately.

I have added only snapshots of static pages for now. All pages with dynamic content will need a custom setup where we specify areas of the page where changes should be ignored. (It's pretty straightforward to do, but I thought it better to keep the initial changeset simple.)

If we find the tests are throwing a lot of false positives, we can also fiddle with the resolution until they pass.

This ticket was mentioned in Slack in #core-css by isabelbrison. View the logs.


10 months ago

#11 @isabel_brison
6 months ago

Update on this: I did some experiments on the linked GitHub PR and concluded it's virtually impossible for the tests to pass when comparing snapshots generated locally vs on Travis. This is probably due to resolution differences, but I tried fiddling with all the variables provided and no luck.

Also, because these tests would mostly be useful when refactoring CSS, I'm not convinced we should run them as part of the Core e2e test suite. So my idea is we could have a separate setup script that would allow anyone doing refactoring work to run them locally as needed.

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


3 months ago

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


3 months ago

This ticket was mentioned in Slack in #core-css by isabelbrison. View the logs.


3 months ago

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


3 months ago

This ticket was mentioned in Slack in #core-css by ryelle. View the logs.


7 weeks ago

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


3 weeks ago

#18 @prbot
2 weeks ago

tellthemachines commented on PR #209:

Update: this is ready for review now!

I have separated out the config from the e2e tests, and added a dedicated script (test:visual) so we can run these tests independently, and they're not automatically part of CI checks for all code changes.

I have also added the generated snapshots folder to .gitignore, so that snapshots are only stored locally. The tests don't really work cross-environment as even tiny differences between envs throw up too many false positives. If we do want to run them on CI at any point, we can set them up to run first on trunk to generate the snapshots, and then diff with the feature branch. That also solves the problem of storing huge .png files in the repo 😄

The tests are currently only running on pages with mostly static content (assuming a dedicated test environment where new content isn't added all the time, and the only things that change with frequency are dashboard contents and update notifications) but we can probably include more pages as long as we explicitly ignore the dynamic elements in them.

#19 @prbot
2 weeks ago

tellthemachines commented on PR #209:

Another thing we'll probably want to add soon is tests for mobile breakpoints; currently we're only testing 1000px wide screens.

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


13 days ago

This ticket was mentioned in Slack in #core-css by notlaura. View the logs.


6 days ago

#22 @prbot
6 days ago

talldan commented on PR #209:

Not sure what's going on, but I had issues running the tests locally. I used the --puppeteer-interactive command and it looks like the tests couldn't login, which is unusual:

@tellthemachines, I had a sudden brainwave. I think this might be related to the version of puppeteer in core—it's pretty outdated compared to Gutenberg, and IIRC the major changes were around page.waitForNavigation.

Thankfully updating should be an easy task as there are no tests to update 😄

#23 @prbot
22 hours ago

tellthemachines commented on PR #209:

Not sure what's going on, but I had issues running the tests locally. I used the --puppeteer-interactive command and it looks like the tests couldn't login, which is unusual:

Hmm, that's funny, it works fine for me locally with npm run test:visual -- --puppeteer-interactive. Can you get it working ok without the interactive flag?

Would be great to see some docs on how the tests work (I imagine you have to generate some initial snapshots first and then test against that on a feature branch?)

That's the expected workflow with this setup, because testing across different environments is incredibly painful, but that's one of the things I'm looking for feedback on! Agree we should publish some docs when this is ready. Maybe in the handbook testing section?

#24 @prbot
19 hours ago

talldan commented on PR #209:

Hmm, that's funny, it works fine for me locally with npm run test:visual -- --puppeteer-interactive. Can you get it working ok without the interactive flag?

The issue also happens without the --puppeteer-interactive flag, I used that to determine what was going wrong.

Would be good to get more folks testing to see if it's just me. If it is just me then I know I need to spend some time fixing my dodgy local environment 😄

#25 @prbot
18 hours ago

danfarrow commented on PR #209:

Would be good to get more folks testing to see if it's just me. If it is just me then I know I need to spend some time fixing my dodgy local environment smile

hi @talldan, I just checked this out. The tests are running, producing visual diffs etc. both with & without --puppeteer-interactive.

Let me know if you need any info about my setup for comparison.

Note: See TracTickets for help on using tickets.