Opened 15 months ago
Last modified 15 months ago
#56044 new defect (bug)
E2E tests are failing silently
Reported by: |
|
Owned by: | |
---|---|---|---|
Milestone: | Awaiting Review | Priority: | normal |
Severity: | normal | Version: | |
Component: | Build/Test Tools | Keywords: | |
Focuses: | Cc: |
Description
The end-to-end test workflow appears to be passing successfully, but there are several assertions that actually fail or encounter an error when examining the output.
Example: https://github.com/WordPress/wordpress-develop/runs/7008828515?check_suite_focus=true#step:13:14
Went back a ways to try and find where the problem started, but hit the upper boundary of the retention limit on GitHub Actions (90 days).
Change History (5)
#2
@
15 months ago
@desrosj and I took a look at the logs going back 90 days. There are silent failures going all the back 90 days. Not sure when these failures started or why.
@isabel_brison @kevin940726 could you take a look? Any ideas?
#3
@
15 months ago
We bubble up the errors in the page's console in https://github.com/WordPress/wordpress-develop/blob/trunk/tests/e2e/config/bootstrap.js#L121, but we don't use @wordpress/jest-console
to throw the errors. Not sure if it's intended or not. Maybe @isabel_brison will know better?
FWIW, these errors mostly came from network failures in CI, not necessary related to the tests themselves.
#4
@
15 months ago
This setup mirrors the Gutenberg one. It was introduced here: https://github.com/WordPress/gutenberg/pull/8721 though the reasoning behind it may not be relevant in Core.
Fwiw I can reproduce the applications-passwords test error locally (though the test still passes), but the error log doesn't provide any meaningful info as to its cause.
#5
@
15 months ago
I can reproduce all of these console errors locally.
These test files throw the errors in my testing:
e2e/specs/profile/applications-passwords.test.js e2e/specs/edit-posts.test.js e2e/specs/empty-trash-restore-trashed-posts.test.js
These test files don't:
e2e/specs/dashboard.test.js e2e/specs/hello.test.js
I included a link to the most recent workflow run in the ticket description, but in case this ticket outlasts the 90 day log retention limit, here are the variations of errors being returned: