Make WordPress Core

Opened 2 years ago

Last modified 2 years ago

#56044 new defect (bug)

E2E tests are failing silently

Reported by: desrosj's profile desrosj Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

The end-to-end test workflow appears to be passing successfully, but there are several assertions that actually fail or encounter an error when examining the output.

Example: https://github.com/WordPress/wordpress-develop/runs/7008828515?check_suite_focus=true#step:13:14

Went back a ways to try and find where the problem started, but hit the upper boundary of the retention limit on GitHub Actions (90 days).

Change History (5)

#1 @desrosj
2 years ago

I included a link to the most recent workflow run in the ticket description, but in case this ticket outlasts the 90 day log retention limit, here are the variations of errors being returned:

PASS tests/e2e/specs/edit-posts.test.js (22.74 s)
  ● Console
    console.error
      Failed to load resource: the server responded with a status of 404 (Not Found)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
PASS tests/e2e/specs/profile/applications-passwords.test.js
● Console
    console.error
      Failed to load resource: the server responded with a status of 409 (Conflict)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75

#2 @hellofromTonya
2 years ago

@desrosj and I took a look at the logs going back 90 days. There are silent failures going all the back 90 days. Not sure when these failures started or why.

@isabel_brison @kevin940726 could you take a look? Any ideas?

#3 @kevin940726
2 years ago

We bubble up the errors in the page's console in https://github.com/WordPress/wordpress-develop/blob/trunk/tests/e2e/config/bootstrap.js#L121, but we don't use @wordpress/jest-console to throw the errors. Not sure if it's intended or not. Maybe @isabel_brison will know better?

FWIW, these errors mostly came from network failures in CI, not necessary related to the tests themselves.

#4 @isabel_brison
2 years ago

This setup mirrors the Gutenberg one. It was introduced here: https://github.com/WordPress/gutenberg/pull/8721 though the reasoning behind it may not be relevant in Core.

Fwiw I can reproduce the applications-passwords test error locally (though the test still passes), but the error log doesn't provide any meaningful info as to its cause.

#5 @SergeyBiryukov
2 years ago

I can reproduce all of these console errors locally.

These test files throw the errors in my testing:

e2e/specs/profile/applications-passwords.test.js
e2e/specs/edit-posts.test.js
e2e/specs/empty-trash-restore-trashed-posts.test.js

These test files don't:

e2e/specs/dashboard.test.js
e2e/specs/hello.test.js
Note: See TracTickets for help on using tickets.