E2E tests are failing silently

[desrosj]

The end-to-end test workflow appears to be passing successfully, but there are several assertions that actually fail or encounter an error when examining the output.

Example: ​https://github.com/WordPress/wordpress-develop/runs/7008828515?check_suite_focus=true#step:13:14

Went back a ways to try and find where the problem started, but hit the upper boundary of the retention limit on GitHub Actions (90 days).

[desrosj]

I included a link to the most recent workflow run in the ticket description, but in case this ticket outlasts the 90 day log retention limit, here are the variations of errors being returned:

PASS tests/e2e/specs/edit-posts.test.js (22.74 s)
  ● Console
    console.error
      Failed to load resource: the server responded with a status of 404 (Not Found)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75

PASS tests/e2e/specs/profile/applications-passwords.test.js
● Console
    console.error
      Failed to load resource: the server responded with a status of 409 (Conflict)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75

[hellofromTonya]

@desrosj and I took a look at the logs going back 90 days. There are silent failures going all the back 90 days. Not sure when these failures started or why.

@isabel_brison @kevin940726 could you take a look? Any ideas?

[kevin940726]

We bubble up the errors in the page's console in ​https://github.com/WordPress/wordpress-develop/blob/trunk/tests/e2e/config/bootstrap.js#L121, but we don't use @wordpress/jest-console to throw the errors. Not sure if it's intended or not. Maybe @isabel_brison will know better?

FWIW, these errors mostly came from network failures in CI, not necessary related to the tests themselves.

[isabel_brison]

This setup mirrors the Gutenberg one. It was introduced here: ​https://github.com/WordPress/gutenberg/pull/8721 though the reasoning behind it may not be relevant in Core.

Fwiw I can reproduce the applications-passwords test error locally (though the test still passes), but the error log doesn't provide any meaningful info as to its cause.

[SergeyBiryukov]

I can reproduce all of these console errors locally.

These test files throw the errors in my testing:

e2e/specs/profile/applications-passwords.test.js
e2e/specs/edit-posts.test.js
e2e/specs/empty-trash-restore-trashed-posts.test.js

These test files don't:

e2e/specs/dashboard.test.js
e2e/specs/hello.test.js