Make WordPress Core

Opened 20 months ago

Last modified 20 months ago

#56044 new defect (bug)

E2E tests are failing silently

Reported by: desrosj's profile desrosj Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

The end-to-end test workflow appears to be passing successfully, but there are several assertions that actually fail or encounter an error when examining the output.

Example: https://github.com/WordPress/wordpress-develop/runs/7008828515?check_suite_focus=true#step:13:14

Went back a ways to try and find where the problem started, but hit the upper boundary of the retention limit on GitHub Actions (90 days).

Change History (5)

#1 @desrosj
20 months ago

I included a link to the most recent workflow run in the ticket description, but in case this ticket outlasts the 90 day log retention limit, here are the variations of errors being returned:

PASS tests/e2e/specs/edit-posts.test.js (22.74 s)
  ● Console
    console.error
      Failed to load resource: the server responded with a status of 404 (Not Found)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
PASS tests/e2e/specs/profile/applications-passwords.test.js
● Console
    console.error
      Failed to load resource: the server responded with a status of 409 (Conflict)
      119 |
      120 | 		// eslint-disable-next-line no-console
    > 121 | 		console[ logFunction ]( text );
          | 		^
      122 | 	} );
      123 | }
      124 |
      at config/bootstrap.js:121:3
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75
          at Array.map (<anonymous>)
      at Object.emit (../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:56)
      at Page.emit (../../node_modules/puppeteer-core/src/common/EventEmitter.ts:102:18)
      at Page._onLogEntryAdded (../../node_modules/puppeteer-core/src/common/Page.ts:781:12)
      at ../../node_modules/puppeteer-core/src/common/Page.ts:575:49
      at ../../node_modules/puppeteer-core/vendor/mitt/src/index.ts:88:75

#2 @hellofromTonya
20 months ago

@desrosj and I took a look at the logs going back 90 days. There are silent failures going all the back 90 days. Not sure when these failures started or why.

@isabel_brison @kevin940726 could you take a look? Any ideas?

#3 @kevin940726
20 months ago

We bubble up the errors in the page's console in https://github.com/WordPress/wordpress-develop/blob/trunk/tests/e2e/config/bootstrap.js#L121, but we don't use @wordpress/jest-console to throw the errors. Not sure if it's intended or not. Maybe @isabel_brison will know better?

FWIW, these errors mostly came from network failures in CI, not necessary related to the tests themselves.

#4 @isabel_brison
20 months ago

This setup mirrors the Gutenberg one. It was introduced here: https://github.com/WordPress/gutenberg/pull/8721 though the reasoning behind it may not be relevant in Core.

Fwiw I can reproduce the applications-passwords test error locally (though the test still passes), but the error log doesn't provide any meaningful info as to its cause.

#5 @SergeyBiryukov
20 months ago

I can reproduce all of these console errors locally.

These test files throw the errors in my testing:

e2e/specs/profile/applications-passwords.test.js
e2e/specs/edit-posts.test.js
e2e/specs/empty-trash-restore-trashed-posts.test.js

These test files don't:

e2e/specs/dashboard.test.js
e2e/specs/hello.test.js
Note: See TracTickets for help on using tickets.