WordPress.org

Make WordPress Core

Opened 6 months ago

Closed 6 months ago

#47622 closed defect (bug) (fixed)

The E2E test has a wrong assertion

Reported by: hideokamoto Owned by: youknowriad
Milestone: 5.3 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: commit
Focuses: javascript Cc:
PR Number:

Description

The E2E test succeeds even if we enter whatever value
https://github.com/WordPress/wordpress-develop/blob/master/tests/e2e/specs/hello.test.js#L4-L10

Expect behavior

The test should fail when we put invalid text condition.
And I tried to run the following test code, the E2E test has been passed.

Code

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const title = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( title ).not.toBeNull();
	} );
} );

Result

> wp-scripts test-e2e --config ./jest.config.js

 PASS  __tests__/specs/index.test.js
  Hello World
    ✓ Should load properly (1645ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.169s, estimated 4s

Why?
I think the page.$x() methods return an array.
So the title variable should not be null.

Debug code / result

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const title = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( title ).not.toBeNull();
	} );
} );
> wp-scripts test-e2e --config ./jest.config.js

 PASS  __tests__/specs/index.test.js
  Hello World
    ✓ Should load properly (1645ms)

  console.log __tests__/specs/index.test.js:9
    []

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.169s, estimated 4s
Ran all test suites.

Possible Solution

We should change the assertion.
The page.$x methods return an array, so we need to check the array length.

-> Should fail the test

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const nodes = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( nodes.length ).not.toEqual( 0 )
	} );
} );

-> Should pass the test

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const nod[]es = await page.$x(
			'//h2[contains(text(), "Welcome to WordPress!")]'
		);
		expect( nodes.length ).not.toEqual( 0 )
	} );
} );

Attachments (1)

47622.patch (585 bytes) - added by hideokamoto 6 months ago.
The patch for the ticket

Download all attachments as: .zip

Change History (4)

@hideokamoto
6 months ago

The patch for the ticket

#1 @SergeyBiryukov
6 months ago

  • Milestone changed from Awaiting Review to 5.3

#2 @youknowriad
6 months ago

  • Keywords commit added

Tested the patch and it works great. Thanks for the fix.

I wonder if we have similar assertions in Gutenberg tests :)

#3 @youknowriad
6 months ago

  • Owner set to youknowriad
  • Resolution set to fixed
  • Status changed from new to closed

In 45575:

Build/Test Tools: Fix the hello e2e test assertion.

The previous assertion was always valid because we assumed it returned a single object,
while in reality it was returning an array.

Props hideokamoto.
Fixes #47622.

Note: See TracTickets for help on using tickets.