The E2E test has a wrong assertion

[hideokamoto]

The E2E test succeeds even if we enter whatever value
​https://github.com/WordPress/wordpress-develop/blob/master/tests/e2e/specs/hello.test.js#L4-L10

Expect behavior

The test should fail when we put invalid text condition.
And I tried to run the following test code, the E2E test has been passed.

Code

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const title = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( title ).not.toBeNull();
	} );
} );

Result

> wp-scripts test-e2e --config ./jest.config.js

 PASS  __tests__/specs/index.test.js
  Hello World
    ✓ Should load properly (1645ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.169s, estimated 4s

Why?
I think the page.$x() methods return an array.
So the title variable should not be null.

Debug code / result

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const title = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( title ).not.toBeNull();
	} );
} );

> wp-scripts test-e2e --config ./jest.config.js

 PASS  __tests__/specs/index.test.js
  Hello World
    ✓ Should load properly (1645ms)

  console.log __tests__/specs/index.test.js:9
    []

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.169s, estimated 4s
Ran all test suites.

Possible Solution

We should change the assertion.
The page.$x methods return an array, so we need to check the array length.

-> Should fail the test

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const nodes = await page.$x(
			'//h2[contains(text(), "aaaaaaaaaaa")]'
		);
		expect( nodes.length ).not.toEqual( 0 )
	} );
} );

-> Should pass the test

const { visitAdminPage } = require('@wordpress/e2e-test-utils');

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const nod[]es = await page.$x(
			'//h2[contains(text(), "Welcome to WordPress!")]'
		);
		expect( nodes.length ).not.toEqual( 0 )
	} );
} );

[hideokamoto]

The patch for the ticket

[youknowriad]

Tested the patch and it works great. Thanks for the fix.

I wonder if we have similar assertions in Gutenberg tests :)

[youknowriad]

In 45575:

Build/Test Tools: Fix the hello e2e test assertion.

The previous assertion was always valid because we assumed it returned a single object,
while in reality it was returning an array.

Props hideokamoto.
Fixes #47622.