Report if populate_network() fails during test installation

[TimothyBlynJacobs]

While looking into a ​Gutenberg phpunit issue we discovered that the root cause of tests failing with <h1>Error establishing a database connection</h1> was that populate_network() was failing because of a missing WP_TESTS_EMAIL.

I think it'd be helpful if we reported to the user if populate_network() returned a WP_Error instance

Trac ticket: https://core.trac.wordpress.org/ticket/50251

[jorbin]

In 47904:

Build/Test: Die with an error if populate_network fails

If you are missing WP_TESTS_EMAIL, populate_network will fail and it can be hard to debug. As populate_network can return a wp_error object, we can detect that and display the error to a user.

See: ​https://github.com/WordPress/gutenberg/pull/22613
Fixes: #50251
Props: TimothyBlynJacobs

[azaozz]

[47904] would prevent the tests from running when WP_TESTS_DOMAIN or WP_TESTS_EMAIL are not defined, however this points to bigger/other problems. WP_TESTS_DOMAIN and WP_TESTS_EMAIL are defined in https://core.trac.wordpress.org/browser/tags/5.4.1/wp-tests-config-sample.php#L68 and should always be present.

As far as I see the actual problem here is that wp-tests-config.php is not present (loaded), or that not all of the constants (settings) there are replaced when that file is not used. Perhaps it would be better to validate that all needed constants are present while doing the bootstrap instead of attempting to run and failing.

This gets a bit trickier as phpunit/includes/install.php is run in another PHP process from cli, see https://core.trac.wordpress.org/browser/tags/5.4.1/tests/phpunit/includes/bootstrap.php#L100.

[TimothyBlynJacobs]

that not all of the constants (settings) there are replaced when that file is not used.

Our issue was that not all of the constants were loaded, and tracking this down was incredibly difficult because the eventual error message that gets generated is seemingly unrelated.

Perhaps it would be better to validate that all needed constants are present while doing the bootstrap instead of attempting to run and failing.

I think checking that all the constants are defined as expected would be great. I think the populate_network check is still valuable as well because the checks there are more intricate than just checking for a non-empty value.

[azaozz]

Replying to TimothyBlynJacobs:

I think the populate_network check is still valuable as well

Yes, definitely. Just thinking it would be good to get to the "absolute bottom" of this and fix it once and for all :)

[SergeyBiryukov]

In 48592:

Build/Test Tools: Check if all the required constants are defined before running the test suite.

Follow-up to [47904].

Props azaozz, TimothyBlynJacobs, SergeyBiryukov.
Fixes #50251.