Run core unit tests from src again, again

[johnbillion]

Previously: #45863

When a developer first runs the test suite locally they are prompted to create a wp-tests-config.php file based on the wp-tests-config-sample.php file. This file defines the test directory for the core tests as build rather than src.

Running the core tests from build results in several problems:

1. Running a build is slow. It copies all the files and builds, validates, and minifies all the CSS and JS. None of this should be necessary for PHP testing.
2. A developer iterating on a patch in the source file has no way of knowing that their file is not actually being tested when running the tests, unless they run the build each time or start and run the file watcher. This is an easy step to forget.
3. PHP errors display a stack trace from build instead src.
4. Breakpoint debugging isn't fun as it also uses the stack trace from build instead of src.
5. Any test which requires a build step is not a unit test and should be tested using other means, for example as a validation step during the build process.

Current Status

There are two tests that do not pass when running the test suite from the src directory.

The first is an error:

1) Tests_Dependencies_Scripts::test_no_source_mapping
UnexpectedValueException: RecursiveDirectoryIterator::__construct(wp/build/): failed to open dir: No such file or directory

wp/tests/phpunit/tests/dependencies/scripts.php:1406

The second is a test failure:

1) Tests_Dependencies_Scripts::test_wp_add_inline_script_before_after_concat_with_core_dependency
Failed asserting that two strings are identical.

[snip]

wp/tests/phpunit/includes/abstract-testcase.php:677
wp/tests/phpunit/tests/dependencies/scripts.php:755

Tasks

1. For Tests_Dependencies_Scripts::test_no_source_mapping, find a more appropriate way to assert that built JS files do not contain a source map and remove this PHP unit test.
2. For Tests_Dependencies_Scripts::test_wp_add_inline_script_before_after_concat_with_core_dependency (ticket #36392), fix the fact that this test expects .min.js in its output rather than using the return value of wp_scripts_get_suffix().
3. Increase the reliability of tests which can mess up files if an assertion fails (tickets to follow).
4. Change the default value of ABSPATH back to src in wp-config-sample.php.
5. Investigate whether Travis and/or GitHub Actions should run their tests from src or build.

[johnbillion]

#51735

https://core.trac.wordpress.org/ticket/51734

[johnbillion]

Update: There's actually many more errors and failures on a fresh checkout because the wp-includes/js directory gets tested but it's part of the build process.

​https://github.com/WordPress/wordpress-develop/pull/717/checks?check_run_id=1376708592

[peterwilsoncc]

Tests_Embed_Template::test_js_no_ampersands_in_compiled() is a special case in which a built file is tested against in the unit tests, the /build/ directory is hard-coded and it fails silently if the file doesn't exist.

Please be sure to allow for this in any refactoring.

[johnbillion]

In 49601:

Build/Test Tools: Improve the reliability of the test that ensures correct dependency order when inline scripts are used.

This test already ensures WP_Scripts->do_concat is true, therefore it has no dependency on SCRIPT_DEBUG being false. This means the test can run in an environment where the .min suffix is not used.

This change allows for the test to pass in this situation.

See #36392, #51734, #51344

[iandunn]

Related: r48847 updates phpunit.xml.dist to track code coverage against the build/ folder.

[gziolo]

Yes, I would be in favor of that change. JS unit tests in Gutenberg always run from sources to avoid the need to build each time you want to test something. It was a common mistake that someone forgot to run the build and used stale files when testing code.

[johnbillion]

Current status of running the tests from src without any built files in place: 30 errors, all due to missing files, and 2 failures.

There were 30 errors:

1) Tests_Dependencies_jQuery::test_presence_of_jquery_no_conflict
file_get_contents(/var/www/src/wp-includes/js/jquery/jquery.js): failed to open stream: No such file or directory

/var/www/tests/phpunit/tests/dependencies/jquery.php:43

2) Tests_Dependencies_Scripts::test_no_source_mapping
UnexpectedValueException: RecursiveDirectoryIterator::__construct(/var/www/build/): failed to open dir: No such file or directory

/var/www/tests/phpunit/tests/dependencies/scripts.php:1421

3) Tests_Formatting_Emoji::test_unfiltered_emoji_cdns
readfile(/var/www/src/wp-includes/js/wp-emoji-loader.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/formatting.php:5719
/var/www/tests/phpunit/includes/utils.php:353
/var/www/tests/phpunit/tests/formatting/Emoji.php:16

4) Tests_Formatting_Emoji::test_filtered_emoji_svn_cdn
readfile(/var/www/src/wp-includes/js/wp-emoji-loader.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/formatting.php:5719
/var/www/tests/phpunit/includes/utils.php:353
/var/www/tests/phpunit/tests/formatting/Emoji.php:34

5) Tests_Formatting_Emoji::test_filtered_emoji_png_cdn
readfile(/var/www/src/wp-includes/js/wp-emoji-loader.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/formatting.php:5719
/var/www/tests/phpunit/includes/utils.php:353
/var/www/tests/phpunit/tests/formatting/Emoji.php:55

6) Test_oEmbed_Controller::test_request_invalid_format
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:298

7) Test_oEmbed_Controller::test_request_json
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:322

8) Test_oEmbed_Controller::test_request_static_front_page
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:365

9) Test_oEmbed_Controller::test_request_xml
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:410

10) Test_oEmbed_Controller::test_rest_pre_serve_request
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:479

11) Test_oEmbed_Controller::test_rest_pre_serve_request_wrong_format
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:493

12) Test_oEmbed_Controller::test_rest_pre_serve_request_wrong_method
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:505

13) Test_oEmbed_Controller::test_proxy_with_internal_url
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/embed.php:647
/var/www/src/wp-includes/class-wp-oembed-controller.php:188
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:695

14) Test_oEmbed_Controller::test_proxy_with_static_front_page_url
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/embed.php:647
/var/www/src/wp-includes/class-wp-oembed-controller.php:188
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/oembed/controller.php:742

15) Tests_oEmbed_Response_Data::test_get_oembed_response_data
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:19

16) Tests_oEmbed_Response_Data::test_get_oembed_response_data_author
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:55

17) Tests_oEmbed_Response_Data::test_get_oembed_response_data_with_public_true_custom_post_status
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:145

18) Tests_oEmbed_Response_Data::test_get_oembed_response_data_maxwidth_too_high
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:180

19) Tests_oEmbed_Response_Data::test_get_oembed_response_data_maxwidth_too_low
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:189

20) Tests_oEmbed_Response_Data::test_get_oembed_response_data_maxwidth_invalid
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:198

21) Tests_oEmbed_Response_Data::test_get_oembed_response_data_with_thumbnail
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:221

22) Tests_oEmbed_Response_Data::test_get_oembed_response_data_for_attachment
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/tests/phpunit/tests/oembed/getResponseData.php:240

23) Tests_Embed_Template::test_oembed_output_post
readfile(/var/www/src/wp-includes/js/wp-emoji-loader.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/formatting.php:5719
/var/www/src/wp-includes/formatting.php:5656
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/class-wp-hook.php:316
/var/www/src/wp-includes/plugin.php:484
/var/www/src/wp-includes/theme-compat/header-embed.php:29
/var/www/src/wp-includes/template.php:730
/var/www/src/wp-includes/template.php:676
/var/www/src/wp-includes/general-template.php:48
/var/www/src/wp-includes/theme-compat/embed.php:13
/var/www/tests/phpunit/tests/oembed/template.php:27

24) Tests_Embed_Template::test_oembed_output_post_with_thumbnail
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:1182
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/class-wp-hook.php:316
/var/www/src/wp-includes/plugin.php:484
/var/www/src/wp-includes/theme-compat/footer-embed.php:18
/var/www/src/wp-includes/template.php:730
/var/www/src/wp-includes/template.php:676
/var/www/src/wp-includes/general-template.php:92
/var/www/src/wp-includes/theme-compat/embed.php:24
/var/www/tests/phpunit/tests/oembed/template.php:59

25) Tests_Embed_Template::test_get_post_embed_html
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/tests/phpunit/tests/oembed/template.php:283

26) Tests_Embed_Template::test_js_no_ampersands
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/tests/phpunit/tests/oembed/template.php:296

27) Tests_WP_oEmbed::test_wp_filter_pre_oembed_result_prevents_http_request_for_internal_permalinks
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/embed.php:647
/var/www/src/wp-includes/embed.php:1239
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/class-wp-oembed.php:391
/var/www/tests/phpunit/tests/oembed/wpOembed.php:36

28) Tests_WP_oEmbed::test_wp_filter_pre_oembed_result_prevents_http_request_when_viewing_the_post
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/embed.php:647
/var/www/src/wp-includes/embed.php:1239
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/class-wp-oembed.php:391
/var/www/tests/phpunit/tests/oembed/wpOembed.php:51

29) WP_Test_REST_Schema_Initialization::test_build_wp_api_client_fixtures
file_get_contents(/var/www/src/wp-includes/js/wp-embed.js): failed to open stream: No such file or directory

/var/www/src/wp-includes/embed.php:458
/var/www/src/wp-includes/embed.php:672
/var/www/src/wp-includes/class-wp-hook.php:292
/var/www/src/wp-includes/plugin.php:212
/var/www/src/wp-includes/embed.php:577
/var/www/src/wp-includes/class-wp-oembed-controller.php:131
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:1139
/var/www/src/wp-includes/rest-api/class-wp-rest-server.php:985
/var/www/tests/phpunit/includes/spy-rest-server.php:57
/var/www/tests/phpunit/tests/rest-api/rest-schema-setup.php:452

30) Tests_Shortcode::test_php_and_js_shortcode_attribute_regexes_match
file_get_contents(/var/www/src/wp-includes/js/shortcode.js): failed to open stream: No such file or directory

/var/www/tests/phpunit/tests/shortcode.php:749

--

There were 2 failures:

1) Tests_Dependencies_MediaElementjs::test_exclusion_of_flash
Failed asserting that 0 is greater than 0.

/var/www/tests/phpunit/tests/dependencies/mediaelementjs.php:28

2) Tests_Dependencies_Scripts::test_wp_add_inline_script_before_after_concat_with_core_dependency
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
 <script type='text/javascript' src='http://example.com' id='test-example-js'></script>
 <script type='text/javascript' src='/wp-includes/js/dist/vendor/wp-polyfill.js' id='wp-polyfill-js'></script>
 <script type='text/javascript' id='wp-polyfill-js-after'>
-( 'fetch' in window ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-fetch.js"></scr' + 'ipt>' );( document.contains ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-node-contains.js"></scr' + 'ipt>' );( window.DOMRect ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-dom-rect.js"></scr' + 'ipt>' );( window.URL && window.URL.prototype && window.URLSearchParams ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-url.js"></scr' + 'ipt>' );( window.FormData && window.FormData.prototype.keys ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-formdata.js"></scr' + 'ipt>' );( Element.prototype.matches && Element.prototype.closest ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-element-closest.js"></scr' + 'ipt>' );( 'objectFit' in document.documentElement.style ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-object-fit.min.js"></scr' + 'ipt>' );
+( 'fetch' in window ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-fetch.js"></scr' + 'ipt>' );( document.contains ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-node-contains.js"></scr' + 'ipt>' );( window.DOMRect ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-dom-rect.js"></scr' + 'ipt>' );( window.URL && window.URL.prototype && window.URLSearchParams ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-url.js"></scr' + 'ipt>' );( window.FormData && window.FormData.prototype.keys ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-formdata.js"></scr' + 'ipt>' );( Element.prototype.matches && Element.prototype.closest ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-element-closest.js"></scr' + 'ipt>' );( 'objectFit' in document.documentElement.style ) || document.write( '<script src="http://example.org/wp-includes/js/dist/vendor/wp-polyfill-object-fit.js"></scr' + 'ipt>' );
 </script>
 <script type='text/javascript' src='/wp-includes/js/dist/dom-ready.js' id='wp-dom-ready-js'></script>
 <script type='text/javascript' src='/wp-includes/js/dist/hooks.js' id='wp-hooks-js'></script>

/var/www/tests/phpunit/includes/abstract-testcase.php:677
/var/www/tests/phpunit/tests/dependencies/scripts.php:770

[johnbillion]

I'm working through fixing these in the PR at ​https://github.com/WordPress/wordpress-develop/pull/717

[johnbillion]

@peterwilsoncc I've been thinking about where the most appropriate place for the wp-embed.min.js no-ampersand assertion is. It's not a unit test, and it's not a JavaScript test, it's actually an assertion that should be performed against the build process, so I've added a Grunt task that runs as part of the build task. It verifies that the file exists and does not contain an ampersand. It uses assert() so the build fails if it does contain one. I pushed to ​https://github.com/WordPress/wordpress-develop/pull/717 if you'd like to take a look.

I'm going to add assertions for the other two tests that rely on the build directory.

[gziolo]

I'm going to add assertions for the other two tests that rely on the build directory.

That's a good idea. I guess there are more assertions we could add in the future, like whether the assets file for WordPress packages consumed from npm was generated.

[johnbillion]

PR is ready for review

[johnbillion]

cc @atimmer @netweb

[prbot]

​desrosj commented on ​PR #717:

Whoops! I put requested changes, but they are more suggestions and questions!

Performance testing npm installation without a cache. See #717.

[prbot]

​johnbillion commented on ​PR #717:

Feedback addressed. I've changed the code coverage reports to run against src too because there's no functional difference now the build-specific PHPUnit tests have been converted into JS tests and build assertions.

[johnbillion]

In 50441:

Build/Test Tools: Switch back to running the PHPUnit test suite against the src directory instead of build.

Some PHPUnit tests were concerned with the state of files in the build directory. In order to allow the tests to run without requiring a build to be run first, these have been moved into assertions that run after the build step (and therefore cause it to fail if they do not pass), or into QUnit tests as necessary.

Various other PHPUnit tests implictly depend on built JavaScript files being present. These files are now touched during the test setup to avoid PHP warnings if the build files are not present.

The wp-tests-config-sample.php file and the GitHub Actions configuration have also been changed so ABSPATH uses src instead of build, therefore allowing the PHPUnit tests to be run without a build having to be run first. This means all new local installations of WordPress will use src for PHPUnit testing. If you would like to switch your existing installation over then change the location of ABSPATH in wp-tests-config.php to point to src instead of build.

Props peterwilsoncc, iandunn, gziolo, desroj, johnbillion

Fixes #51734
See #45863

[prbot]

​johnbillion commented on ​PR #717:

https://core.trac.wordpress.org/changeset/50441

[desrosj]

[50441] looks good, but there's one issue we did not spot in the PR.

It looks like the ​test reporting job is failing. Since WordPress is no longer prepared in a different job, the Git repo no longer needs to be re-established. This should be easy to fix by ​removing the following step.

See [49786] for more details on why this step is there in the first place.

I'm also going to mark this as fixed-major because I'd like to attempt to backport it to all branches receiving updates. When backporting #50401, I tried to keep all workflow files as consistent as possible to make backporting improvements to testing setups as easy as possible.

[johnbillion]

In 50442:

Build/Test Tools: Avoid attempting to redeclare the origin remote prior to sending the test results to the test reporting API.

Props desroj
See #51734

[desrosj]

[50441-50442] look good for backporting.

https://core.trac.wordpress.org/ticket/51734

In [50441] the tests were switched to run against src by default instead of build.

This introduced one issue, the test_php_and_js_shortcode_attribute_regexes_match() now fails when run against build because it assumes the js/_enqueues/wp/shortcode.js file exists, which it does not in build. Running the tests against build is still possible so we should fix this, especially as existing contributors will see this unless (or until) they switch to using src for their tests.

## To test:

1. Delete the build directory if you have one
2. Change ABSPATH in wp-tests-config.php to point to src
3. Run npm run env:restart
4. Run npm run test:php -- --filter test_php_and_js_shortcode_attribute_regexes_match and confirm it passes
5. Change ABSPATH in wp-tests-config.php to point to build
6. Run npm run env:restart
7. Run npm run build
8. Run npm run test:php -- --filter test_php_and_js_shortcode_attribute_regexes_match and confirm it passes

[johnbillion]

In 50464:

Build/Test Tools: PHPUnit test improvements:

• Switch back to running the PHPUnit test suite against the src directory instead of build
• Increase the reliability of backing up the mu-plugins directory during tests
• Split the PHPUnit tests for PHP versions below 7.0 in half, allowing them to run in parallel and reduce the overall test run duration on GitHub Actions

Merges [50441-50444] to the 5.7 branch.

Fixes #51734, #51735, #52645

[johnbillion]

In 50501:

Build/Test Tools: Ensure the test_php_and_js_shortcode_attribute_regexes_match() test can run when testing using either the src or build directory.

Props alexstine, desrosj

See #51734

[SergeyBiryukov]

@johnbillion Should [50501] be backported to the 5.7 branch too?

[prbot]

​johnbillion commented on ​PR #1061:

https://core.trac.wordpress.org/changeset/50501

[johnbillion]

Yep let's backport this too.

[desrosj]

Looks good to backport.

[desrosj]

In 50509:

Build/Test Tools: Ensure the test_php_and_js_shortcode_attribute_regexes_match() test can run when testing using either the src or build directory.

Reviewed by desrosj, johnbillion.
Props alexstine, desrosj.
Merges [50501] to the 5.7 branch.
Fixes #51734.

Trac ticket: https://core.trac.wordpress.org/ticket/51734

[jrf]

So... this broke the code coverage reporting as it seems it was forgotten to update the code coverage configuration in the phpunit.xml.dist file.

Last commit on which code coverage was run is commit [50436]
See: ​https://app.codecov.io/gh/WordPress/wordpress-develop/

It can also been seen in the GH Actions code coverage runs which report:

==> Reading reports
    - file not found at wp-code-coverage-single-clover-cded2eb52aa71a398d1d0cd87f8674b383512c0b.xml
--> No coverage data found.

I'll upload the patch which should fix this in a moment and which basically reverts [48847].
Note: this patch will also need to be backported.

I'll also upload a second patch (optional) which would (should) allow for manually triggering code coverage runs. Committing that patch would allow us, in the future, to test that specific build changes do not break the code coverage build by manually triggering a build on a branch/PR.

[johnbillion]

Good spot, thanks. Moved this to #52786.

[desrosj]

In 50592:

Build/Test Tools: Fix code coverage reporting to generate report from src.

Since [50441-50442] switched the test workflows to run from src instead of build, code coverage reporting has stopped working. This was caused by the code coverage configuration continuing to reference build. This corrects the configuration so coverage reporting can resume.

This change also introduces the workflow_dispatch event to the workflow, which will allow committers to manually run the workflow when desired. For example, to confirm changes to the test suite do not break reporting.

This also adds the phpunit.xml.dist and tests/phpunit/multisite.xml files to the paths list. Since these files are responsible for configuring the test suite and code coverage reporting, any changes to them should verify that no problems were introduced.

Props jrf, johnbillion.
Fixes #52786. See #51734.

[prbot]

​jrfnl commented on ​PR #1092:

Closing as merged.

[desrosj]

In 50601:

Build/Test Tools: Backport several build tool fixes & improvements.

This backports a handful of test fixes related to generating code coverage reporting, as well as some devDependency updates.

Backports [50591-50595,50597-50598] to the 5.7 branch.
See #52624, #52786, #51734.

[desrosj]

In 50602:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.6 branch.

This backports several build and test tool improvements to the 5.6 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.6 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50603:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.5 branch.

This backports several build and test tool improvements to the 5.5 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.5 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50604:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.4 branch.

This backports several build and test tool improvements to the 5.4 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.4 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50605:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.3 branch.

This backports several build and test tool improvements to the 5.3 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.3 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50606:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.2 branch.

This backports several build and test tool improvements to the 5.2 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.2 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.