Split slow tests into their own job on PHP 5.6 and run them in parallel

[johnbillion]

The test suite takes ~16 minutes to run on GitHub Actions, which was recently reduced from ~26 minutes in #52548. The approach taken was to split up the jobs so each one runs faster, and then run them in parallel so the overall duration is reduced.

The PHPUnit tests on PHP 5.6 are the slowest jobs by far, taking ~12 minutes compared to 5 to 7 minutes for all other PHP versions. If we can reduce the time that the PHP 5.6 runs take then we reduce the overall duration.

I propose that for the PHP 5.6 runs specifically, we split the tests in half by selecting the slowest test groups whose combined time is roughly half the time of the whole test run, running them in a separate job, and excluding them from the main job. As the jobs will run in parallel this will reduce the overall duration.

After ​a bunch of testing I've determined that the external-http, media, and restapi groups combined take roughly the same time as all the other tests combined.

• It's known that the external-http tests are slow which is why they are run separately by default
• The media tests are especially slow on PHP 5.6
• The restapi test group contains a large number of tests and therefore consumes a large proportion of the total duration

Running these groups separately ​reduces the overall duration by around 4 minutes.

Combining this group separation with running the tests from src instead of build means ​the overall duration dips below 9 minutes.

So, why not do the same for PHP versions other than 5.6? The answer is that for all other PHP versions there is no net gain because the time saved by splitting the tests in half is less than the time taken for the job to be set up and run. Only with PHP 5.6 does the time saving outweigh the time taken to set up the job.

https://core.trac.wordpress.org/ticket/52645

Feedback addressed

[johnbillion]

In 50444:

Build/Test Tools: Split the PHPUnit tests for PHP versions below 7.0 in half, allowing them to run in parallel and reduce the overall test run duration on GitHub Actions.

The external-http, media, and restapi groups combined take roughly the same time as all the other tests combined. Splitting the test run in two speeds up the PHP 5.6 test runs which are the slowest by quite a margin. The net reduction in duration is only seen on PHP versions older than 7.0 where the reduction is greater than the time taken for the job to be set up.

Props peterwilsoncc, desrosj, johnbillion

Special shout out to manzoorwanijk whose article about running steps in parallel started me down this path.

Fixes #52645

https://core.trac.wordpress.org/changeset/50444

[desrosj]

Let's include this with the [50441-50442] backports.

[johnbillion]

In 50464:

Build/Test Tools: PHPUnit test improvements:

• Switch back to running the PHPUnit test suite against the src directory instead of build
• Increase the reliability of backing up the mu-plugins directory during tests
• Split the PHPUnit tests for PHP versions below 7.0 in half, allowing them to run in parallel and reduce the overall test run duration on GitHub Actions

Merges [50441-50444] to the 5.7 branch.

Fixes #51734, #51735, #52645

[desrosj]

In 50602:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.6 branch.

This backports several build and test tool improvements to the 5.6 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.6 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50603:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.5 branch.

This backports several build and test tool improvements to the 5.5 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.5 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50604:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.4 branch.

This backports several build and test tool improvements to the 5.4 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.4 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50605:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.3 branch.

This backports several build and test tool improvements to the 5.3 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.3 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50606:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.2 branch.

This backports several build and test tool improvements to the 5.2 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.2 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50622:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.1 branch.

This backports several build and test tool improvements to the 5.1 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.1 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50624:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.0 branch.

This backports several build and test tool improvements to the 5.0 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.0 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50625:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.9 branch.

This backports several build and test tool improvements to the 4.9 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.9 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50626:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.7 branch.

This backports several build and test tool improvements to the 3.7 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 3.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50635:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.8 branch.

This backports several build and test tool improvements to the 4.8 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.8 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50636:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.7 branch.

This backports several build and test tool improvements to the 4.7 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50637:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.6 branch.

This backports several build and test tool improvements to the 4.6 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.6 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50638:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.5 branch.

This backports several build and test tool improvements to the 4.5 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.5 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50639:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.4 branch.

This backports several build and test tool improvements to the 4.4 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.4 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50640:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.3 branch.

This backports several build and test tool improvements to the 4.3 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.3 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50642:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.2 branch.

This backports several build and test tool improvements to the 4.2 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.2 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50643:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.1 branch.

This backports several build and test tool improvements to the 4.1 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.1 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50644:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.0 branch.

This backports several build and test tool improvements to the 4.0 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.0 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50645:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.9 branch.

This backports several build and test tool improvements to the 3.9 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.9 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50646:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.8 branch.

This backports several build and test tool improvements to the 3.8 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.8 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.