Make WordPress Core

Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#52645 closed task (blessed) (fixed)

Split slow tests into their own job on PHP 5.6 and run them in parallel

Reported by: johnbillion's profile johnbillion Owned by: johnbillion's profile johnbillion
Milestone: 5.7 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch commit dev-reviewed
Focuses: Cc:

Description

The test suite takes ~16 minutes to run on GitHub Actions, which was recently reduced from ~26 minutes in #52548. The approach taken was to split up the jobs so each one runs faster, and then run them in parallel so the overall duration is reduced.

The PHPUnit tests on PHP 5.6 are the slowest jobs by far, taking ~12 minutes compared to 5 to 7 minutes for all other PHP versions. If we can reduce the time that the PHP 5.6 runs take then we reduce the overall duration.

I propose that for the PHP 5.6 runs specifically, we split the tests in half by selecting the slowest test groups whose combined time is roughly half the time of the whole test run, running them in a separate job, and excluding them from the main job. As the jobs will run in parallel this will reduce the overall duration.

After a bunch of testing I've determined that the external-http, media, and restapi groups combined take roughly the same time as all the other tests combined.

  • It's known that the external-http tests are slow which is why they are run separately by default
  • The media tests are especially slow on PHP 5.6
  • The restapi test group contains a large number of tests and therefore consumes a large proportion of the total duration

Running these groups separately reduces the overall duration by around 4 minutes.

Combining this group separation with running the tests from src instead of build means the overall duration dips below 9 minutes.

So, why not do the same for PHP versions other than 5.6? The answer is that for all other PHP versions there is no net gain because the time saved by splitting the tests in half is less than the time taken for the job to be set up and run. Only with PHP 5.6 does the time saving outweigh the time taken to set up the job.

Change History (29)

This ticket was mentioned in PR #1030 on WordPress/wordpress-develop by johnbillion.


3 years ago
#1

  • Keywords has-patch added; needs-patch removed

johnbillion commented on PR #1030:


3 years ago
#2

Feedback addressed

#3 @johnbillion
3 years ago

  • Milestone changed from Awaiting Review to 5.8

#4 @johnbillion
3 years ago

  • Owner set to johnbillion
  • Resolution set to fixed
  • Status changed from new to closed

In 50444:

Build/Test Tools: Split the PHPUnit tests for PHP versions below 7.0 in half, allowing them to run in parallel and reduce the overall test run duration on GitHub Actions.

The external-http, media, and restapi groups combined take roughly the same time as all the other tests combined. Splitting the test run in two speeds up the PHP 5.6 test runs which are the slowest by quite a margin. The net reduction in duration is only seen on PHP versions older than 7.0 where the reduction is greater than the time taken for the job to be set up.

Props peterwilsoncc, desrosj, johnbillion

Special shout out to manzoorwanijk whose article about running steps in parallel started me down this path.

Fixes #52645

#5 @johnbillion
3 years ago

  • Keywords fixed-major added
  • Resolution fixed deleted
  • Status changed from closed to reopened

#7 @desrosj
3 years ago

  • Keywords commit dev-reviewed added; fixed-major removed
  • Milestone changed from 5.8 to 5.7

Let's include this with the [50441-50442] backports.

This ticket was mentioned in Slack in #core by hellofromtonya. View the logs.


3 years ago

#9 @johnbillion
3 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 50464:

Build/Test Tools: PHPUnit test improvements:

  • Switch back to running the PHPUnit test suite against the src directory instead of build
  • Increase the reliability of backing up the mu-plugins directory during tests
  • Split the PHPUnit tests for PHP versions below 7.0 in half, allowing them to run in parallel and reduce the overall test run duration on GitHub Actions

Merges [50441-50444] to the 5.7 branch.

Fixes #51734, #51735, #52645

#10 @desrosj
3 years ago

In 50602:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.6 branch.

This backports several build and test tool improvements to the 5.6 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.6 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

#11 @desrosj
3 years ago

In 50603:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.5 branch.

This backports several build and test tool improvements to the 5.5 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.5 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

#12 @desrosj
3 years ago

In 50604:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.4 branch.

This backports several build and test tool improvements to the 5.4 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.4 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#13 @desrosj
3 years ago

In 50605:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.3 branch.

This backports several build and test tool improvements to the 5.3 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.3 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#14 @desrosj
3 years ago

In 50606:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.2 branch.

This backports several build and test tool improvements to the 5.2 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.2 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#15 @desrosj
3 years ago

In 50622:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.1 branch.

This backports several build and test tool improvements to the 5.1 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.1 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#16 @desrosj
3 years ago

In 50624:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.0 branch.

This backports several build and test tool improvements to the 5.0 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.0 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#17 @desrosj
3 years ago

In 50625:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.9 branch.

This backports several build and test tool improvements to the 4.9 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.9 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#18 @desrosj
3 years ago

In 50626:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.7 branch.

This backports several build and test tool improvements to the 3.7 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 3.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#19 @desrosj
3 years ago

In 50635:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.8 branch.

This backports several build and test tool improvements to the 4.8 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.8 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#20 @desrosj
3 years ago

In 50636:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.7 branch.

This backports several build and test tool improvements to the 4.7 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#21 @desrosj
3 years ago

In 50637:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.6 branch.

This backports several build and test tool improvements to the 4.6 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.6 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#22 @desrosj
3 years ago

In 50638:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.5 branch.

This backports several build and test tool improvements to the 4.5 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.5 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#23 @desrosj
3 years ago

In 50639:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.4 branch.

This backports several build and test tool improvements to the 4.4 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.4 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#24 @desrosj
3 years ago

In 50640:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.3 branch.

This backports several build and test tool improvements to the 4.3 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.3 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#25 @desrosj
3 years ago

In 50642:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.2 branch.

This backports several build and test tool improvements to the 4.2 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.2 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#26 @desrosj
3 years ago

In 50643:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.1 branch.

This backports several build and test tool improvements to the 4.1 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.1 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#27 @desrosj
3 years ago

In 50644:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.0 branch.

This backports several build and test tool improvements to the 4.0 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.0 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#28 @desrosj
3 years ago

In 50645:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.9 branch.

This backports several build and test tool improvements to the 3.9 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.9 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#29 @desrosj
3 years ago

In 50646:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.8 branch.

This backports several build and test tool improvements to the 3.8 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.8 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

Note: See TracTickets for help on using tickets.