Make WordPress Core

Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#52548 closed task (blessed) (fixed)

Run Multisite tests in parallel during CI

Reported by: johnbillion's profile johnbillion Owned by: johnbillion's profile johnbillion
Milestone: 5.7 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch commit
Focuses: Cc:

Description

Related: #50401

The PHPUnit test workflow takes around 26 minutes (±2 minutes) to run on GitHub Actions. Each job within each run runs the whole test suite, with a total of 8 jobs (7 distinct PHP versions plus an extra job for 7.4 with memcached).

We should take more advantage of GitHub Actions' parallel processing of jobs within a workflow. If we reduce the times for each job by spreading their work across multiple jobs and all jobs still run in parallel, we reduce the overall workflow time as long as the time saving is greater than the time taken to set up each job (download the artefact, install dependencies, and start the containers).

One way to do this is by splitting the Multisite tests into their own jobs and adding a multisite flag to the matrix. This very roughly halves the time for the actual PHPUnit tests in each job, and in practice reduces the total duration of the workflow from ~26 minutes to ~16 minutes.

Items to consider before this can be implemented:

  • How to correctly report the results to the WordPress.org host test results collector

Change History (28)

This ticket was mentioned in PR #1009 on WordPress/wordpress-develop by johnbillion.


3 years ago
#1

  • Keywords has-patch added

desrosj commented on PR #1009:


3 years ago
#2

Also, it's probably also worthwhile to copy this idea over to the test coverage workflow as well.

johnbillion commented on PR #1009:


3 years ago
#3

Opened https://core.trac.wordpress.org/ticket/52561 for the coverage report.

Just one remaining item for this change then - will the hosting test result reporter handle two submissions (one for single site and one for Multisite) ok? Should this be a separate ticket to prevent it from blocking this change?

johnbillion commented on PR #1009:


3 years ago
#4

For now I set multisite to false for the report job. I'll open a separate ticket for that too.

I think this is ready to go. Anything else from you @desrosj ?

#5 @johnbillion
3 years ago

  • Keywords commit added
  • Milestone changed from Awaiting Review to 5.7

#6 @johnbillion
3 years ago

  • Owner set to johnbillion
  • Resolution set to fixed
  • Status changed from new to closed

In 50379:

Build/Test Tools: Split the single site and Multisite PHPUnit test runs on CI into separate jobs so they run in parallel.

This reduces the total duration of the test suite from ~26 minutes to ~16 minutes. Hurrah!

Fixes #52548

This ticket was mentioned in Slack in #core by sergey. View the logs.


3 years ago

#9 @desrosj
3 years ago

In 50602:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.6 branch.

This backports several build and test tool improvements to the 5.6 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.6 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

#10 @desrosj
3 years ago

In 50603:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.5 branch.

This backports several build and test tool improvements to the 5.5 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.5 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

#11 @desrosj
3 years ago

In 50604:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.4 branch.

This backports several build and test tool improvements to the 5.4 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.4 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#12 @desrosj
3 years ago

In 50605:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.3 branch.

This backports several build and test tool improvements to the 5.3 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.3 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#13 @desrosj
3 years ago

In 50606:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.2 branch.

This backports several build and test tool improvements to the 5.2 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • The ability to run PHPUnit tests from src instead of build [50441-50443].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.2 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#14 @desrosj
3 years ago

In 50622:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.1 branch.

This backports several build and test tool improvements to the 5.1 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.1 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#15 @desrosj
3 years ago

In 50624:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.0 branch.

This backports several build and test tool improvements to the 5.0 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.0 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#16 @desrosj
3 years ago

In 50625:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.9 branch.

This backports several build and test tool improvements to the 4.9 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.9 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#17 @desrosj
3 years ago

In 50626:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.7 branch.

This backports several build and test tool improvements to the 3.7 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 3.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#18 @desrosj
3 years ago

In 50635:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.8 branch.

This backports several build and test tool improvements to the 4.8 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.8 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#19 @desrosj
3 years ago

In 50636:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.7 branch.

This backports several build and test tool improvements to the 4.7 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#20 @desrosj
3 years ago

In 50637:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.6 branch.

This backports several build and test tool improvements to the 4.6 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.6 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#21 @desrosj
3 years ago

In 50638:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.5 branch.

This backports several build and test tool improvements to the 4.5 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.5 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#22 @desrosj
3 years ago

In 50639:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.4 branch.

This backports several build and test tool improvements to the 4.4 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.4 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#23 @desrosj
3 years ago

In 50640:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.3 branch.

This backports several build and test tool improvements to the 4.3 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.3 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#24 @desrosj
3 years ago

In 50642:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.2 branch.

This backports several build and test tool improvements to the 4.2 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.2 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#25 @desrosj
3 years ago

In 50643:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.1 branch.

This backports several build and test tool improvements to the 4.1 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.1 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#26 @desrosj
3 years ago

In 50644:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.0 branch.

This backports several build and test tool improvements to the 4.0 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.0 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#27 @desrosj
3 years ago

In 50645:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.9 branch.

This backports several build and test tool improvements to the 3.9 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.9 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

#28 @desrosj
3 years ago

In 50646:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.8 branch.

This backports several build and test tool improvements to the 3.8 branch. Most notably, this includes:

  • The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
  • Splitting single site and multisite tests into parallel jobs [50379].
  • Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
  • Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
  • Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.8 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

Note: See TracTickets for help on using tickets.