Run GitHub Action workflows for testing old branches on a schedule

[desrosj]

On TravisCI, the tests for each old branch would run periodically on a cron to ensure that they continue to pass as time progresses, even when changes are not made on a regular basis.

The schedule looked like this:

• Daily for the currently supported minor version.
• Weekly for the next oldest version.
• Monthly for all other older branches.

These crons would not run if the branch had a build within the last 24 hours.

This is a bit difficult to accomplish with the way GitHub Actions currently work. Though there is a schedule event that triggers workflow runs, it only works within the primary branch.

There is a ​`workflow_dispatch` event that allows manually triggering workflows, which is ​supported in the REST API.

Trac ticket: https://core.trac.wordpress.org/ticket/52653.

Besides the personal access token I would need to configure, I think this is in a good spot. But, I'm not sure how we'd be able to test it without adding the workflow_dispatch trigger to an older branch. Then we could change this new workflow to run on pull_request_target, and see if it is triggered successfully. Otherwise I don't think this will run.

[desrosj]

I have a proof of concept working over on my fork, but for some reason two of the workflows are not triggering correctly, despite having workflow_dispatch. When I dispatch the event by manually sending a curl request, the workflow triggers successfully.

I've ​opened an issue upstream to see if the action maintainer has any thoughts. That option is much cleaner, so I'd prefer to use that over crafting our own curl requests.

I've gotten this working by using the actions/github-script action and the workflow_dispatch event.

Here's how I've tested:

• Created a branch off of 5.5 called 5.5-test.
• Added the workflow_dispatch event to each workflow in that test branch.
• Added the pull_request trigger event to the new, test-old-branches.yml file.
• ​Opened a PR for my fork to merge this ​branch and trigger the workflow with a paired down list of versions.
• Observed ​manual runs dispatched for the `5.5-test` branch for each workflow.
• Polished the​workflow and pushed again to confirm all branches receive jobs for their appropriate workflows.

One downfall of manually dispatching workflows is that the branch does not display to the right of the workflow like push or pull_request events. But, they do show when filtering by branch, and after clicking into a workflow, you can click the SHA value to see the commit the workflow was triggered for (it will always be the most recent commit within the branch). This is a GHA limitation, though. I was unable to find any way to change the workflow name, or display more information on the run.

Previously on TravisCI, the older branches ran monthly and more recent ones weekly. This runs them all weekly. There was also a setting to only run scheduled builds if there had not been one run in the previous 24 hours for a branch. That has also not been copied over.

Is there a need for the current branch (currently 5.7) to be included in this matrix?

The schedule event trigger will only work in the primary branch. If 5.7 is not included, it would only be tested if a commit is made.

[desrosj]

In 50590:

Build/Test Tools: Run test workflows on old branches on a schedule.

On TravisCI, old branches still receiving security updates were tested on a regular basis. This ensured tests continued to pass as time passed even if updates were not made to these branches.

On GitHub Actions, there is no interface to configure this (TravisCI had a UI), but there is a schedule event that can trigger workflow runs on cron that can be used to accomplish the same thing.

This introduces a workflow file that runs twice a month (on the 1st and 15th) to verify the tests within older branches.

Because the schedule event only runs within the primary branch, the appropriate workflows in each old branch will be triggered manually through the workflow_dispatch trigger using the GitHub REST API. workflow_dispatch will need to be added to all workflows in all old branches in order for the event to dispatch successfully.

Fixes #52653.

[desrosj]

Reopening to backport.

[desrosj]

In 50591:

Build/Test Tools: Run test workflows on old branches on a schedule.

On TravisCI, old branches still receiving security updates were tested on a regular basis. This ensured tests continued to pass as time passed even if updates were not made to these branches.

On GitHub Actions, there is no interface to configure this (TravisCI had a UI), but there is a schedule event that can trigger workflow runs on cron that can be used to accomplish the same thing.

This introduces a workflow file that runs twice a month (on the 1st and 15th) to verify the tests within older branches.

Because the schedule event only runs within the primary branch, the appropriate workflows in each old branch will be triggered manually through the workflow_dispatch trigger using the GitHub REST API. workflow_dispatch will need to be added to all workflows in all old branches in order for the event to dispatch successfully.

Merges [50590] to the 5.7 branch.
See #52653.

[desrosj]

In 50602:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.6 branch.

This backports several build and test tool improvements to the 5.6 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.6 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50603:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.5 branch.

This backports several build and test tool improvements to the 5.5 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50592,50598] to the 5.5 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667, #52786.

[desrosj]

In 50604:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.4 branch.

This backports several build and test tool improvements to the 5.4 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.4 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50605:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.3 branch.

This backports several build and test tool improvements to the 5.3 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.3 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50606:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.2 branch.

This backports several build and test tool improvements to the 5.2 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• The ability to run PHPUnit tests from src instead of build [50441-50443].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50267,50299,50379,50387,50413,50416,50432,50435-50436,50441-50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.2 branch.
See #50401, #51734, #51801, #51802, #52548, #52608, #52612, #52623, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50622:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.1 branch.

This backports several build and test tool improvements to the 5.1 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.1 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50624:

Build/Test Tools: Backport GitHub Action and build improvements to the 5.0 branch.

This backports several build and test tool improvements to the 5.0 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [45317,50267,50379,50387,50413,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50579,50590,50598] to the 5.0 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50625:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.9 branch.

This backports several build and test tool improvements to the 4.9 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.9 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50626:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.7 branch.

This backports several build and test tool improvements to the 3.7 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 3.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50635:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.8 branch.

This backports several build and test tool improvements to the 4.8 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.8 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50636:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.7 branch.

This backports several build and test tool improvements to the 4.7 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50413,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.7 branch.
See #50401, #51801, #51802, #52548, #52608, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50637:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.6 branch.

This backports several build and test tool improvements to the 4.6 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.6 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50638:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.5 branch.

This backports several build and test tool improvements to the 4.5 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.5 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50639:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.4 branch.

This backports several build and test tool improvements to the 4.4 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.4 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50640:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.3 branch.

This backports several build and test tool improvements to the 4.3 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.3 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50642:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.2 branch.

This backports several build and test tool improvements to the 4.2 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435,50436,50444,50446,50473,50474,50476,50479,50485,50486,50487,50545,50579,50590] to the 4.2 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50643:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.1 branch.

This backports several build and test tool improvements to the 4.1 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.1 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50644:

Build/Test Tools: Backport GitHub Action and build improvements to the 4.0 branch.

This backports several build and test tool improvements to the 4.0 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 4.0 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50645:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.9 branch.

This backports several build and test tool improvements to the 3.9 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.9 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

[desrosj]

In 50646:

Build/Test Tools: Backport GitHub Action and build improvements to the 3.8 branch.

This backports several build and test tool improvements to the 3.8 branch. Most notably, this includes:

• The changes required to allow each workflow to be triggered by the workflow_dispatch event so that tests can be run on a schedule [50590].
• Splitting single site and multisite tests into parallel jobs [50379].
• Split slow tests into separate, parallel jobs for PHP <= 5.6 [50444].
• Better branch and path scoping for GitHub Action workflows when running on pull_request [50432,50479].
• Several devDependency updates.

Merges [50379,50387,50416,50432,50435-50436,50444,50446,50473-50474,50476,50479,50485-50487,50545,50590] to the 3.8 branch.
See #50401, #51801, #51802, #52548, #52612, #52624, #52625, #52645, #52653, #52658, #52660, #52667.

Merged into Core in https://core.trac.wordpress.org/changeset/50590.

[desrosj]

In 50648:

Build/Test Tools: Prevent the workflow for testing old branches from running on forks.

This adds a conditional statement to ensure the workflow that tests old branches only runs on the official wordpress-develop mirror. This prevents it from running on forks and private mirrors.

See #52653.

[desrosj]

This week the first scheduled run of this workflow occurred: ​https://github.com/WordPress/wordpress-develop/actions/runs/707069966.

Branches 3.7, and 4.9-5.7 all were tested successfully (the other branches did not have the required changes to be run). Here are some notes.

The workflow was scheduled for the first of the month at 00:00 UTC. When jobs are scheduled, GitHub takes all of the jobs with the same scheduled time and places them in a pool and they are run as resources are available. This resulted in the workflow actually starting at ~02:34. This delay is not a huge deal, but it's worth noting it's not always immediate.

The next scheduled run is on the 15th at 00:00 UTC. After that occurs, I think that 52653.diff​ should be committed to do a few things.

First, it splits the two scheduled dates into two separate crons. This allows us to detect which day it is (1st or 15th).

Second, it introduces a conditional statement using the separate crons above to only test the two most recent branches twice per month, and all other branches once per month. This more closely matches our testing setup on Travis, and prevents some needless testing.

And finally, because an overwhelming number of projects are most likely running workflows at 00:00 on the 1st, it changes all branches to being tested on the 15th, and only the 2 most recent on the 1st.

Trac ticket:

[desrosj]

In 50763:

Build/Test Tools: Make adjustments to how often old branches are tested.

This adjusts the workflow responsible for testing old branches to only test old branches once per month. This more closely resembles the testing practices previously in place on TravisCI.

The latest, supported branch will continue to be tested twice per month.

This also makes adjustments so that the workflow is run whenever changes are made to it. This will help verify changes without having to wait until the next scheduled run.

Fixes #52653.

Merged into Core in https://core.trac.wordpress.org/changeset/50763.