Make WordPress Core

Opened 7 weeks ago

Closed 5 weeks ago

#65749 closed enhancement (fixed)

Allow the PHPUnit test runner to be set by a repository variable

Reported by: lancewillett Owned by: lancewillett
Priority: normal Milestone: 7.1
Component: Build/Test Tools Version:
Severity: normal Keywords: has-patch
Cc: Focuses:

Description

The PHPUnit workflows run on a fixed runner (ubuntu-24.04), passed as the os input to reusable-phpunit-tests-v3.yml and consumed at runs-on: ${{ inputs.os }}.

This makes the runner overridable by a repository variable, so maintainers can direct the PHPUnit matrices to a specific runner, for example a dedicated runner during high-load release windows, without editing the workflow.

reusable-phpunit-tests-v3.yml changes its runs-on to:

runs-on: ${{ vars.PHPUNIT_RUNNER || inputs.os }}

With the PHPUNIT_RUNNER variable unset (the default), jobs run on inputs.os exactly as today, so there is no behavior change. It is backward-compatible for every branch that calls the workflow at @trunk; no input is added or made required.

Change History (11)

This ticket was mentioned in PR #12745 on WordPress/wordpress-develop by @lancewillett.


7 weeks ago
#1

Allows the PHPUnit runner to be directed by a repository variable, defaulting to the current runner.

reusable-phpunit-tests-v3.yml changes its runs-on to ${{ vars.PHPUNIT_RUNNER || inputs.os }}. With the PHPUNIT_RUNNER variable unset (the default), jobs run on inputs.os (ubuntu-24.04) exactly as before. When a maintainer sets PHPUNIT_RUNNER to a runner label, the PHPUnit matrices run on that runner instead—useful for directing them to a dedicated runner during high-load release windows, without editing the workflow.

  • No behavior change by default; the variable is unset.
  • Backward-compatible for every branch that calls this workflow @trunk; no input is added or made required.
  • vars is available in runs-on, so the override is valid.

Trac: https://core.trac.wordpress.org/ticket/65749

@lancewillett commented on PR #12745:


7 weeks ago
#2

For this to work, @bazza or another GitHub admin will need to flip the PHPUNIT_RUNNER variable before / after release using GitHub.com settings for WordPress.org.

---

### How to route the PHPUnit matrices to a specific runner

This change is a no-op until a PHPUNIT_RUNNER repository variable exists. To direct the PHPUnit jobs to a runner (for example, a larger release-runner during a high-load release window):

Turn it on

  1. Repo → Settings → Secrets and variables → Actions → Variables.
  2. New repository variable → Name: PHPUNIT_RUNNER, Value: the runner label (e.g. release-runner).
  3. New PHPUnit runs pick it up immediately. In-flight runs are unaffected; re-run if you need them moved.

Turn it off

  1. Same page → delete the PHPUNIT_RUNNER variable.
  2. PHPUnit jobs fall back to the default runner (ubuntu-24.04).

Notes

  • Scoped per repository—set it only on the exact repo you want to redirect; other repos are unaffected.
  • Larger hosted runners are billed per-minute, so set it only for the window and remove it after.
  • Changing the variable needs repo admin access.

#3 @lancewillett
7 weeks ago

  • Owner set to lancewillett
  • Resolutionfixed
  • Status newclosed

In 62891:

Build/Test Tools: Allow the PHPUnit runner to be set by a repository variable.

Change runs-on in the reusable PHPUnit workflow to ${{ vars.PHPUNIT_RUNNER || inputs.os }}. With the variable unset (the default), jobs run exactly as before.

When a maintainer sets PHPUNIT_RUNNER to a runner label, the PHPUnit matrices run on that runner instead—useful for directing them to a dedicated runner during high-load release windows, without editing the workflow.

No input is added or made required, so every branch that calls this workflow at @trunk stays compatible.

Developed in: https://github.com/WordPress/wordpress-develop/pull/12745

Props barry.
Fixes #65749.

#4 @lancewillett
7 weeks ago

  • Milestone Awaiting Review7.1

#5 @desrosj
6 weeks ago

  • Resolution fixed
  • Status closedreopened

I've been thinking about this change, and the PHPUnit testing workflow wordpress-develop is not the only repository responsible for amassing a queue due to long-running jobs. The performance testing workflows both here and in Gutenberg, and the E2E test workflow in Gutenberg can also cause a traffic jam.

I'm going to rename the variable to something more generic so that expanding it's usage to to other workflows and repositories can be considered.

#6 @desrosj
6 weeks ago

  • Resolutionfixed
  • Status reopenedclosed

In 62974:

Build/Test Tools: Make runner override variable more general.

[62891] introduced the ability to override the runner used for a GitHub Actions job using a repository or organization variable. While initially named PHPUNIT_RUNNER, overriding the runner for a specific job could be useful in more situations.

This renames the variable chacked to RUNNER_GROUP.

Fixes #65749.

#7 @desrosj
6 weeks ago

In 63003:

Build/Test Tools: Further refine runner override variable name.

This changes RUNNER_GROUP to RUNNERS_NAME to avoid confusion with the runs-on.group setting, which is configured in a completely different way.

Props lancewillet.
See #65749.

#8 @lancewillett
6 weeks ago

  • Resolution fixed
  • Status closedreopened

#9 @lancewillett
6 weeks ago

Reopening. [62891] only covered reusable-phpunit-tests-v3.yml, so the override does not reach every branch that runs the PHPUnit matrices.

Branches 4.7 through 5.1 call reusable-phpunit-tests-v1.yml, and branches 5.2 through 5.8 call reusable-phpunit-tests-v2.yml, both at @trunk. Neither workflow reads the variable; each still has a bare runs-on: ${{ inputs.os }}. Setting RUNNERS_NAME at the repository or organization level therefore has no effect on those twelve branches.

A follow-up PR should apply the same one-line change to both files. With the variable unset, jobs run on inputs.os exactly as before, so there is no change by default.

This ticket was mentioned in PR #12847 on WordPress/wordpress-develop by @lancewillett.


6 weeks ago
#10

Extends r62891 to the two older reusable PHPUnit workflows.

That change added the runs-on override to reusable-phpunit-tests-v3.yml alone, so only branches calling that workflow can be redirected.

Branches 4.7 through 5.1 call reusable-phpunit-tests-v1.yml, and branches 5.2 through 5.8 call reusable-phpunit-tests-v2.yml, both at @trunk. Neither reads the variable, so setting RUNNERS_NAME at the repository or organization level has no effect on those twelve branches.

This applies the same one-line change to both files. With the variable unset, jobs run on inputs.os exactly as before, so there is no change by default.

Each of the three reusable PHPUnit workflows contains exactly one runs-on, so no other line in these files needs the same treatment.

## Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5
Used for: Identifying the gap by comparing which branches call which reusable workflow, drafting the two-line change, and drafting this description. Reviewed and verified by me before opening.

#11 @SergeyBiryukov
5 weeks ago

  • Resolutionfixed
  • Status reopenedclosed

In 63159:

Build/Test Tools: Extend the PHPUnit runner variable to the v1 and v2 workflows.

This commit extends r62891 to the two older reusable PHPUnit workflows.

That change added the runs-on override to reusable-phpunit-tests-v3.yml alone, so only branches calling that workflow can be redirected.

Branches 4.7 through 5.1 call reusable-phpunit-tests-v1.yml, and branches 5.2 through 5.8 call reusable-phpunit-tests-v2.yml, both at @trunk. Neither reads the variable, so setting RUNNERS_NAME at the repository or organization level has no effect on those twelve branches.

This applies the same one-line change to both files. With the variable unset, jobs run on inputs.os exactly as before, so there is no change by default.

Each of the three reusable PHPUnit workflows contains exactly one runs-on, so no other line in these files needs the same treatment.

Developed in https://github.com/WordPress/wordpress-develop/pull/12847.

Follow-up to r62891, r62974, r63003.

Props lancewillett.
Fixes #65749.

Note: See TracTickets for help on using tickets.