Save PHPUnit timing artifacts to find and fix slow tests

[lancewillett]

WordPress core's PHPUnit configuration generates per-test timing data on every run, and CI discards it. As a result, nobody can answer "which tests are slow?" without running the suite locally.

Current behaviour

phpunit.xml.dist configures a JUnit log:

<logging>
    <log type="junit" target="tests/phpunit/build/logs/junit.xml" />
</logging>

That file records a time attribute for every test case. Two things then throw it away:

• /tests/phpunit/build is listed in .gitignore.
• No workflow uploads it. Neither phpunit-tests.yml nor reusable-phpunit-tests-v3.yml contains an actions/upload-artifact step, and a code search for "junit" across .github returns no results.

.phpunit.result.cache is also gitignored, so PHPUnit's own result cache — which records test durations and backs --order-by=duration — does not survive a run either.

The data is produced and destroyed on every run of phpunit-tests.yml.

Why this matters

Without published timing:

• Slow tests can only be found by running the suite locally, so nobody does it systematically and there is no trend over time.
• Duration-based job splitting is impossible. Other large PHP projects pack parallel test bins using historical JUnit timings; core cannot, because it has no historical timings.
• Claims about test performance cannot be checked against a shared source. Ticket #52645 previously looked at splitting slow tests, and re-testing that conclusion today means re-deriving the measurements from scratch.

Proposed change

Add an actions/upload-artifact step to the PHPUnit workflow that uploads tests/phpunit/build/logs/junit.xml on the scheduled run only.

Scheduled-only keeps the storage cost negligible while still producing a consistent time series against a known matrix. Pull-request runs vary by branch and would add cost for little analytical gain.

Two implementation details worth settling in review:

• A short retention period is likely appropriate.
• The matrix produces one log per job, so the artifact name should derive from the PHP and database versions to keep them distinguishable.

This is deliberately scoped to preserving data that already exists. It changes no test and adds no dependency.

Motivating data

From a single local run of trunk at 6c0f205fd6 — one Docker run on one machine, so directional rather than authoritative:

• 30,865 tests, 1,066 test classes, 234.1s of summed per-test time.
• 50% of that time is in 22 classes (2.1% of all classes).
• 25% is in 7 classes.
• REST API test classes account for 49.1% of it, across 61 classes.

If those proportions hold on CI hardware, the suite is concentrated enough that a small number of targeted fixes would move the total meaningfully. Publishing the artifact is what would let anyone confirm or refute that against real CI runs rather than one contributor's laptop.

Related

• #62004 — PHPUnit 10/11/12 readiness. The <logging> syntax above is the PHPUnit 9 form and changes in PHPUnit 10, so this should stay consistent with that work.
• #52645 — earlier work on splitting slow tests into separate jobs.
• #33968 — earlier unit-test performance work.

Not to be confused with the Performance Tests workflow, which measures WordPress page-load metrics rather than the runtime of the test suite itself.

[lancewillett]

Open question: where the file is hosted for anyone to read.

GitHub Actions artifacts attach to the workflow run itself. Once uploaded, the JUnit log appears in the Artifacts section of that run's summary page: downloadable as a zip from the UI, or from the command line:

gh run download <run-id> --repo WordPress/wordpress-develop -n <artifact-name>

No new hosting, storage, or external service is involved. This uses the artifact storage already attached to every workflow run.

Core already does exactly this in three workflows, so the convention is established rather than invented here:

• reusable-performance-test-v2.yml — name: performance-<multisite|single>-<memcached|default>-<subject>, if-no-files-found: error
• reusable-end-to-end-tests.yml — name: failures-artifacts-<run_id>, if-no-files-found: ignore
• reusable-build-package.yml — name: wordpress-develop, path: develop.zip

All three pin the same action reference: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1.

That also settles the artifact-naming detail in the description. Following the same pattern, the PHPUnit log would be named from the matrix dimensions — something like junit-php<version>-<db-type><db-version>: giving one artifact per job, distinguishable at a glance on the run page.

[johnbillion]

​The performance tests use the CodeVitals service to report their metrics: ​https://codevitals.run/public/WordPress/wordpress-develop/metrics . This might be a candidate to store metrics for the performance of the tests themselves.

## What this changes

The scheduled PHPUnit workflow now:

• writes separate JUnit XML reports for the main, AJAX, external HTTP, multisite-files, and Xdebug runs;
• uploads one artifact per matrix job, with the configuration in its name;
• retains each artifact for 14 days; and
• uploads partial results when a test segment fails.

Pull request, push, coverage, and manually dispatched runs do not upload these artifacts.

## Storage plan

This draft uses GitHub Actions artifacts for raw JUnit files. That keeps the source data attached to its workflow run and follows existing repository patterns.

CodeVitals remains a candidate for long-term trends. A follow-up can parse a canonical CI result and publish a small set of stable aggregate metrics there. Sending every test case as a CodeVitals metric would create excessive metric cardinality and would not preserve the raw report.

## Testing

• actionlint passed.
• zizmor 1.24.1 reported no findings.
• A targeted PHPUnit run confirmed that the command-line JUnit path creates a separate report without replacing the configured report.

[lancewillett]

​https://github.com/WordPress/wordpress-develop/pull/13070

End-to-end artifact validation passed.

The canonical PHPUnit job temporarily allowed the pull request event to exercise the exact upload step:

• ​Workflow run 31969896671
• ​PHP 8.5/MySQL 8.4 test-reporting job
• ​phpunit-timing-results artifact

The downloaded artifact contained a well-formed, 8,909,462-byte junit.xml file with timing for 30,871 test cases. The upload and download both completed successfully.

Commit 8ccbcd4b82 restored the scheduled-only condition after the test. The final workflow also passed actionlint, and zizmor reported zero findings.

## What this changes

The canonical scheduled PHPUnit job publishes six aggregate timing metrics to the existing public CodeVitals dashboard:

• PHP 8.5;
• MySQL 8.4;
• single site;
• default domain and cache configuration.

The publisher reads the existing JUnit report immediately after the main test suite, before later test-group runs replace the file. It streams the XML, creates a small JSON payload, and sends no test names or per-test records.

This complements #13070. That pull request preserves the raw JUnit report for detailed analysis; this one adds the visible trend layer.

## Metrics

Key	Display name	Unit	Default
phpunit-suite-time	PHPUnit main suite time	s	Visible
phpunit-p95-test-time	PHPUnit test time (p95)	ms	Other
phpunit-p99-test-time	PHPUnit test time (p99)	ms	Other
phpunit-max-test-time	PHPUnit maximum test time	ms	Other
phpunit-tests-over-500ms	PHPUnit tests over 500 ms	tests	Other
phpunit-tests-over-1s	PHPUnit tests over 1 s	tests	Other

Keeping one metric visible by default avoids crowding the existing dashboard. The other five remain available from the Other selector for diagnosis.

The definitions must exist in CodeVitals before the first submission. Suggested priorities are 100–105 in the order above, with a regression noise floor of 0 until scheduled runs establish a baseline.

## How to validate

After the first scheduled run, open:

​https://codevitals.run/public/WordPress/wordpress-develop/metrics

The PHPUnit main suite time card should appear. Select it to view the trend by commit. Use Other to add the percentile, maximum, and slow-test-count metrics.

The parser can be checked locally with:

npm run test:php -- --filter Tests_Includes_JUnit_Timing_Metrics

A previously captured CI report containing 30,871 timed test cases produced:

• suite: 281.19193 seconds;
• p95: 49.723 milliseconds;
• p99: 100.294 milliseconds;
• maximum: 3,446.888 milliseconds;
• over 500 milliseconds: 31 tests;
• over 1 second: 19 tests.

An authenticated CodeVitals submission remains a pre-merge validation item. The token stays in the existing CODEVITALS_PROJECT_TOKEN Actions secret and is never available to local or pull-request runs.

## Testing

• Focused PHPUnit tests passed: 4 tests, 6 assertions.
• PHP syntax checks and WordPress PHP coding standards passed.
• actionlint passed for both changed workflow files.
• The command-line publisher returned all six numeric metrics from both focused and full CI JUnit reports.
• The CodeVitals endpoint returned the expected authentication error for an invalid test token.

@johnbillion Thanks for suggesting CodeVitals on https://core.trac.wordpress.org/ticket/65887#comment:4.

CodeVitals requires GitHub repository admin access to manage metric definitions. Could you add these six definitions for WordPress/wordpress-develop?

Key	Display name	Unit	Priority	Visible by default	Noise floor
---	---	---	---:	---	---:
phpunit-suite-time	PHPUnit main suite time	s	100	Yes	0
phpunit-p95-test-time	PHPUnit test time (p95)	ms	101	No	0
phpunit-p99-test-time	PHPUnit test time (p99)	ms	102	No	0
phpunit-max-test-time	PHPUnit maximum test time	ms	103	No	0
phpunit-tests-over-500ms	PHPUnit tests over 500 ms	tests	104	No	0
phpunit-tests-over-1s	PHPUnit tests over 1 s	tests	105	No	0

The service silently drops unknown metric keys. This PR verifies that all six were accepted, so the scheduled publishing step will fail clearly if any definition is missing.

Once these exist, I can complete the authenticated submission and verify the public dashboard: ​https://codevitals.run/public/WordPress/wordpress-develop/metrics

[lancewillett]

​https://github.com/WordPress/wordpress-develop/pull/13083 is ready for CodeVitals; needs a GitHub org admin to add the new metrics definitions.

Thanks @desrosj -- fixed pushed ​https://github.com/WordPress/wordpress-develop/commit/bfd963734e36ce76100679536b3271f807aaf957

is there any reason why we can not submit test results for every commit?

I originally limited this to weekly runs for storage and consistency, but updated it to publish after every push to trunk based on your feedback.

PR runs remain excluded because their branch-specific results are less comparable—and would create a LOT more storage that isn't that helpful for stats. At least, not now.

3ff9c1994664a98458da6a1bd5ece797dd25a678 addresses review feedback.

We are limiting the metrics to trunk runs to prove the concept before adding all PR commits which can come later. For now, they'd add more storage and might not create an apples-to-apples comparison for an early look at slow tests.

@adimoldovan Curious your take on the approach. Is there a better way to store the test run results and display it in a dashboard somewhere?

@adimoldovan Curious your take on the approach. Is there a better way to store the test run results and display it in a dashboard somewhere?

@lancewillett The approach works if the goal is to store the trend. If what we want is signal to optimize slow tests, none of the six aggregates can name a test. #13070 already gives us time, file, and line per case, so publishing the top slow tests or those over a threshold as Github annotations would get us there. That could be a good signal in PRs too, so an author can spot a slow test they added.

## What this changes
Adds a Flag slow PHPUnit tests step to the canonical PHP 8.5 report job. It parses the JUnit report and, for tests over a threshold (default 1s), emits GitHub Actions warning annotations and a run-summary table naming the slowest tests. Runs on pull requests and pushes to trunk.

This complements #13083: those six CodeVitals aggregates store the trend but cannot name a test. This one names the slow ones, so a PR author can spot a slow test they added.

## Behavior

• Advisory only: runs even on failed test runs (always()) and never fails the build (continue-on-error).
• Inline annotations capped at 10 (GitHub's per-step limit); the summary table lists more.
• Container-absolute paths normalized to repository-relative so annotations resolve.
• A testcase without timing data (e.g. a skipped test) is ignored, not an error.

## Use of AI Tools
AI assistance: Yes
Tool(s): Codex (via Claude Code), hardened with an adversarial multi-lens review
Used for: drafting the workflow step and PHP script, and reviewing it before submission.

[lancewillett]

Follow-up: added a third piece under this ticket. ​PR 13212 flags slow individual tests directly in CI.

The three layers:

• ​PR 13083 publishes six aggregate timings to CodeVitals for the trend.
• ​PR 13070 preserved the raw JUnit report as an artifact for offline analysis.
• ​PR 13212 parses the JUnit report in-job and emits GitHub Actions warning annotations plus a run-summary table for tests over a threshold, on pull requests and trunk pushes. Advisory only; it never fails the build.

The aggregates store the trend but cannot name a test; PR 13212 names the slow ones, so a PR author can spot a slow test they added.

Since both consumers (13083 and 13212) read the JUnit report in-job, the uploaded artifact in PR 13070 no longer has a consumer, so I plan to close it in favor of 13212 once it lands.

Good call. Pushed a follow-up (d4505808ad) that fails the test explicitly when tempnam() returns false, so it can't reach unlink() in teardown.

Heads up: I plan to close this in favor of #13212 once it lands. Both timing consumers -- the CodeVitals aggregate (#13083) and the slow-test annotations (#13212) -- read the JUnit report in-job, so the uploaded artifact here has no consumer. If an offline-analysis need appears later, the upload is a few lines to restore.

Good call. Pushed a follow-up (d4505808ad) that fails the test explicitly when tempnam() returns false, so it can't reach unlink() in teardown.

@adimoldovan Thanks for your feedback. Opened ​https://github.com/WordPress/wordpress-develop/pull/13212 for flagging the slow tests by name using annotations.

Cross-PR note: #13212 also parses the PHPUnit JUnit report for slow-test annotations. Once this lands, #13212 will follow up to share this parser and traversal instead of opening the report a second time.

---
_{Adversarial review · <code>gpt-5</code>}

Direct review fixes are in ac5472e2688e2bd86d6a0d99cfa6c4c3d20cdb32. One architectural follow-up remains: #13083 is still open. Once it lands, I’ll update this branch to share its JUnit traversal instead of opening the report twice.

---
_{Adversarial review · <code>gpt-5</code>}

This can land once I have edit access to CodeVitals, or a GitHub admin is able to add those new metrics definitions.

See ​https://make.wordpress.org/systems/?p=3009

@youknowriad Can you help Lance add these new metrics to Codevitals?

Hey folks, I added the metrics per the table in the PR description. Anyone with admin rights on the repo should be able to do so, modify them...

[lancewillett]

In 63410:

Build/Test Tools: Publish PHPUnit timing metrics to CodeVitals.

The canonical PHPUnit job now publishes six aggregate timing metrics to the
public CodeVitals dashboard after every push to trunk: total suite time, the
p95, p99 and maximum per-test times, and counts of tests over 500ms and over
1s. The publisher reads the JUnit report the suite already produces, streams
it, and sends only aggregates. No test names or per-test records leave CI.

The metric definitions are verified on submission, so a missing definition
fails the step rather than silently dropping data.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/13083

Props desrosj, adrianmoldovanwp, johnbillion, youknowriad.
See #65887.

Committed to trunk in r63410 (ea95d00cedc193b0200af257e13120dbdb9205b3).

[lancewillett]

In 63412:

Build/Test Tools: Flag slow PHPUnit tests in GitHub Actions.

The canonical PHP 8.5 reporting job now emits warning annotations and a run-summary table for tests that exceed one second. The advisory step runs after successful or failed test runs and skips cancelled jobs.

The annotation pass reuses the streaming JUnit parser introduced in [63410] and passes its six aggregate metrics to CodeVitals through a temporary file. This keeps the workflow to one traversal of the report.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/13212

Props adrianmoldovanwp.
See #65887.

Commit landed in https://core.trac.wordpress.org/changeset/63412

Closing in favor of #13083 and #13212, which landed in r63410 and r63412. Together they publish aggregate timing trends and identify individual slow tests without retaining the full JUnit report as an artifact.

[lancewillett]

The work tracked here is complete.

Closing as fixed.

[63410] publishes six aggregate PHPUnit timing metrics from the canonical trunk job to the ​public CodeVitals dashboard. The first trunk submission populated all six metrics successfully.

[63412] adds GitHub Actions warnings and a run-summary table for individual tests over one second. It reuses the same streaming JUnit traversal and passes the aggregate metrics to the CodeVitals publisher through a temporary file, so the report is read once.

The full JUnit artifact proposed in ​PR #13070 no longer has a consumer, so that PR was closed as superseded.