Opened 4 weeks ago
Last modified 21 hours ago
#65893 accepted task (blessed)
Many PHPUnit tests are not well isolated
| Reported by: | mdawaffe | Owned by: | lancewillett |
|---|---|---|---|
| Priority: | low | Milestone: | 7.2 |
| Component: | Build/Test Tools | Version: | |
| Severity: | trivial | Keywords: | has-patch has-unit-tests |
| Cc: | Focuses: | tests |
Description
In talking about tests at WCUS 2026, we reminded ourselves that many of our PHPUnit tests leak state and/or depend on previously leaked state. This is easy to see by running the tests in an arbitrary order:
npm run test:php -- --order-by=random
This lack of isolation can result in varied problems:
- A test cannot be run by itself
- A test class cannot be run by itself
- A test passes because of some previously leaked state and not because the test is actually testing the correct thing
- etc.
Reducing state leaking reduces fragility, increases confidence that tests are testing what they should be testing, and is a likely prerequisite for future parallel test running.
This ticket is for sharing a grab-bag of unrelated test fixes we worked on during WCUS.
Change History (62)
This ticket was mentioned in PR #13091 on WordPress/wordpress-develop by @mdawaffe.
4 weeks ago
#1
- Keywords has-patch has-unit-tests added
#2
@
4 weeks ago
Given the random ordering, it seems like it would be interesting to take each test suite and perform a high number of runs on it with random ordering to see what issues appear.
@mdawaffe commented on PR #13091:
4 weeks ago
#3
I understand this doesn't close the ticket, given the random test ordering this just found whatever appeared quickly but more rounds of testing likely reveal more issues.
Yeah - there are at least dozens of tests that are flaky in this way: I only looked at this handful. I don't think it's worth it to try to fix them all in one ticket, but you all know best :)
I suspect most of the test issues are reasonably straightforward: an agent looping through
npm run test:php -- --order-by=random --stop-on-failure
for a while would likely fix the low-hanging fruit.
#5
@
4 weeks ago
PR #13091 is ready to land, I can handle that when commits are open again.
I'm dedicating some resources to discover and fix more issues and expect to produce more PRs.
This ticket was mentioned in PR #13181 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#6
## Summary
- Resets admin menu registries and plugin-dependency statics, removes temporary MU-plugin fixtures, isolates list-table screens, and makes updater open_basedir tests account for an external Composer vendor directory.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized admin-group runs passed; each completed 1,024 tests and 2,721 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13182 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#7
## Summary
- Resets the forced-SSL function-static fixture and resynchronizes timezone-choice translation state after locale-changing data sets.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized functions-group runs passed; each completed 1,212 tests and 2,499 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13183 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#8
## Summary
- Removes hard-coded site-ID, directory-existence, lazy-load queue, external-DNS, and wp_installing state assumptions from multisite fixtures.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized multisite-group runs passed; each completed 763 tests and 1,805 assertions with two expected skips.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13184 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#9
## Summary
- Resets permalink structure at the start of core test teardown so in-memory rewrite state remains synchronized with the database state restored by rollback.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized post-group runs passed; each completed 933 tests and 2,641 assertions with one expected skip.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13185 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#10
## Summary
- Isolates REST authentication flags, block registrations, widget factories, sidebar state, theme features, script globals, and mocked URL validation fixtures.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized REST API group runs passed; each completed 3,550 tests.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13186 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#11
## Summary
- Clears the global comment fixture during teardown so the no-comment case cannot inherit a comment from a preceding test.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized comment-group runs passed; each completed 627 tests and 1,530 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13187 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#12
## Summary
- Avoids mutating shared user IDs, refreshes cached capability fixtures, scopes INFORMATION_SCHEMA lookup to the active database, and isolates the constant-defining capability test.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized user-group runs passed; each completed 1,341 tests and 4,528 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13188 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#13
## Summary
- Restores sticky and canonical-query fixtures and uses the recognized case-sensitive ID orderby value to make child-post selection deterministic.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized query-group runs passed; each completed 1,892 tests and 4,420 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13189 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#14
## Summary
- Initializes WP_Scripts before case-specific URL filters and re-primes locale state around privacy and user-request email fixtures.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized l10n-group runs passed; each completed 266 tests and 804 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13190 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#15
## Summary
- Resets loading-optimization state before each test, replaces a live crop URL with a local fixture, accepts generated caption-ID suffixes, and derives expected attachment names from actual uploads.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized media-group runs passed; each completed 842 tests and 2,220 assertions with 52 expected skips.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13191 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#16
## Summary
- Restores the Ajax callback baseline during every test setup so a preceding test cannot permanently remove callbacks required by another case.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized Ajax-group runs passed; each completed 190 tests and 1,212 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13192 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#17
## Summary
- Runs the layout variation block-gap test in a separate process because the rendering function intentionally retains an otherwise unresettable function-static global-styles snapshot.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized block-supports-group runs passed; each completed 400 tests and 532 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13193 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#18
## Summary
- Flushes rewrite rules after initial taxonomy registration and clears leaked HTTPS server state during rewrite-suite teardown.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized rewrite-group runs passed; each completed 1,393 tests and 2,826 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13194 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#19
## Summary
- Uses a genuinely unregistered block fixture for negative global-style assertions and restores theme support, stylesheet, filter, and theme-JSON cache state.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized themes-group runs passed; each completed 625 tests and 7,297 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13195 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#20
## Summary
- Synchronizes the translation controller with the restored site locale during base teardown and restores the original WP_Locale object after locale-switcher tests.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized i18n-group runs passed; each completed 267 tests and 809 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13196 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#21
## Summary
- Restores the font-face theme-root fixture and corrects the theme-switching flag cleanup so font-face tests do not inherit process state.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- 50 consecutive randomized fonts-group runs passed; each completed 228 tests and 926 assertions.
- Replayed the original failing seed and focused contaminator/consumer cases during diagnosis.
- PHP syntax checks passed for every changed PHP file.
git diff --checkpassed.
This ticket was mentioned in PR #13205 on WordPress/wordpress-develop by @jonsurrell.
4 weeks ago
#23
## Summary
- Restores the incoming stylesheet after block-template and editor fixtures switch themes.
- Applies the same teardown discipline to the remaining blocks-group fixtures that switch themes.
- Prevents stale Theme JSON state from changing later block rendering.
- Keeps the changes confined to PHPUnit tests and fixtures.
## Verification
- Minimized two independent contaminators of the block-rendering columns case and replayed both original failing seeds.
- 100 consecutive fresh randomized blocks-group runs passed on the combined campaign branch after the complete theme-switch audit; each completed 1,080 tests and 2,825 assertions. Both relevant patch IDs and all seven changed file trees match this isolated branch.
- Seeds 1789029001 through 1789029100 all exited cleanly.
- PHP syntax checks and
git diff --checkpassed.
@lancewillett commented on PR #13091:
12 days ago
#27
Committed to trunk in https://core.trac.wordpress.org/changeset/63435
#28
follow-up:
↓ 29
@
12 days ago
This ticket remains open for the remaining PHPUnit test isolation fixes.
@jonsurrell I noticed several of your other PRs are still in draft. Do you want help to update/refresh them?
#29
in reply to: ↑ 28
@
11 days ago
Replying to lancewillett:
@jonsurrell I noticed several of your other PRs are still in draft. Do you want help to update/refresh them?
Please feel free. They're in my backlog but I have some other priorities right now. If you'd like to take them on you're welcome to.
There's also a follow-up from the change that landed as r63435.
@jonsurrell commented on PR #13192:
11 days ago
#30
This was discovered independently and duplicated by https://github.com/WordPress/wordpress-develop/pull/13214 (already landed in Gutenberg as https://github.com/WordPress/gutenberg/pull/81889.
@lancewillett commented on PR #13182:
11 days ago
#31
Updated this branch with current trunk and fixed the PHPCS assignment-alignment warning in wpTimezoneChoice.php.\n\nValidation: targeted PHPCS passes. The focused PHPUnit file could not run because the local test configuration file is not present.
@lancewillett commented on PR #13205:
11 days ago
#32
Updated this branch with current trunk and fixed the PHPCS assignment-alignment warning in wpBlockPatternsRegistry.php.
Validation: targeted PHPCS passes. The focused PHPUnit file could not run because the local test configuration file is not present.
This ticket was mentioned in PR #13392 on WordPress/wordpress-develop by @lancewillett.
11 days ago
#33
## Summary
- Prepares font-face test theme fixtures in their shared base class.
- Removes redundant
set_up_before_class()overrides from the four font-face test classes.
## Testing
./vendor/bin/phpunit --group fontface(29 tests, 42 assertions)- PHPCS for the five changed files
Trac: https://core.trac.wordpress.org/ticket/65893
Follow-up to the review discussion in #13091.
@lancewillett commented on PR #13196:
11 days ago
#34
@lancewillett commented on PR #13181:
11 days ago
#35
Refreshed with current trunk in bf55661314; the original isolation change remains intact. Validation passed: PHPCS for the four changed files, focused PHPUnit (59 tests, 252 assertions), and a randomized replay (seed 13181; 59 tests, 252 assertions). The focused run is single-site; CI will provide the full matrix.
@lancewillett commented on PR #13184:
11 days ago
#36
Updated in ebfa2e2134.
The deterministic multisite failure came from the two custom-logo tests writing permalink_structure directly while the base teardown now resets rewrite state through set_permalink_structure(). That left the database option and in-memory rewrite object out of sync.
The tests now set the Reading options first and use set_permalink_structure() to initialize and flush rewrite state. With the synchronized route, /blog/ resolves to the configured page_for_posts; get_custom_logo() intentionally omits aria-current for its home link when the queried object is that posts page, so the two stale Blog index expectations now assert its absence.
Validation:
- Single-site focused custom-logo cases: 8 tests, 16 assertions passed.
- Multisite focused custom-logo cases: 8 tests, 16 assertions passed.
- PHPCS passed for both changed PHPUnit files.
- PHP syntax checks and
git diff --checkpassed.
Fresh CI is queued for the updated branch.
@lancewillett commented on PR #13183:
11 days ago
#37
Refreshed with current trunk in a non-rewriting merge: a9beabef99b4c68250b2ee94824866366180eb51.
Validation passed: git diff --check and vendor/bin/phpcs --standard=phpcs.xml.dist for the five changed PHP test files. I did not complete the focused PHPUnit run locally; the new CI run supplies the test matrix.
@lancewillett commented on PR #13186:
11 days ago
#38
Refreshed with current trunk in 4706632923; the original comment-global isolation change remains intact. Validation passed: PHPCS for both changed files, the targeted test (7 tests, 7 assertions), and the concrete comment-file set (23 tests, 24 assertions), including a randomized replay (seed 13186; 23 tests, 24 assertions). The focused runs are single-site; CI will provide the full matrix.
@lancewillett commented on PR #13188:
11 days ago
#39
Refreshed with current trunk in a non-rewriting merge: ae25ec1bf3cc59c95f227498a26e1e0632dbe1ef.
Validation passed: git diff --check; vendor/bin/phpcs --standard=phpcs.xml.dist for the three changed PHP test files; and a PHPUnit randomized replay (seed 13188) covering canonical and query tests: 1,071 tests, 1,203 assertions.
Limitation: this was a targeted replay; the new CI run provides the full matrix.
@lancewillett commented on PR #13190:
11 days ago
#40
Refreshed with current trunk in a non-rewriting merge: 51c5e195a34c3b66dacc9c8e9cd086b1cea6ee31.
Validation passed: git diff --check; vendor/bin/phpcs --standard=phpcs.xml.dist for the three changed PHP test files; and a focused randomized PHPUnit contamination replay (seed 13190): 7 tests, 32 assertions.
Limitation: this was a targeted replay; the new CI run provides the full matrix.
@lancewillett commented on PR #13191:
11 days ago
#41
Refreshed with current trunk in a non-rewriting merge: ee9071a60674e024be4d91bae804b03dba3b7f47.
Validation passed: git diff --check; vendor/bin/phpcs --standard=phpcs.xml.dist for the changed Ajax fixture; and a focused randomized Ajax replay (seed 13191): 15 tests, 31 assertions.
The prior one-off CI timeout had no assertion failure. Fresh CI has started and will provide matrix coverage; local validation was targeted.
@lancewillett commented on PR #13185:
11 days ago
#42
Updated this branch with current trunk. The REST block-type change is already byte-for-byte in r63435, so the resulting PR diff retains only the other nine isolation/DNS fixture files.\n\nValidation passed: PHPCS on all changed files, , and focused REST/DNS coverage (15 tests, 92 assertions) in normal order and randomized order with seed 13185. CI has been triggered by this push.
@lancewillett commented on PR #13193:
11 days ago
#43
Updated this branch with a non-rewriting merge of current trunk (1339550b82).
Validation completed:
- PHPCS on both changed files
- Focused rewrite and conditional-query tests: 213 tests, 955 assertions
- Randomized rewrite group with seeds 13193 and 13195: 1,393 tests, 2,826 assertions each
git diff --check
@lancewillett commented on PR #13187:
11 days ago
#44
Updated with 38efccb762 after merging current trunk.
The separate-process capability test previously passed user ID 0 to map_meta_cap(). On multisite, that user is not a super admin, so file-management capabilities map to do_not_allow instead of the test’s original expected primitive capabilities. The test now creates an administrator fixture, grants it super-admin privileges on multisite, and revokes those privileges in finally.
Validation:
- PHPCS: all four changed user test files
- Multisite
mapMetaCap.php: 35 tests, 96 assertions - Multisite randomized replay of the four changed user test files (seed 65893): 763 tests, 3,136 assertions
- Single-site randomized replay of the same files (seed 65894): 753 tests, 2,903 assertions
- PHP syntax checks and
git diff --check
@lancewillett commented on PR #13194:
11 days ago
#45
Updated with a non-rewriting merge of current trunk (d39fddbc6f) and moved post-thumbnails and foobar cleanup into unconditional Tests_Theme_Support::tear_down() cleanup. The in-test removal assertions remain unchanged.
Validation completed:
- PHPCS on all changed PHP files
- Focused template and theme tests: 116 tests, 556 assertions
- Randomized themes group with seeds 13194 and 13195: 627 tests, 7,276 assertions each (15 skipped)
git diff --check
@mukesh27 commented on PR #13392:
10 days ago
#46
@lancewillett commented on PR #13392:
10 days ago
#48
#49
@
6 days ago
Status and how to help.
Landed so far: [63435], the first batch of leaked-state fixes from #13091, and [63466], the font-face fixture cleanup.
Thirteen pull requests are open. All were written by @jonsurrell and refreshed against current trunk on September 3. Six are ready for review with green matrices, and none has a review yet. Reviewing one is the most useful thing anyone can do on this ticket right now.
Ready for review:
- https://github.com/WordPress/wordpress-develop/pull/13181 — admin:
admin/includesPlugin.php,admin/plugin-dependencies/base.php,admin/wpAutomaticUpdater.php,admin/wpListTable.php - https://github.com/WordPress/wordpress-develop/pull/13186 — comment globals, in
includes/abstract-testcase.phpandcomment/getCommentAuthorUrlLink.php - https://github.com/WordPress/wordpress-develop/pull/13187 — user fixtures:
user/capabilities.php,user/countUserPosts.php,user/mapMetaCap.php,user/query.php - https://github.com/WordPress/wordpress-develop/pull/13191 — Ajax hooks, in
includes/testcase-ajax.php - https://github.com/WordPress/wordpress-develop/pull/13193 — rewrite and query:
query/conditionals.php,rewrite.php - https://github.com/WordPress/wordpress-develop/pull/13205 — themes after block tests: six files under
blocks/plustheme/wpThemeGetBlockPatterns.php
How to review. Check out the branch and run the touched directory in random order with two seeds, then once in default order. The path goes straight to PHPUnit:
npm run test:php -- tests/phpunit/tests/user --order-by=random --random-order-seed=1 npm run test:php -- tests/phpunit/tests/user --order-by=random --random-order-seed=2 npm run test:php -- tests/phpunit/tests/user
A useful review confirms three things: the patch restores state in tear_down() or in the base testcase rather than special-casing one test, the random-order failure it targets no longer reproduces, and no assertion was lost on the way. composer lint on the touched files should be clean. Leave the review on the PR and a one-line "tested" note here.
How to find more. Pick a directory under tests/phpunit/tests/, run it under random order with two or three seeds, and fix whatever fails. Comment here with the directory before starting so two people do not take the same one, then open one PR per directory against this ticket.
For orientation only: a full run at seed 65893 on August 16 (trunk 888f0d9a4a, PHP 8.3, MySQL 9.7, single site) produced 7 errors and 68 failures across 41 classes. That is one local run with no default-order baseline, so it is a place to start and not a count. The classes with the most failures were Tests_Interactivity_API_WpInteractivityAPI (6), Tests_Comment_wpHandleCommentSubmission (5), then Test_WP_Customize_Custom_CSS_Setting, Tests_User_Capabilities, WP_Test_REST_Comments_Controller and WP_Test_REST_Users_Controller (4 each), and Tests_Block_Supports_WpStripCustomCssFromBlocks and Tests_WP_Customize_Manager (3 each). Several of those are covered by the PRs above; the Customize and Interactivity API ones are not.
Related: #53781 (embed template; environment-sensitive, and not reproducing on current trunk), #37207 (rewrite endpoints, PR #12730 is open and green), #53011 (the 2021 setUp and tearDown review this continues).
The call for contributors is published on Make/Core and links to this ticket: https://make.wordpress.org/core/2026/09/08/join-the-phpunit-test-cleanup-for-7-2/
#50
@
6 days ago
Sequencing note from today's discussion in #core-build-test-tools (https://wordpress.slack.com/archives/C08D0Q6BHNY/p1788901570764939): isolation fixes here and the class split on #65208 collide when they touch the same files. @johnbillion raised it, @pbearne suggested the order.
- One claim per directory covers both tickets. When you claim a directory here, say whether you are also splitting it, so nobody starts the other half from current trunk.
- Directories with an open isolation PR (the thirteen listed above) are spoken for. Split work there waits until the PR lands.
- Everywhere else, split first, then isolate the smaller files.
@lancewillett commented on PR #13194:
5 days ago
#51
Reviewed a922788480; no blocking findings. The cleanup restores theme state before the parent teardown, and the negative global-styles assertion now uses an unregistered block fixture.
Validation:
- All three affected classes passed in single-site and multisite: 116 tests and 556 assertions each.
- The themes group passed with random seeds 13194 and 13195: 633 tests and 7,289 assertions each, with 15 skips.
- PHPCS passed all three changed PHP files.
The six canceled CI jobs exceeded the 20-minute limit during Build WordPress, before PHPUnit started. I reran those jobs; the retry is still running.
---
<sub>Adversarial review · <code>gpt-6</code></sub>
![(please configure the [header_logo] section in trac.ini)](/chrome/site/your_project_logo.png)
There are a bunch of PHPUnit tests that implicitly depend on state leaked from other tests. This leaky state isn't observed in "normal" test runs, but can be easily seen by running:
Leaky state can cause several problems:
Trac ticket: https://core.trac.wordpress.org/ticket/65893
## Use of AI Tools
AI assistance: Yes
Tool: Claude Code
Model: Opus 5
Used for: Analysis for failing tests, initial implementation; final implementation reviewed/edited by me.