Add a plugin compatibility testing workflow to catch fatals with popular plugins before release

[adamsilverstein]

Claude drafted this ticket, prompted and edited by me:

WordPress 7.1 shipped and sites running WP Rocket began fataling on every request - Uncaught TypeError: substr(): Argument #1 ($string) must be of type string, int given in Cloudflare.php:562. Austin Ginder reported all of his customer sites went offline until he manually patched the plugin: ​https://x.com/austinginder/status/2090199834787541074

Core's CI is thorough about testing core itself - PHPUnit, e2e, install and upgrade paths - but nothing currently checks that a new version of WordPress can boot with popular plugins active. That check happens informally through beta testers, hosts, and plugin authors, and this incident shows the gap: a fatal on every request, on a top-tier plugin, made it through a full release cycle.

Proposed approach

Add a GitHub Actions workflow (following the structure of the existing Installation Tests and Upgrade Tests workflows) that:

• Fetches the current top 100 plugins by popularity from the WordPress.org plugins API at run time, so there is no hardcoded list to go stale.
• Installs WordPress via WP-CLI at the version under test (latest, nightly, or a specific version, selectable via workflow_dispatch so release leads can point it at a beta or RC).
• Installs and activates each plugin one at a time, in isolation, and verifies WordPress still runs: activation exit code, a WP-CLI load check, and front end / admin requests asserting no 5xx responses and no fatals in the debug log.
• Reports a per-plugin pass/fail table in the workflow summary, tests all plugins even when one fails, and fails the run if any plugin fatals.
• Runs weekly against nightly on a schedule, plus on demand.

To start this would be signal-only (scheduled and manual runs with Slack notifications on failure) rather than blocking - a broken third-party plugin should not turn core CI red on every commit.

Worth noting the limits honestly: the .org API only covers free directory plugins, so this exact incident (WP Rocket is a premium plugin) would not have been caught directly - but the same class of type-change fatal could just as easily land in any of the free plugins with millions of installs, and activation smoke testing is exactly where it would surface. Testing plugins in combination, premium plugin zips, and multisite are possible follow-ups.

Full acceptance criteria and an implementation plan are written up here: ​https://github.com/adamsilverstein/wordpress-develop/issues/64

Patch incoming (draft PR being prepared, will link it here once it is up).

What do others think - is a run of the top 100 the right starting scope, and does any of this overlap with existing plugin directory infrastructure?

AI Use

Description written with Claude Code; the linked PR contains Claude-written code and says so in its own AI Use section. I will review and test.

_Claude Code built this workflow, requirements and steering from me:_

Adds a workflow that installs the top plugins from the WordPress.org directory one at a time against a given version of WordPress and checks that nothing fatals. The idea is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version, taking real sites offline - while there is still time to fix core or reach the plugin author. See ​https://github.com/adamsilverstein/wordpress-develop/issues/64 and ​https://x.com/austinginder/status/2090199834787541074

A Trac ticket is being opened for this and will be linked here once it exists. Core needs a ticket before anything can be committed, so this stays a draft until then.

Fixes ​https://github.com/adamsilverstein/wordpress-develop/issues/64

## How it works

plugin-compatibility.yml queries https://api.wordpress.org/plugins/info/1.2/ for the most popular plugins at run time, so there is no list to go stale, and splits the slugs into 5 shards. Each shard calls reusable-plugin-compatibility.yml, which installs WordPress with WP_DEBUG and WP_DEBUG_LOG on, starts php -S, and then for each plugin installs it, activates it, runs wp eval, requests / and /wp-login.php, checks wp-content/debug.log, and removes the plugin before moving to the next one.

WP_DEBUG_DISPLAY is left off on purpose so the site behaves the way a production site does - a fatal is an empty page and an HTTP 500 rather than a printed stack trace. The fatal error handler is disabled too, otherwise recovery mode swallows the fatal and deactivates the plugin mid-test.

Runs are workflow_dispatch (so a release lead can point it at a beta or RC) and weekly against nightly. It is signal-only, not a check on every commit - a third party plugin breaking shouldn't turn core CI red on unrelated work.

## How has this been tested

The workflow has not run on GitHub Actions yet. Fork runs are blocked by the github.repository == 'WordPress/wordpress-develop' guards that the other workflows use, so the first real run will have to happen after this lands or via a temporary guard removal on a branch. Everything below was verified locally instead.

actionlint 1.7.12 with shellcheck 0.10.0 on PATH reports 0 errors for both new files, and 0 errors across the whole .github/workflows directory. zizmor 1.24.1 with --persona=regular --strict-collection reports one unpinned-images finding on the database service image, which is the same finding it already reports for install-testing.yml and reusable-upgrade-testing.yml, and it exits 0 in the --format=sarif mode the lint workflow uses.

The matrix builder was run locally against the live API. With plugin-count=100 it returned 100 slugs split into 5 shards of 20, each valid JSON. With plugin-count=10 it returned 5 shards of 2. With a count smaller than the shard count (3) it returned 3 shards of 1 rather than empty shards.

The per-plugin loop was extracted from the YAML and run unchanged in a container with WP-CLI, PHP 8.3 and MySQL 8.4, which is as close to the runner as could be managed locally:

• The top 10 popular plugins against nightly - all 10 passed, and the whole thing including the WordPress download took about 85 seconds, so a shard of 20 should sit well under the 20 minute target.
• A nonexistent slug was reported as SKIPPED and did not fail the job.
• Four deliberately fataling test plugins, one per detection path, all reported FAIL with the right reason: fatal on plugin load caught at activation, fatal on template_redirect caught as HTTP 500 on /, fatal on shutdown caught at activation, and a fatal in wp_footer (which still returns HTTP 200 because output already started) caught in debug.log.
• After a fataling plugin, wp-content/plugins and active_plugins were both back to a clean state and the next plugin in the list still ran and passed.

Not tested: multisite, PHP versions other than 8.3, MariaDB, and the Slack notification and failed-workflow jobs, which are copied from install-testing.yml unchanged.

## Types of changes

• Add .github/workflows/plugin-compatibility.yml, the caller, which builds the plugin matrix and fans out to 5 shards.
• Add .github/workflows/reusable-plugin-compatibility.yml, which installs WordPress and tests one shard of plugins in isolation.
• Write a per-plugin results table to the workflow summary and fail the job only when a plugin fatals.

## Open questions

• Is 5 shards of 20 the right shape? The count is an input, so a release lead could run 250 against an RC, but that would make each shard 50 plugins.
• Should this hook into .version-support-*.json for the PHP version instead of pinning 8.3, or is one current PHP version the right scope for a smoke test?
• Premium plugins like WP Rocket itself can't be fetched from the .org API, so the exact incident that prompted this wouldn't have been caught. Is there an appetite for a vendor-supplied zip input down the road, or does that raise too many licensing questions?
• Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.

Trac ticket: https://core.trac.wordpress.org/ticket/65920

## AI Use

Code and description written with 🤖 Claude Code, working from acceptance criteria in the linked issue. I will review and test.

[JeffPaul]

Installs and activates each plugin one at a time, in isolation, and verifies WordPress still runs: activation exit code, a WP-CLI load check, and front end / admin requests asserting no 5xx responses and no fatals in the debug log.

We may want to ensure a new env spins up for each plugin, as some may not properly clean up after themselves on uninstall and leave around detritus that may impact other plugins runs. It wasn't clear if you were planning to re-use the same environment for all 100 plugins runs, so I wanted to call it out explicitly that we should probably use a fresh install for each plugin run.

Runs weekly against nightly on a schedule, plus on demand.

This feels like maybe a later phase, somewhat because this will be a LOT of GHA minutes to churn through this weekly and just because we have free access to them doesn't mean we should overuse them. I think allowing a Release Lead or their delegate to run at Beta 1, RC 1, and ahead of/on release day seems sufficient for now.

[JeffPaul]

In parallel it might also be worthwhile to create a GHA from the project that would run a basic e2e test like you're doing here that plugin devs could make use of in their repos (and run daily, weekly, on every PR, or whenever based on their needs) that does a simple plugin activation and basic checks like you're suggesting here?

Perhaps the default workflow would run against trunk (so they get early alerts if a new release cycle is breaking something for them), latest (which is hopefully their tested-up-to version in most cases), and then whatever their stated plugin minimum is so that we're helping folks test against what's coming and their plugin min and max supported versions.

I tend to set up this very thing on most plugins I help maintain, so having something more reusable from the project (that folks could then extend by adding more custom e2e tests to match their plugin functionality) would make this a more easily repeatable process for folks?

Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.

While Tide is still running, its been generally unsupported for years and I've long considered trying to get it shut down as I'm unsure how many entities are even making use of its available audit data set (with the exception of the PHP Compat Checker plugin). If there was interest in having this functionality more formally supported within Tide and then in consuming the resulting data, then that seems great to me but we'd likely want to get someone / a team / a sponsor to help ensure Tide continues to be supported after this implementation completes.

[jeffr0]

While I applaud this effort, the reality is, the core dev team should not have to take any steps to make sure third-party plugins work with versions of WordPress that are in development. That’s supposed to be the third-party’s responsibility.

[JeffPaul]

@jeffr0 that may have unconsciously or inadvertently been the stance for some time now, but with repeated instances of WordPress site owners running into fatal errors after major WordPress updates its within the sphere of concern for those working on core to do what we can to help prevent those negative experiences for site owners. Being able to check that a popular plugin will fatal and try to preemptively resolve that feels like a worthwhile path to walk to try and help site owners avoid those negative experiences.

Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?

This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has Requires Plugins headers then pre-install the required plugins first to allow the original plugin to be tested.

[jorbin]

One thing I think should be required of each of the plugins that are used here is a blueprint.json file so that it is set up properly. We may need to modify things slightly so it uses multiple PHP versions (maybe just highest and lowest supported?)

Runs weekly against nightly on a schedule, plus on demand.

This feels like maybe a later phase, somewhat because this will be a LOT of GHA minutes to churn through this weekly and just because we have free access to them doesn't mean we should overuse them. I think allowing a Release Lead or their delegate to run at Beta 1, RC 1, and ahead of/on release day seems sufficient for now.

I think monthly and on demand would make sense. Monthly so that we can get feedback earlier in alpha cycles but doesn't eat through all the minutes. On demand, so we can kick this off after each pre-release version.

While I applaud this effort, the reality is, the core dev team should not have to take any steps to make sure third-party plugins work with versions of WordPress that are in development. That’s supposed to be the third-party’s responsibility.

Sadly, not all of the ecosystem has been responsible. This testing should help the users of WordPress which is ultimately what everyone wants.

_Claude ran the workflow at full scale and pulled this from the logs:_

Proof the check does its job - a full 200 plugin run on a fork caught a real fatal. eps-301-redirects 2.85 installs and activates cleanly, then fatals the moment WP-CLI loads WordPress with the plugin active:

Full run results: 186 passed, 1 failed, 13 skipped (WooCommerce add-ons with an unmet Requires Plugins header), in 4m32s wall time across 8 shards. Runs: ​https://github.com/adamsilverstein/wordpress-develop/actions/runs/32406588454 (nightly) and ​https://github.com/adamsilverstein/wordpress-develop/actions/runs/32407388297 (latest, pictured), triggered from ​https://github.com/adamsilverstein/wordpress-develop/pull/65 with the repository guards loosened, since manual dispatch is not available until the workflow exists on trunk.

Worth noting how the triage played out: the failure reproduces identically on latest and nightly, so it is a pre-existing plugin bug rather than a core regression. WP-CLI loads WordPress inside a function scope, so the plugin's file-level $EPS_Redirects_Plugin never becomes a real global; when init fires on a plain permalink install, plugin_resources() calls add_admin_message() on that null. That points at a useful follow-up for this workflow: automatically re-check any failure against the latest stable release, so nightly-only failures (likely core regressions) stand out from plugins that were already broken.

Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?

This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has Requires Plugins headers then pre-install the required plugins first to allow the original plugin to be tested.

Good suggestion, thanks!

_Claude took a swing at this one, here is where it landed:_

Done in 67005f527b. A plugin that declares Requires Plugins now gets those dependencies installed and activated before it is activated itself, so the four WooCommerce extensions that were being recorded as SKIPPED are actually tested.

The header is read with get_plugin_data() rather than a grep, so the same rules apply here as when core decides whether a plugin's requirements are met. Chains are followed a level at a time in case a dependency declares one of its own, and anything already installed is left alone, which is what terminates a circular declaration.

On the isolation worry from the original question: after the dependencies are activated and before the plugin under test is, the front page and login screen are requested on their own. A site already broken at that point is the dependency's doing, so the plugin under test is skipped rather than blamed for it, and the debug log is cleared at the same point for the same reason. The results table gains an "Also active" column so a failure can be read in context.

Tested in a container on PHP 8.3 with MySQL 8.4 and WP-CLI 2.12:

• woocommerce-payments, google-listings-and-ads, woocommerce-paypal-payments and woocommerce-gateway-stripe all install woocommerce and pass. Running the previous version of the script against the same setup still reports them as SKIPPED, so that is the change rather than a difference in the environment.
• Fixture plugins for the cases that are hard to find in the directory: a two level chain activates deepest first and passes, a dependency that is not on WordPress.org is skipped naming it, a dependency that fatals on load is skipped against the dependency, a dependency that fatals on the front end is caught by the baseline check, two plugins requiring each other terminate instead of looping, and a plugin that fatals on its own with a healthy dependency active still fails.
• The four deliberate fatal fixtures still fail for the right reason, and wp-content/plugins and active_plugins were clean after every run.

One thing that turned up while re-testing: instagram-feed 6.12.0 fatals on wp_loaded against 7.1 on a fresh database, same wp_get_image_editor( WP_Error ) trace as before but reached directly rather than through cron. The previous version of the script reports it identically, so it looks like a genuine plugin bug rather than a side effect of this change.

The job timeout went from 30 to 45 minutes, since activating something the size of WooCommerce once per extension is not free. Does that seem like the right trade, or would you rather cap how many dependencies get pulled in?

Testing the local runner turned up something that changes the story on eps-301-redirects, which I had held up earlier as proof this catches real fatals. Installing it by hand through wp-admin, nothing broke, so I had Claude dig into why the workflow disagreed.

_Claude chased the discrepancy down, here is what came back:_

The site is healthy with the plugin active. Front page 200, wp-login.php 200, /wp-admin/ 302, and no debug.log written at all. The only check that failed was the WP-CLI boot step.

WP-CLI requires wp-settings.php from inside WP_CLI\Runner->load_wordpress(), so a plugin's file-scope $var = ... becomes a local of that method rather than a global. A probe must-use plugin reporting from init shows it plainly:

HTTP:   global is visible
WP-CLI: global is MISSING

eps-301-redirects assigns $EPS_Redirects_Plugin = new EPS_Redirects_Plugin(); at plugin.php:904 and reads it back with global $EPS_Redirects_Plugin; inside plugin_resources() on init. Under WP-CLI that is null, so line 390 calls ->add_admin_message() on nothing.

Line 390 only runs when permalink_structure is empty, which is what a fresh wp core install leaves behind. Setting /%postname%/ makes wp eval print loaded-ok; setting it back to plain brings the fatal back.

So it is real plugin fragility, but not something a visitor can reach, and not a core regression. A workflow whose job is to catch what breaks real sites should not fail a plugin for it.

The WP-CLI boot check now runs last and records a fatal as a note rather than a failure. A plugin that is healthy over HTTP passes, with the reason in the Details column. Anything that genuinely fatals on load still fails, because it fails the HTTP checks first. The order matters for a second reason: the WP-CLI fatal is written to the debug log, which the log check would otherwise pick up.

Both directions were verified. eps-301-redirects on nightly now passes with the note and exits 0. A fixture must-use plugin that fatals on template_redirect and not under WP-CLI still fails with "The request to / returned HTTP 500" and exits 1.

Sorry for the noise on the earlier claim - the check works, but that particular plugin was the wrong example for it.

[sanayasir]

### QA Feedback — Rank Math SEO

I tested Rank Math SEO with the new WordPress version and did not encounter any compatibility or loading issues.

Test results:

• ✅ Plugin installed successfully
• ✅ Plugin activated successfully
• ✅ WordPress loaded without errors
• ✅ Frontend loaded successfully
• ✅ WP Admin loaded successfully
• ✅ No HTTP 5xx responses observed
• ✅ No PHP fatal errors observed
• ✅ No relevant errors found in the debug log

Result: PASS ✅

Based on this test environment, Rank Math SEO appears to be fully compatible with the new WordPress version. No issues were identified during the compatibility smoke test.