#43863 closed defect (bug) (fixed)
Fix broken check in testcase class to prevent multisite-only or single site-only tests from being executed
Reported by: | flixos90 | Owned by: | flixos90 |
---|---|---|---|
Milestone: | 5.1 | Priority: | normal |
Severity: | normal | Version: | |
Component: | Build/Test Tools | Keywords: | has-patch |
Focuses: | multisite | Cc: |
Description
There have been issues reported (for example https://wordpress.slack.com/archives/C18723MQ8/p1524646274000249) about multisite-only tests incorrectly being executed in single site context and vice-versa.
While core's PHPUnit configuration for single-site ignores tests with a group of ms-required
and its multisite configuration ignores tests with a group of ms-excluded
, there is also a check present in the WP_UnitTestCase
class itself to skip those tests appropriately. This is for cases where the PHPUnit configuration doesn't ignore the respective groups, which is common in many plugins, most of which don't include a multisite-specific phpunit.xml
file (and even if they did, the ms-required
and ms-excluded
groups are a relatively recent change that don't necessarily have to be excluded in the PHPUnit configuration).
Long story short, if you run into this error, your PHPUnit configuration probably doesn't ignore the groups correctly. However, the actual problem is that the check in WP_UnitTestCase
doesn't work correctly at all, making the "fallback" unusable and broken.
WP_UnitTestCase
overrides the checkRequirements()
method from PHPUnit's test case class, retrieves the current test's doc annotations and checks against $annotations['group']
whether one of ms-required
or ms-excluded
is present to then appropriately skip the test. However, the "group" key in the $annotations
array (which comes from PHPUnit's getAnnotations()
method) will never be present since $annotations
always contains a "class" and a "method" key only - those then each contain a "group" key. So the code needs to be adjusted to reflect the correct format of the annotations array.
43863.diff fixes the issue by checking groups using the correct structure of the return value from PHPUnit's
getAnnotations()
. After diving into the PHPUnit codebase and some "blame" history, I noticed that that structure has always been like that, for now more than 9 years (when it was first introduced) - so there aren't any issues that would depend on certain PHPUnit versions.If you want to verify that this change actually fixes the issue, remove the
ms-required
group from your PHPUnit configuration file's exclude list and run the test suite (in non-multisite mode). Without the fix from this ticket, multisite-only tests are still executed, obviously causing fatal errors. With the fix from this ticket, they are correctly skipped.