Make WordPress Core

Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#43863 closed defect (bug) (fixed)

Fix broken check in testcase class to prevent multisite-only or single site-only tests from being executed

Reported by: flixos90's profile flixos90 Owned by: flixos90's profile flixos90
Milestone: 5.1 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch
Focuses: multisite Cc:

Description

There have been issues reported (for example https://wordpress.slack.com/archives/C18723MQ8/p1524646274000249) about multisite-only tests incorrectly being executed in single site context and vice-versa.

While core's PHPUnit configuration for single-site ignores tests with a group of ms-required and its multisite configuration ignores tests with a group of ms-excluded, there is also a check present in the WP_UnitTestCase class itself to skip those tests appropriately. This is for cases where the PHPUnit configuration doesn't ignore the respective groups, which is common in many plugins, most of which don't include a multisite-specific phpunit.xml file (and even if they did, the ms-required and ms-excluded groups are a relatively recent change that don't necessarily have to be excluded in the PHPUnit configuration).
Long story short, if you run into this error, your PHPUnit configuration probably doesn't ignore the groups correctly. However, the actual problem is that the check in WP_UnitTestCase doesn't work correctly at all, making the "fallback" unusable and broken.

WP_UnitTestCase overrides the checkRequirements() method from PHPUnit's test case class, retrieves the current test's doc annotations and checks against $annotations['group'] whether one of ms-required or ms-excluded is present to then appropriately skip the test. However, the "group" key in the $annotations array (which comes from PHPUnit's getAnnotations() method) will never be present since $annotations always contains a "class" and a "method" key only - those then each contain a "group" key. So the code needs to be adjusted to reflect the correct format of the annotations array.

Attachments (1)

43863.diff (984 bytes) - added by flixos90 7 years ago.

Download all attachments as: .zip

Change History (4)

@flixos90
7 years ago

#1 @flixos90
7 years ago

  • Keywords has-patch added; needs-patch removed

43863.diff fixes the issue by checking groups using the correct structure of the return value from PHPUnit's getAnnotations(). After diving into the PHPUnit codebase and some "blame" history, I noticed that that structure has always been like that, for now more than 9 years (when it was first introduced) - so there aren't any issues that would depend on certain PHPUnit versions.

If you want to verify that this change actually fixes the issue, remove the ms-required group from your PHPUnit configuration file's exclude list and run the test suite (in non-multisite mode). Without the fix from this ticket, multisite-only tests are still executed, obviously causing fatal errors. With the fix from this ticket, they are correctly skipped.

#2 @flixos90
7 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 43005:

Tests: Skip multisite-only or single site-only tests correctly based on test doc annotations.

Without the ms-required and ms-excluded groups being marked as excluded in the PHPUnit configurations for the project, those groups were still executed, causing fatal errors. Checking against the groups in the correct structure of the array returned from PHPUnit's Testcase::getAnnotations() ensures that those tests are skipped properly.

Fixes #43863.

#3 @jorbin
6 years ago

  • Milestone changed from 5.0 to 5.1
Note: See TracTickets for help on using tickets.