WordPress.org

Make WordPress Core

Opened 5 months ago

Closed 5 months ago

Last modified 5 months ago

#25871 closed enhancement (fixed)

Always Clean Install in PHPUnit Tests

Reported by: bpetty Owned by: wonderboymusic
Milestone: 3.8 Priority: normal
Severity: normal Version:
Component: Unit Tests Keywords: has-patch
Focuses: Cc:

Description

The unit tests only re-install WordPress from a previous run if using a different config (multisite vs single), it requires a DB upgrade (checking db_version), or wp-tests-config.php changes in any way (hashed). This was originally introduced from @nbachiyski with the test runner rewrite, and modified again by @gcorne to account for wp-tests-config.php.

The problem with doing this lies in these areas though:

  • In transitioning these unit tests to run against the build version instead of src (and we do want to do this soon after this change), we should never leave extra files like .wp-tests-version in the build directory that could be accidentally packaged with WP even if it was ignored in SVN.
  • We should always assume failure from previous unit test runs, and always build and test from a clean install. It doesn't matter if it's the same db_version and install type, anything could have gone wrong and leave us with incorrect testing results affected by previous runs.
  • Most of the time, this is actually ineffective because testers should already be running both the single site, and multisite tests one after the other, causing a re-install with the current code anyway.

Besides the above though, re-installing WordPress takes less than a second compared with the 3 to 5 minutes the actual tests take anyway. It's not slowing down the tests any.

Attachments (1)

ticket-25871-reinstall-tests.patch (1.1 KB) - added by bpetty 5 months ago.

Download all attachments as: .zip

Change History (5)

comment:1 SergeyBiryukov5 months ago

  • Milestone changed from Awaiting Review to 3.8

comment:2 bpetty5 months ago

By the way, the reason I ever looked into this was because I believe this is the reason why @nacin was having so much trouble reproducing a test failure just back in Sept. The problem we were investigating was a failure in wp_guess_url() (#25317) which was only used during installation, and the resulting values stored in the DB were the values triggering test failures with canonical URL tests. If the unit tests were recently run successfully before it was broken, they would continue running successfully even with the broken version until you cleared the database and ran the tests again, or changed your configuration.

comment:3 wonderboymusic5 months ago

  • Owner set to wonderboymusic
  • Resolution set to fixed
  • Status changed from new to closed

In 26095:

Always clean install in PHPUnit Tests.

Props bpetty.
Fixes #25871.

comment:4 nbachiyski5 months ago

Sounds cool, I agree that this might cause weird trouble. The only problem I see is that this will make running individual tests much slower. My usual use case is to run a single test case or test method very very often while developing, in a typical TDD cycle.

The next time I do that I will see if there is a noticeable delay and if so, will open a new ticket.

Note: See TracTickets for help on using tickets.