Always Clean Install in PHPUnit Tests

[bpetty]

The unit tests only re-install WordPress from a previous run if using a different config (multisite vs single), it requires a DB upgrade (checking db_version), or wp-tests-config.php changes in any way (hashed). This was originally introduced from @nbachiyski with the ​test runner rewrite, and modified again by @gcorne to ​account for wp-tests-config.php.

The problem with doing this lies in these areas though:

• In transitioning these unit tests to run against the build version instead of src (and we do want to do this soon after this change), we should never leave extra files like .wp-tests-version in the build directory that could be accidentally packaged with WP even if it was ignored in SVN.
• We should always assume failure from previous unit test runs, and always build and test from a clean install. It doesn't matter if it's the same db_version and install type, anything could have gone wrong and leave us with incorrect testing results affected by previous runs.
• Most of the time, this is actually ineffective because testers should already be running both the single site, and multisite tests one after the other, causing a re-install with the current code anyway.

Besides the above though, re-installing WordPress takes less than a second compared with the 3 to 5 minutes the actual tests take anyway. It's not slowing down the tests any.

[bpetty]

By the way, the reason I ever looked into this was because I believe this is the reason why @nacin was ​having so much trouble reproducing a test failure just back in Sept. The problem we were investigating was a failure in wp_guess_url() (#25317) which was only used during installation, and the resulting values stored in the DB were the values triggering test failures with canonical URL tests. If the unit tests were recently run successfully before it was broken, they would continue running successfully even with the broken version until you cleared the database and ran the tests again, or changed your configuration.

[wonderboymusic]

In 26095:

Always clean install in PHPUnit Tests.

Props bpetty.
Fixes #25871.

[nbachiyski]

Sounds cool, I agree that this might cause weird trouble. The only problem I see is that this will make running individual tests much slower. My usual use case is to run a single test case or test method very very often while developing, in a typical TDD cycle.

The next time I do that I will see if there is a noticeable delay and if so, will open a new ticket.