Streamline tests with installation skips

[soulseekah]

In line with the recent and amazing #42282 feature I'm proposing we add an environment flag WP_TESTS_SKIP_INSTALL that does not reinstall WordPress. This shaves off at at least 5-10 seconds per run allowing almost instant feedback.

Usage: WP_TESTS_SKIP_INSTALL=1 phpunit ...

[johnbillion]

Have you had any problems running the tests in this manner? I spent some time a while ago trying to remove this system() call and I found that the tests would start failing after repeated runs (on the full suite, at least). There are probably some tests that don't reset the state as expected.

[soulseekah]

Indeed, some of the tests are non-repeatable on a dirty install as they either rely on clean state, or get messed up due to state left by following tests. This can be partially checked by adding --repeat 2 when running phpunit, which runs every test twice in a row.

However, the purpose of this skip install is to save time while developing a patch for a test case (or suite maybe, group). At least that's what I do, I don't run the whole test suite every while developing a patch, I just run a specific test. After I'm satisfied with the test, I run the whole suite.

So while developing I'd be doing WP_TESTS_SKIP_INSTALL=1 phpunit --filter test_ticket_43432, which takes a second or two, and then when I'm ready I run a final regular phpunit that takes 5-6 minutes making sure my patch doesn't break anything.

Over the years I've found myself adding variations of the skip install flag time and time again to help streamline development for core tests and plugin tests. In fact I wrote patches for GravityView and Gravity Forms fixing non-repeatable tests, and I've been running both suites without reinstalling core for months at a time without any issues whatsoever.

Coincidentally, using a dirty install while writing tests will ensure that a new test a developer is writing is repeatable and doesn't depend on a clean database or some quirky global state.

In short, full tests: long run, full reinstall, not much gain. Single test: short run, no reinstall, potentially around 50-80% time saved per run. Hope this makes sense.

[iandunn]

I love the idea of speeding up the process. I didn't notice any difference after applying the patch and using WP_TESTS_SKIP_INSTALL=1, though.

I'll try to find some time to dig into it and see if I'm doing something wrong, or if there are some other conditions needed for it to work.

[pento]

@iandunn: Have you had a chance to investigate this further?

[iandunn]

I haven't, and I don't think I'll have time in the near future :(

[pento]

It saves more like 1-2 seconds for me, but that's useful for quick test runs when you just want to run a small group of tests.

[pento]

In 44536:

Tests: Add support for a WP_TESTS_SKIP_INSTALL environment variable.

Defining WP_TESTS_SKIP_INSTALL=1 when running tests will skip the install step. While this shouldn't be used for full test runs, it's useful for saving time when running small groups of tests.

Props soulseekah.
Fixes #43432.