WordPress.org

Make WordPress Core

Opened 4 months ago

Last modified 3 months ago

#43432 new enhancement

Streamline tests with installation skips

Reported by: soulseekah Owned by:
Milestone: 5.0 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

In line with the recent and amazing #42282 feature I'm proposing we add an environment flag WP_TESTS_SKIP_INSTALL that does not reinstall WordPress. This shaves off at at least 5-10 seconds per run allowing almost instant feedback.

Usage: WP_TESTS_SKIP_INSTALL=1 phpunit ...

Attachments (1)

43432.diff (818 bytes) - added by soulseekah 4 months ago.

Download all attachments as: .zip

Change History (5)

@soulseekah
4 months ago

#1 @SergeyBiryukov
4 months ago

  • Milestone changed from Awaiting Review to 5.0

#2 @johnbillion
3 months ago

Have you had any problems running the tests in this manner? I spent some time a while ago trying to remove this system() call and I found that the tests would start failing after repeated runs (on the full suite, at least). There are probably some tests that don't reset the state as expected.

#3 @soulseekah
3 months ago

Indeed, some of the tests are non-repeatable on a dirty install as they either rely on clean state, or get messed up due to state left by following tests. This can be partially checked by adding --repeat 2 when running phpunit, which runs every test twice in a row.

However, the purpose of this skip install is to save time while developing a patch for a test case (or suite maybe, group). At least that's what I do, I don't run the whole test suite every while developing a patch, I just run a specific test. After I'm satisfied with the test, I run the whole suite.

So while developing I'd be doing WP_TESTS_SKIP_INSTALL=1 phpunit --filter test_ticket_43432, which takes a second or two, and then when I'm ready I run a final regular phpunit that takes 5-6 minutes making sure my patch doesn't break anything.

Over the years I've found myself adding variations of the skip install flag time and time again to help streamline development for core tests and plugin tests. In fact I wrote patches for GravityView and Gravity Forms fixing non-repeatable tests, and I've been running both suites without reinstalling core for months at a time without any issues whatsoever.

Coincidentally, using a dirty install while writing tests will ensure that a new test a developer is writing is repeatable and doesn't depend on a clean database or some quirky global state.

In short, full tests: long run, full reinstall, not much gain. Single test: short run, no reinstall, potentially around 50-80% time saved per run. Hope this makes sense.

#4 @iandunn
3 months ago

I love the idea of speeding up the process. I didn't notice any difference after applying the patch and using WP_TESTS_SKIP_INSTALL=1, though.

I'll try to find some time to dig into it and see if I'm doing something wrong, or if there are some other conditions needed for it to work.

Note: See TracTickets for help on using tickets.