WordPress.org

Make WordPress Core

Opened 3 years ago

Closed 23 months ago

Last modified 10 months ago

#43432 closed enhancement (fixed)

Streamline tests with installation skips

Reported by: soulseekah Owned by: pento
Milestone: 5.1 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

In line with the recent and amazing #42282 feature I'm proposing we add an environment flag WP_TESTS_SKIP_INSTALL that does not reinstall WordPress. This shaves off at at least 5-10 seconds per run allowing almost instant feedback.

Usage: WP_TESTS_SKIP_INSTALL=1 phpunit ...

Attachments (1)

43432.diff (818 bytes) - added by soulseekah 3 years ago.

Download all attachments as: .zip

Change History (12)

@soulseekah
3 years ago

#1 @SergeyBiryukov
3 years ago

  • Milestone changed from Awaiting Review to 5.0

#2 @johnbillion
3 years ago

Have you had any problems running the tests in this manner? I spent some time a while ago trying to remove this system() call and I found that the tests would start failing after repeated runs (on the full suite, at least). There are probably some tests that don't reset the state as expected.

#3 @soulseekah
3 years ago

Indeed, some of the tests are non-repeatable on a dirty install as they either rely on clean state, or get messed up due to state left by following tests. This can be partially checked by adding --repeat 2 when running phpunit, which runs every test twice in a row.

However, the purpose of this skip install is to save time while developing a patch for a test case (or suite maybe, group). At least that's what I do, I don't run the whole test suite every while developing a patch, I just run a specific test. After I'm satisfied with the test, I run the whole suite.

So while developing I'd be doing WP_TESTS_SKIP_INSTALL=1 phpunit --filter test_ticket_43432, which takes a second or two, and then when I'm ready I run a final regular phpunit that takes 5-6 minutes making sure my patch doesn't break anything.

Over the years I've found myself adding variations of the skip install flag time and time again to help streamline development for core tests and plugin tests. In fact I wrote patches for GravityView and Gravity Forms fixing non-repeatable tests, and I've been running both suites without reinstalling core for months at a time without any issues whatsoever.

Coincidentally, using a dirty install while writing tests will ensure that a new test a developer is writing is repeatable and doesn't depend on a clean database or some quirky global state.

In short, full tests: long run, full reinstall, not much gain. Single test: short run, no reinstall, potentially around 50-80% time saved per run. Hope this makes sense.

#4 @iandunn
3 years ago

I love the idea of speeding up the process. I didn't notice any difference after applying the patch and using WP_TESTS_SKIP_INSTALL=1, though.

I'll try to find some time to dig into it and see if I'm doing something wrong, or if there are some other conditions needed for it to work.

#5 @jorbin
2 years ago

  • Milestone changed from 5.0 to 5.1

#6 @pento
23 months ago

@iandunn: Have you had a chance to investigate this further?

#7 @iandunn
23 months ago

I haven't, and I don't think I'll have time in the near future :(

#8 @pento
23 months ago

  • Owner set to pento
  • Status changed from new to assigned

#9 @pento
23 months ago

It saves more like 1-2 seconds for me, but that's useful for quick test runs when you just want to run a small group of tests.

#10 @pento
23 months ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 44536:

Tests: Add support for a WP_TESTS_SKIP_INSTALL environment variable.

Defining WP_TESTS_SKIP_INSTALL=1 when running tests will skip the install step. While this shouldn't be used for full test runs, it's useful for saving time when running small groups of tests.

Props soulseekah.
Fixes #43432.

This ticket was mentioned in Slack in #core by david.baumwald. View the logs.


10 months ago

Note: See TracTickets for help on using tickets.