WordPress.org

Make WordPress Core

Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#30304 closed enhancement (fixed)

external-http automated tests should not run by default

Reported by: boonebgorges Owned by: boonebgorges
Milestone: 4.1 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

A number of our automated tests in the groups external-http and http make requests to external URLs. These tests take a long time to run (in excess of 25% of total running time for the suite) and are highly dependent on having a good internet connection. As such, their presence is a disincentive from running the entire test suite when patching/committing. Moreover, our HTTP functionality is fairly isolated, so that it's pretty unlikely to be affected by most core work. This means that running this particular set of tests is of marginal benefit to most devs.

As suggested by nacin at WCSF, we should stop running these tests on a default call to phpunit, like we already do with the extremely slow ajax tests. Here's what needs to be done:

  • Exclude the external-http group in phpunit.xml.dist manifest.
  • Make sure that the external-http group contains all and only those tests that make external requests.
  • Add a task to Gruntfile.js for the external-http group, so it gets run on grunt phpunit and in CI.

Change History (5)

#1 @boonebgorges
7 years ago

  • Owner set to boonebgorges
  • Status changed from new to accepted

The only test outside of the external-http group that makes an external request appears to be test_youtube_com_secure_embed(). It looks to me like that test is a bit more fragile than those that test WP_Http functionality more directly, which is to say I think it's probably wise not to include them in the external-http group for the time being.

#2 @boonebgorges
7 years ago

  • Resolution set to fixed
  • Status changed from accepted to closed

In 30298:

Exclude external-http tests when running phpunit.

The external-http tests are very slow, and Wp_Http functionality is fairly
isolated, so the benefits of skipping these tests by default outweigh the
risks.

A grunt phpunit:external-http subtask has been added, to ensure that the
tests are executed during exhaustive runs of the test suite, such as in
continuous integration.

Fixes #30304.

#3 @rmccue
7 years ago

FWIW, ran into the same problem with Requests testing. We now have a local testing server using only PHP's built-in server to run it, which mirrors most of httpbin.org. Might be worth switching what we can to use that instead.

#4 follow-up: @jorbin
7 years ago

The php built in server is only available in 5.4+

I like the solution of not including it in default runs. Great job boonegorges.

#5 in reply to: ↑ 4 @rmccue
7 years ago

Replying to jorbin:

The php built in server is only available in 5.4+

Indeed; for the specific case of Travis, it's easy to solve, since all versions of PHP are installed at once. If you're not running the local server, we then fall back to httpbin.org, so it's just an added speed boost.

Just something to think about :)

Note: See TracTickets for help on using tickets.