Make WordPress Core

Opened 4 months ago

Last modified 4 months ago

#62171 new enhancement

Test suite: Eliminate flakiness from basic HTTP tests getting rate-limited.

Reported by: dmsnell's profile dmsnell Owned by:
Milestone: Future Release Priority: normal
Severity: normal Version: 6.7
Component: General Keywords:
Focuses: Cc:

Description

A number of tests in the Tests_HTTP_Functions class, such as test_head_404, test_head_redirect, and test_get_response_cookies make basic HTTP calls to w.org, wp.org, and wordpress.org to verify WordPress' core HTTP handling.

However, these requests frequently get rate-limited, possibly because they give the appearance of improper activity. For example, when running a matrix of PHPUnit tests, a single IP address from the Github Actions container makes repeated requests for /screenshots/3.9/awefasdfawef.jpg - a resource that doesn't exist. The test asserts that the response is a 404.

Besides introducing conflating effects and network latency, these tests are inherently fragile because if that resource ever becomes available it will cause the tests to fail even though the code under test hasn't failed. The tests are indirect, measuring different code than they claim to, and asserting that the failure of the actual system under test represents a failure of the purported code under test.

These flakey tests frequently block and reject working, good, and positive contributions to the project, so we would benefit by updating the tests to avoid this.

One possible approach would be to spool up a web server inside the test suite, perhaps as a separate OS process, perhaps as a separate node in the container, which provides a method to deterministically respond as expected.

GET /index.php?status_code=404 HTTP/1.1

HTTP/1.1 404 Not Found

In the original Slack discussion it was also proposed to reuse existing test servers, such as:

  • The server which Requests uses test-server
  • An existing PHP package like mock-webserver
  • An incredibly basic PHP file served via php -S that only provides what the tests need, and is fully controlled in this repository.

Change History (1)

This ticket was mentioned in Slack in #core by dmsnell. View the logs.


4 months ago

Note: See TracTickets for help on using tickets.