Opened 4 months ago
Last modified 4 months ago
#62171 new enhancement
Test suite: Eliminate flakiness from basic HTTP tests getting rate-limited.
Reported by: |
|
Owned by: | |
---|---|---|---|
Milestone: | Future Release | Priority: | normal |
Severity: | normal | Version: | 6.7 |
Component: | General | Keywords: | |
Focuses: | Cc: |
Description
A number of tests in the Tests_HTTP_Functions
class, such as test_head_404
, test_head_redirect
, and test_get_response_cookies
make basic HTTP calls to w.org
, wp.org
, and wordpress.org
to verify WordPress' core HTTP handling.
However, these requests frequently get rate-limited, possibly because they give the appearance of improper activity. For example, when running a matrix of PHPUnit tests, a single IP address from the Github Actions container makes repeated requests for /screenshots/3.9/awefasdfawef.jpg
- a resource that doesn't exist. The test asserts that the response is a 404.
Besides introducing conflating effects and network latency, these tests are inherently fragile because if that resource ever becomes available it will cause the tests to fail even though the code under test hasn't failed. The tests are indirect, measuring different code than they claim to, and asserting that the failure of the actual system under test represents a failure of the purported code under test.
These flakey tests frequently block and reject working, good, and positive contributions to the project, so we would benefit by updating the tests to avoid this.
One possible approach would be to spool up a web server inside the test suite, perhaps as a separate OS process, perhaps as a separate node in the container, which provides a method to deterministically respond as expected.
GET /index.php?status_code=404 HTTP/1.1 HTTP/1.1 404 Not Found
In the original Slack discussion it was also proposed to reuse existing test servers, such as:
- The server which
Requests
uses test-server - An existing PHP package like mock-webserver
- An incredibly basic PHP file served via
php -S
that only provides what the tests need, and is fully controlled in this repository.