Make WordPress Core

Opened 7 weeks ago

Closed 3 weeks ago

Last modified 3 weeks ago

#65722 closed task (blessed) (fixed)

Retry transient Docker image pull failures in PHPUnit workflows

Reported by: lancewillett Owned by: lancewillett
Priority: normal Milestone: 7.1
Component: Build/Test Tools Version:
Severity: normal Keywords: has-patch fixed-major
Cc: Focuses:

Description

The reusable PHPUnit workflow starts its Docker environment through npm run env:start. Docker Compose pulls the required images during that step.

Problem

A transient Docker Hub connection timeout currently aborts the job immediately. Reported errors include:

Client.Timeout exceeded while awaiting headers
context deadline exceeded

The affected image varies, indicating a registry or network failure rather than an image-specific problem. A large matrix increases the chance that one failed pull fails the run.

Proposed change

Add an explicit image-pull step immediately before environment startup.

The step:

  • Pulls wordpress-develop, php, mysql, and cli.
  • Adds memcached only when enabled.
  • Makes at most three pull attempts.
  • Waits 10 seconds after the first failure and 20 seconds after the second.
  • Reports the error and exits after the third failure.

Successfully pulled images remain locally cached for env:start.

Verification

A real Docker Hub timeout cannot be triggered reliably.

Verification completed:

  • Simulated a failed first attempt followed by a successful real pull.
  • Confirmed three simulated failures stopped after exactly three attempts.
  • Confirmed memcached was included only when enabled.
  • Started a PHP 8.4 and MySQL 8.4 environment using the cached images.
  • Completed npm run env:install.
  • Passed actionlint and workflow whitespace checks.

Possible follow-up

Docker Hub authentication could provide stronger protection and higher pull limits. That requires repository credentials and should be considered separately.

Change History (11)

This ticket was mentioned in PR #12703 on WordPress/wordpress-develop by @lancewillett.


7 weeks ago
#1

  • Keywords has-patch added

## Summary

This pre-pulls the Docker images used by the reusable PHPUnit workflow before env:start.

The pull retries at most three times, waiting 10 seconds after the first failure and 20 seconds after the second.

The service list matches the existing Compose context. It pulls wordpress-develop, php, mysql, and cli, adding memcached only when enabled.

## Why

Transient Docker Hub connection timeouts currently fail the affected matrix job immediately. The affected image varies, indicating a registry or network failure rather than an image-specific
problem.

Successfully pulled images remain cached for environment startup.

The real timeout is nondeterministic and was not reproduced.

## Testing

  • Passed actionlint and workflow whitespace checks.
  • Simulated a failed first attempt followed by a successful real pull.
  • Confirmed three simulated failures stopped after exactly three attempts.
  • Confirmed memcached was included only when enabled.
  • Started a PHP 8.4 and MySQL 8.4 environment using the pre-pulled images.
  • Confirmed MySQL became healthy.
  • Completed npm run env:install.

## Possible follow-up

Docker Hub authentication could provide stronger protection and higher pull limits. That requires repository credentials and should be considered separately.

#2 @johnbillion
7 weeks ago

  • Milestone Awaiting Review7.1
  • Type defect (bug)task (blessed)

#3 @lancewillett
7 weeks ago

  • Owner set to lancewillett
  • Resolutionfixed
  • Status newclosed

In 62857:

Build/Test Tools: Retry Docker image pulls in PHPUnit workflows.

Transient Docker registry failures (Docker Hub / GHCR pull timeouts and "premature close" errors) intermittently fail the PHPUnit jobs. Wrap the image pulls in a bounded retry with backoff.

Developed in: https://github.com/WordPress/wordpress-develop/pull/12703

Props barry, adrianmoldovanwp
Fixes #65722

This ticket was mentioned in PR #12937 on WordPress/wordpress-develop by @desrosj.


5 weeks ago
#4

This moves the retry mechanism added in r62857 from the GitHub Actions workflow to the env:pull command, which is a benefit any time the command is run, not just in the PHPUnit workflow.

Trac ticket: Core-65722

## Use of AI Tools

Claude Code created the initial draft of this PR.

#5 @desrosj
5 weeks ago

  • Keywords fixed-major added
  • Resolution fixed
  • Status closedreopened

I'm reopening this to consider my PR with a suggestion to improve this.

Also, I think we should consider backporting this. In the past, it's been quite difficult and time consuming to maintain old branches because the state of the tooling was drastically different in multiple branches. We should try to keep everything in a consistent state as much as possible back through 4.7.

To make this easier and not require individual backport commits for every branch, we've been grouping changes into a single commit after backporting to the current maintained numbered branch, similar to the process the Security Team has been following.

#6 @desrosj
5 weeks ago

  • Resolutionfixed
  • Status reopenedclosed

In 63163:

Build/Test Tools: Move env:pull retry logic.

This improves the retry mechanism added in [62857] for the env:pull command by mvoing the logic from the PHPUnit test workflow into the docker.js script so that every env:pull command benefits reagardless of context.

Follow up to [62857].

Props desrosj, irozum, jorbin.
Fixes #65722.

#7 @desrosj
5 weeks ago

  • Resolution fixed
  • Status closedreopened

Reopening for deep backporting consideration.

#8 @wildworks
4 weeks ago

The 7.1 dry run will start in about 16 hours. As this PR concerns the development environment, it should not be a blocker for the dry run.

#9 @desrosj
3 weeks ago

  • Milestone 7.1

Ticket retargeted after milestone closed

#10 @desrosj
3 weeks ago

  • Resolutionfixed
  • Status reopenedclosed

I am going to close this out. Let's open up a follow up ticket to backport.

#11 @desrosj
3 weeks ago

  • Milestone7.1
Note: See TracTickets for help on using tickets.