Certificate verification failure causing PHP 8.3 and 8.4 workflows to fail

[johnbillion]

Since Saturday August 23rd the PHP 8.3 and 8.4 workflows are failing due to the test containers being unable to connect to MySQL/MariaDB.

Error: Failed to get current SQL modes. Reason: ERROR 2026 (HY000): TLS/SSL error: Certificate verification failure: The certificate is NOT trusted.

Several factors have changed since the last successful run (needs verifying):

• Actions runner container image (from Debian Bullseye to Trixie)
• PHP (from 8.4.10 to 8.4.11 and from 8.3.23 to 8.3.24)
• OpenSSL (from 3.0.16 to 3.5.11)
• Possibly more

Needs investigating to identify the root cause.

• Example failing workflow: ​https://github.com/WordPress/wordpress-develop/actions/runs/17176362547/job/48737885376
• Slack thread: ​https://wordpress.slack.com/archives/C02RQBWTW/p1755973957184769
• Runner container update: ​https://github.com/docker-library/php/pull/1596
• Last successful workflow run: ​https://github.com/WordPress/wordpress-develop/actions/runs/17159678620/job/48686559044
• Debugging PR by @desrosj: ​https://github.com/WordPress/wordpress-develop/pull/9595

Running list of things I've tried:

• setting PYSQL_CLIENT_FLAGS to MYSQLI_CLIENT_SSL_DONT_VERIFY_SERVER_CERT
• Setting skip-ssl in ​a custom cnf file.
• ​Upgrading Docker to the latest version before running scripts (the default on Ubuntu images is currently 28.0.4 where the latest is 28.3.3).
• Mounting the same php-config.ini file as the php container to the cli one.
• Including a basic self-signed certificate.

Trac ticket: https://core.trac.wordpress.org/ticket/63876

This temporarily pins the PHP 8.3 & 8.4 container digests that were generated on June 25, 2025 through the use of a docker-compose.override.yml file.

The image that these containers are built on through ​https://github.com/WordPress/wpdev-docker-images changed recently, resulting in failures trying to install WordPress within the environment.

Trac ticket: Core-63876

[johnbillion]

In 60660:

Build/Test Tools: Temporarily pin the PHP 8.3 and 8.4 container images to an earlier digest to work around issues with the newer images.

This pins the images to PHP 8.3.10 and 8.4.11 on Debian Bullseye pending further investigation into the root cause of the certificate verification failure affecting the connection to the database containers.

This also removes memcached from the test matrix pending further investigation into the missing Memcached executable.

Props desrosj, bernhard-reiter, SirLouen, johnbillion.

See #63876

https://core.trac.wordpress.org/changeset/60660

[desrosj]

I wanted to document some debugging findings with the help of ChatGPT. For everything below, I had [60659] checked out (the last commit prior to the temporary band-aid introduced in [60660]), 8.3-fpm as the value for LOCAL_PHP in my .env file, the default MySQL 8.4 version, and xDebug/Memcached disabled.

I started with these commands.

# From host:
docker compose exec cli bash

# Try a strict TLS connection using the same host as the local environment
mysql -h mysql -u root -p"$MYSQL_ROOT_PASSWORD" \
  --ssl-mode=VERIFY_IDENTITY \
  --ssl-ca=/etc/ssl/certs/ca-certificates.crt \
  -e 'SELECT VERSION(), @@sql_mode;'

This gave me an error: mysql: unknown variable 'ssl-mode=VERIFY_IDENTITY'. According to ChatGPT:

That error means your CLI is the MariaDB client, not Oracle’s MySQL client. MariaDB’s mysql does not support --ssl-mode=VERIFY_IDENTITY, so it treats it like a bogus “variable” and fails.

Interesting. Running mysql --version confirmed this: mysql from 11.8.2-MariaDB, client 15.2 for debian-linux-gnu (x86_64) using EditLine wrapper

The previous command to test the TLS connection needs to be changed a bit:

mysql -h mysql -u root -p"$MYSQL_ROOT_PASSWORD" \
  --ssl \
  --ssl-ca=/etc/ssl/certs/ca-certificates.crt \
  --ssl-verify-server-cert \
  -e 'SHOW STATUS LIKE "Ssl_cipher"; SELECT VERSION(), @@sql_mode;'

This outputs a similar error to the one in the description: ERROR 2026 (HY000): TLS/SSL error: self-signed certificate in certificate chain.

Next, I ran some commands to figure out what the mysql container is serving for a certificate:

# Fetch the chain and let OpenSSL verify against the system bundle
openssl s_client -connect mysql:3306 -starttls mysql -servername mysql -showcerts \
  -CAfile /etc/ssl/certs/ca-certificates.crt </dev/null 2>/dev/null | \
  awk '/^subject=|^issuer=|^Verify return code|BEGIN CERTIFICATE|END CERTIFICATE/ {print}'

# Also show negotiated cipher to confirm you reached TLS
openssl s_client -connect mysql:3306 -starttls mysql -servername mysql </dev/null 2>/dev/null | \
  awk '/Protocol|Cipher/ {print}'

This outputs the following:

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=CN=MySQL_Server_8.4.6_Auto_Generated_Server_Certificate
issuer=CN=MySQL_Server_8.4.6_Auto_Generated_CA_Certificate
Verify return code: 19 (self-signed certificate in certificate chain)

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384

Switching over to the mysql container using docker compose exec mysql bash, I next ran this: mysql -u root -p"$MYSQL_ROOT_PASSWORD" -e "SHOW VARIABLES LIKE 'ssl%'; SHOW STATUS LIKE 'Ssl_%';"

This output the following:

+---------------------------+-----------------+
| Variable_name             | Value           |
+---------------------------+-----------------+
| ssl_ca                    | ca.pem          |
| ssl_capath                |                 |
| ssl_cert                  | server-cert.pem |
| ssl_cipher                |                 |
| ssl_crl                   |                 |
| ssl_crlpath               |                 |
| ssl_fips_mode             | OFF             |
| ssl_key                   | server-key.pem  |
| ssl_session_cache_mode    | ON              |
| ssl_session_cache_timeout | 300             |
+---------------------------+-----------------+
+--------------------------------+--------------------------+
| Variable_name                  | Value                    |
+--------------------------------+--------------------------+
| Ssl_accept_renegotiates        | 0                        |
| Ssl_accepts                    | 11                       |
| Ssl_callback_cache_hits        | 0                        |
| Ssl_cipher                     |                          |
| Ssl_cipher_list                |                          |
| Ssl_client_connects            | 0                        |
| Ssl_connect_renegotiates       | 0                        |
| Ssl_ctx_verify_depth           | 18446744073709551615     |
| Ssl_ctx_verify_mode            | 5                        |
| Ssl_default_timeout            | 0                        |
| Ssl_finished_accepts           | 10                       |
| Ssl_finished_connects          | 0                        |
| Ssl_server_not_after           | Aug 31 17:48:40 2035 GMT |
| Ssl_server_not_before          | Sep  2 17:48:40 2025 GMT |
| Ssl_session_cache_hits         | 0                        |
| Ssl_session_cache_misses       | 0                        |
| Ssl_session_cache_mode         | SERVER                   |
| Ssl_session_cache_overflows    | 0                        |
| Ssl_session_cache_size         | 128                      |
| Ssl_session_cache_timeout      | 300                      |
| Ssl_session_cache_timeouts     | 0                        |
| Ssl_sessions_reused            | 0                        |
| Ssl_used_session_cache_entries | 0                        |
| Ssl_verify_depth               | 0                        |
| Ssl_verify_mode                | 0                        |
| Ssl_version                    |                          |
+--------------------------------+--------------------------+

CN=MySQL_Server_8.4.6_Auto_Generated_Server_Certificate/CN=MySQL_Server_8.4.6_Auto_Generated_CA_Certificate confirms that the database container is using the auto-generated certificate issued by a private, untrusted certificate authority at the time and date I first ran npm run env:start.

I've tried several ways to get the PHP/CLI containers to recognize and trust this custom certificate authority, but haven't had any luck. I'll follow up with documenting that next time I have a chance.

[desrosj]

Today a working meeting took place in the form of a Slack Huddle in the #core-build-test-tools channel (​full thread). In attendance was @desrosj, @johnbillion, @jorbin, and @davidbaumwald. The intention was to record the huddle, but it seems that is not supported by Slack.

Instead, here are some notes, and the ​full thread can be found in Slack.

Understanding so far

The upstream official Docker PHP images were recently updated, switching the Debian base image reference from bullseye to trixie. This change ​happened on August 7th, and these ​updated images were published on August 26th.

The workflow that publishes the wordpressdevelop images to Docker Hub began ​failing on July 12th due to the Debian 10 packages being moved to the archive (this was fixed in ​PR-173). Because fail-fast is set to true, no containers were published from July 12th to August 23rd when PR-173 was merged.

At that point, the upstream base image change began flowing into the local environment containers maintained by WordPress.

The MySQL/MariaDB containers auto-generate a self-signed certificate when they are spun up for the first time. Something related to the change of the underlying version of Debian is changing the expectations of the SSL certificate chain.

Some additional things that have been tried

The following approaches have been explored thanks to varying levels of input from different AI models, however they did not fix the issue fully or at all.

• Using a script to generate a certificate set locally using a trusted certificate authority, mounting the certificates to the appropriate places within the database containers, and telling the database to use those certificates.
• Downloading the relevant details about the auto-generated certificate/certificate authority from the database container and attempting to add the CA to the CLI container's trust store.
• Using a script to generate a certificate set using a locally generated certificate authority before mounting to the database container and adding the CA to the trust store for both the database and CLI containers.

Questions explored during the call

What changed in the new trixie base container vs. how it was previously in bullseye?

As expected, a ​lot changed between Debian 11 (bullsey) and 13 (Trixie). This felt too broad to begin the search.

Can different base images be used?

While the images based on bullseye were removed, the images based on Debian 12 (bookworm) still remain. A ​PR to the wpdev-docker-images repository was opened to specify Debian 12 (bookworm) as the base image to see if the problem was present. The bookworm-based images did not have the same issue, so the change causing the issue can be narrowed down further to between Debian 12 and 13.

Are the versions of OpenSSL different? And how do they differ? Could possibly be changing how strict the certificate chain must be.

Yes, the versions of OpenSSL were wildly different.

Does the failure also happen in PHP 8.5?

Yes the same issue happens in the wordpressdevelop/php:8.5-fpm images.

Which versions of Docker's official PHP images would be affected by this

The change to the underlying base image was made to the PHP 8.1-8.5 containers. So any of these versions should see the problem.

Why is the problem only surfacing in PHP 8.3 and 8.4?

Docker only seems to publish updates to PHP images when there is a change in the PHP version being included. Because ​8.1 & 8.2 are now only receiving security support, no changes have been made to the underlying PHP version for these images since the version of Debian was changed. However when (if) a security update is published for 8.1 and 8.2, new versions of the containers will be published and the issue would have presented for those versions.

Observations

The issue is only happening in the cli container and not the php one, despite both containers interacting with the mysql container. The cli image has the virtual-mysql-client installed while the php one does not.

As noted earlier in the ticket, mysql --version outputs the following within the cli container: mysql from 11.8.2-MariaDB, client 15.2 for debian-linux-gnu (x86_64) using EditLine wrapper. Before the failures began, the wordpressdevelop/cli image shipped with mysql Ver 15.1 Distrib 10.11.11-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper.

Looking into this further, it appears that ​MariaDB 11.4 included several changes to how SSL connections and certificate verification is handled. Notably:

• SSL is now enabled in the server by default.
• Clients now require SSL and have --ssl-verify-server-cert enabled by default (​MDEV-31857).
• Use --disable-ssl or --disable-ssl-verify-server-cert to revert to the old behavior.

This caused connections from the MariaDB client (always the same, even if connecting to a MySQL container because the client is installed within the wordpressdevelop/cli container) to the WP-CLI container because the certificate chain is no longer allowed to contain a self-signed certificate.

After some trial and error, instructing the MariaDB client to use ssl=0 fixes the issue, however ​WP-CLI uses the `--no-defaults` flag for `wp db` commands, which causes this config value to be ignored and the commands to continue to fail. Therefore the --defaults flag needs to be used when calling wp db commands.

The recent issues with the memcached container proved to be unrelated to the SSL issues. They were solved by ​wpdev-docker-images#181. Adding memcached jobs back to the test matrix resulted in those jobs passing.

In order to experience the original certificate chain issue, someone must be using the wordpressdevelop/cli Docker image OR an image with a mysql client running 11.4.1-MariaDB or higher AND relying on the default auto-generated certificate within the database container OR implementing a custom SSL cert that was not issued by a trusted certificate authority.

A side effect of this fix is that the Tests_Image_Editor_Imagick::test_resizes_are_small_for_16bit_images and Tests_Image_Editor_Imagick::test_png_color_type_is_preserved_after_resize test methods are now failing for an unknown reason. The plan is to temporarily skip these when the test suite runs on PHP >= 8.1 until the cause can be determined. The tests are not completely disabled because they will run on PHP 7.2-8.0.

[desrosj]

Here are some other related links that were shared:

• ​https://peterkellner.net/2023-08-02-enforcing-ssl-connections-in-mysql-using-docker/
• ​https://dev.to/keploy/securing-databases-with-ssl-in-docker-containers-44h2
• ​https://techsparx.com/software-development/docker/damp/mysql-ssl-connection.html

https://core.trac.wordpress.org/changeset/60735

[johnbillion]

Fixed in [60735].

[desrosj]

Reopening because this will need to be backported to all versions with support for PHP 8.1 or higher.

Trac ticket: Core-63876

Trac ticket: Core-63876

This needs to be combined with [60736] so the tests will pass in the branch. Want to do it in the same PR/commit?

@johnbillion Yes, that makes sense. Added r60736 here as well.

This backports the following to restore a passing build in the 6.7 branch:

• b6e708d9ebddf58005c9c1c911ab13391ad14fe9 for fixing the TLS/SSL connection issue in newer PHP 8.x containers (see Core-63876).
• 9ca38ce47b7a8a9c9e916e6aff39ec772ccbba55 to skip add media tests that currently fail on ImageMagick 7.x, and improves error handling within tests (see Core-63932).

Trac ticket: Core-63876

This backports the following to restore a passing build in the 6.7 branch:

• b6e708d9ebddf58005c9c1c911ab13391ad14fe9 for fixing the TLS/SSL connection issue in newer PHP 8.x containers (see Core-63876).
• 9ca38ce47b7a8a9c9e916e6aff39ec772ccbba55 to skip add media tests that currently fail on ImageMagick 7.x, and improves error handling within tests (see Core-63932).

Trac ticket: Core-63876

This backports the following to restore a passing build in the 6.7 branch:

• b6e708d9ebddf58005c9c1c911ab13391ad14fe9 for fixing the TLS/SSL connection issue in newer PHP 8.x containers (see Core-63876).
• 9ca38ce47b7a8a9c9e916e6aff39ec772ccbba55 to skip add media tests that currently fail on ImageMagick 7.x, and improves error handling within tests (see Core-63932).
• A change to checkout a specific version of the WordPress Importer plugin on PHP < 7.2. Support for these versions has been dropped.

This backports the following to restore a passing build in the 6.7 branch:

• b6e708d9ebddf58005c9c1c911ab13391ad14fe9 for fixing the TLS/SSL connection issue in newer PHP 8.x containers (see Core-63876).
• 9ca38ce47b7a8a9c9e916e6aff39ec772ccbba55 to skip add media tests that currently fail on ImageMagick 7.x, and improves error handling within tests (see Core-63932).
• A change to checkout a specific version of the WordPress Importer plugin on PHP < 7.2. Support for these versions has been dropped (Core-63983)

[desrosj]

In 60744:

Build/Test Tools: Fix tooling and PHPUnit test issues.

This fixes the PHPUnit test suite for the 6.8 branch by:

• Fixing issues with the env:install script when using newer PHP images.
• Temporarily disables failing tests with ImageMagick 7 is in use.

Backports [60735] and [60736] to the 6.8 branch.

See #63876, #63932.

Merged in https://core.trac.wordpress.org/changeset/60744.

[desrosj]

In 60745:

Build/Test Tools: Fix tooling and PHPUnit test issues.

This fixes the PHPUnit test suite for the 6.7 branch by:

• Fixing issues with the env:install script when using newer PHP images.
• Temporarily disables failing tests with ImageMagick 7 is in use.

Backports [60735] and [60736] to the 6.7 branch.

See #63876, #63932.

https://core.trac.wordpress.org/changeset/60745

[desrosj]

In 60746:

Build/Test Tools: Fix tooling and PHPUnit test issues.

This fixes the PHPUnit test suite for the 6.6 branch by:

• Fixing issues with the env:install script when using newer PHP images.
• Temporarily disables failing tests with ImageMagick 7 is in use.

Backports [60735] and [60736] to the 6.6 branch.

See #63876, #63932.

https://core.trac.wordpress.org/changeset/60746

[desrosj]

Missed the actual reopen button before.

[desrosj]

In 60747:

Build/Test Tools: Fix tooling and PHPUnit test issues.

This fixes the PHPUnit test suite for the 6.5 branch by:

• Fixing issues with the env:install script when using newer PHP images.
• Temporarily disables failing tests with ImageMagick 7 is in use.

Backports [60735] and [60736] to the 6.5 branch.

See #63876, #63932.

Fixed in r60748.

[desrosj]

In 60749:

Build/Test Tools: Fix tooling and PHPUnit test issues.

This fixes the PHPUnit test suite for the 6.4 branch by:

• Fixing issues with the env:install script when using newer PHP images.
• Temporarily disables failing tests with ImageMagick 7 is in use.

Backports [60735] and [60736] to the 6.4 branch.

See #63876, #63932.

Fixed with r60749 and 460750

[jorbin]

6.8.3 was a security only release. Punting though 6.8.4 is not planned as a release