#34935 closed defect (bug) (fixed)
Removed SSL certificates causing errors in WP 4.4
Reported by: | DvanKooten | Owned by: | rmccue |
---|---|---|---|
Milestone: | 4.4.1 | Priority: | normal |
Severity: | normal | Version: | 4.4 |
Component: | HTTP API | Keywords: | has-patch https commit |
Focuses: | Cc: |
Description
In #30434, a bunch of certificates were removed from the bundled root certificate file.
Unfortunately, it seems this is forcing plugin author to disable sslverify
again as certain versions of CURL / OpenSSL are not liking the change.
I'm still working out the exact configuration that's causing the issue to pop-up but all I know is that disabling sslverify
in WP_HTTP wasn't necessary for WordPress 3.7 up to WordPress 4.3.x.
User-submitted error reports:
https://wordpress.org/support/topic/error-connecting-to-mailchimp-ssl-certificate-problem?replies=8
http://myonlinesecurity.co.uk/wordpress-4-4-update-breaks-itself-with-ssl-certificate-problem-unable-to-get-local-issuer-certificate/
https://github.com/ibericode/mailchimp-for-wordpress/issues/219
Attachments (1)
Change History (29)
#2
@
9 years ago
I feared this would come up again.. Unfortunately the curl cacert bundles are seemingly sometimes incompatible with older versions of curl for an unknown reason.. It's a bug deep within curl (possibly fixed at some point) which I don't think we can easily work around.
Can everyone please include details of the curl version, openssl version (or whatever ssl library curl is using) AND the php version affected.
#3
@
9 years ago
And to clarify, I don't think it's the removal, rather the adding or reordering of certs. We had this issue before, and were able to fix it by moving one certificate to the start of the file, it's trial and error to find which one though.
#5
@
9 years ago
I see many people complaining about it in forums of my plugins... But everyone ends up solving by updating or installing again cURL and OpenSSL.
#7
@
9 years ago
Some of my users are posting server details in this support topic on w.org
PHP: 5.5 (cPanel) cURL: 7.19.7 OpenSSL: 1.0.1e
Apache: 2.4.12 (Win32) PHP: 5.6.8 MySQL: 5.6.24 cURL: 7.40.0 OpenSSL: 1.0.1l
PHP: 5.3.8 CentOS release 5.5 (Final) CURL: 7.15.5 OpenSSL: OpenSSL/0.9.8b
Older versions of OpenSSL seem to be the only consistent thing, as CURL versions are all over the place (from 7.19.x up to 7.40.x) as are PHP versions.
#8
@
9 years ago
I think this might be https://github.com/openssl/openssl/commit/6c03af135b285429a71ab3dac953ad9a70d8a1ac but not sure yet. Will try and debug ASAP.
This ticket was mentioned in Slack in #core by rmccue. View the logs.
9 years ago
#10
follow-up:
↓ 24
@
9 years ago
- Keywords has-patch needs-testing added
- Owner set to rmccue
- Status changed from new to assigned
Managed to replicate using OpenSSL only from the command line. Appears to be broken in 1.0.1e and fixed in 1.0.1q, so some git bisect
magic lead me to this commit. This then lead me on to this ticket; this comment goes into specifics about it: (Use guest/guest as username/password)
Recently, Mozilla has started to cleanup the Mozilla CA trust list and
remove CA certificates that use a weaker 1024-bit RSA key. I'll call
them legacy CAs.
When upgrading the trust store used by openssl to exclude the legacy CA
certificates, by default, openssl (e.g. s_client) can no longer verify
the server certificate of several popular SSL/TLS servers, examples are
www.flickr.com and www.amazon.com.
The cURL page for the certificate bundle also mentions this:
RSA-1024 removed
Around early September 2014, Mozilla removed the trust bits from the certs in their CA bundle that were still using RSA 1024 bit keys. This may lead to TLS libraries having a hard time to verify some sites if the library in question doesn't properly support "path discovery" as per RFC 4158. (That includes OpenSSL and GnuTLS.)
This file is linked as the last version built from the NSS store before they started removing these certs.
These certificates have been removed for being 1024-bit:
- GTE CyberTrust Global Root
- Thawte Server CA
- Thawte Premium Server CA
- Verisign Class 3 Public Primary Certification Authority
- Verisign Class 3 Public Primary Certification Authority - G2
- ValiCert Class 1 VA
- ValiCert Class 2 VA
- ValiCert Class 3 VA (incorrectly called RSA Root Certificate 1)
- Entrust.net Secure Server CA
- Equifax Secure Global eBusiness CA
- Equifax Secure eBusiness CA 1
- NetLock Business (Class B) Root
- NetLock Express (Class C) Root
- Verisign Class 3 Public Primary Certification Authority
(Importantly, note that "Verisign Class 3 Public Primary Certification Authority - G2" is the current root certificate for api.paypal.com
.)
After some more checking, the removed ones are all 1024 bit, with the exception of the following (links to reasoning for removal):
- RSA Root Certificate 1
- America Online Root Certification Authority 1
- America Online Root Certification Authority 2
- TDC Internet Root CA
- AC Raíz Certicámara S.A.
- TC TrustCenter Class 3 CA II
- E-Guven Kok Elektronik Sertifika Hizmet Saglayicisi
Therefore: I believe we should be able to take our existing bundle, and pull the 1024 bit certificates back in.
Attaching patched version of the CA bundle that adds the 1024 bit certificates back; this fixes resolution for me via OpenSSL on the command line, but needs testing on a site that's broken.
#11
@
9 years ago
Feedback from one customer who was affected:
Me: Alternatively, you can replace the wp-includes/certificates/ca-bundle.crt file with the one I attached. Please make a copy of the original one first. If you're on 4.4, this file should fix the issue. Let me know whether it does or doesn't.
Customer: OK, so that fixed the SSL error.
Another feedback:
Good news, i have updated to wo 4.4 applied the new certificate and all working now.
Exact steps i followed were:
Deactivated all plugins
Performed the wp automated updated.
Then replaced the ca-bundle.crt with the new one u gave
Next activated woo commerce
Made a dummy purchase -- All worked
Next activated subscriptions
Made a dummy purchase -- All worked
Activated all other plugins
Made a dummy purchase -- All worked
Deactivated plugin- Temporary SSLverify Disable for WooCommerce Subscriptions (sets sslverify to false)
Made a dummy purchase -- All worked
So that is very good news indeed - thanks for your help in solving that
#12
follow-up:
↓ 13
@
9 years ago
Hello guys. I'm seeing this issue when trying to receive IPN feedback from PayPal in a WP 4.4 running on PHP 5.5. Do you have any ca-bundle.crt file I can betatest please?
#13
in reply to:
↑ 12
;
follow-up:
↓ 15
@
9 years ago
Replying to Kent Brockman:
Hello guys. I'm seeing this issue when trying to receive IPN feedback from PayPal in a WP 4.4 running on PHP 5.5. Do you have any ca-bundle.crt file I can betatest please?
Hi Kent,
@rmccue's fix worked for a customer for the same issue. You'd need to check out the wordpress development svn branch and apply the patch.
#14
in reply to:
↑ description
@
9 years ago
ca-bundle.crt resolved the issue here:
cURL 7.19.7
OpenSSL 1.0.1e-fips 11 Feb 2013
and here:
cURL 7.24.0
OpenSSL 1.0.1e-fips 11 Feb 2013
#15
in reply to:
↑ 13
@
9 years ago
Thanks! For those of us not using git nor svn and that could land here, you can use the following file as ca-bundle.crt and I can confirm my problems were solved after using it:
https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt
Hope it helps others. Best regards
#17
follow-ups:
↓ 18
↓ 21
@
9 years ago
- Keywords changed from has-patch, needs-testing, https to has-patch needs-testing https
Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release. Using the test results to allow WordPress to detect incompatible versions of cURL and OpenSSL on the server, and warn the WordPress admin to have their host update cURL or OpenSSL, would provide admins actionable feedback, so they can solve their own issue, even if the ca-bundle.crt isn't perfect.
#18
in reply to:
↑ 17
;
follow-up:
↓ 19
@
9 years ago
Replying to toddlahman:
Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release. Using the test results to allow WordPress to detect incompatible versions of cURL and OpenSSL on the server, and warn the WordPress admin to have their host update cURL or OpenSSL, would provide admins actionable feedback, so they can solve their own issue, even if the ca-bundle.crt isn't perfect.
Following feedback from users who had this problem and we asked them to update openssl and / or curl, they came back with that being fairly hard, as they're part of the server distribution they used.
One of them provided feedback from the host: they updated to 1.0.2, but php did not pick that change up. It's a lot easier to bundle a working ca-bundle.crt than asking hosts to update their openssl.
#19
in reply to:
↑ 18
@
9 years ago
Absolutelly agree with @javorszky: people using shared hosting will be trapped in a cage because vast majority of hosting providers are not able/willing to upgrade their OS or templates due to just a few shared hosting customer complaints.
A well implemented ca-bundle.crt file demonstrated to do the magic. The tricky part is... keeping that .crt file up to date, given the lots of updates that may arise. All in all, is a foolproof solution. I bet that monitoring changes in a couple of git ca bundles (like this one: https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt ) will be enough to keep up with it.
#20
@
9 years ago
Hosting provider input here: Very standard cPanel shared hosting environment, running CentOS 6.7 OS with yum updates, as well as cPanel updates, every night. OpenSSL reports as version "1.0.1e-fips" but has backported patches (the most recent being June 23, 2015).
We had one customer running WooCommerce get bit by this ca-bundle issue. Specifically, a paid plugin for the Elavon payment gateway (purchased from WooCommerce, now owned by Automattic) generating 500 errors. WooCommerce tech support blamed our server configuration and suggested we "reinstall OpenSSL and Curl". I did not do that... however, after I updated ca-bundle.crt, the problem was immediately corrected.
If Automattic thinks this issue is a shared-hosting provider problem, then they should immediately contact cPanel tech support. There are likely hundreds of thousands of web servers set up exactly like ours, in this very standard shared hosting configuration.
#21
in reply to:
↑ 17
@
9 years ago
- Keywords commit added; needs-testing removed
Replying to toddlahman:
Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release.
It is a huge amount of effort to test different cURL and OpenSSL versions (as various versions need to be used); finding and verifying this bug took me ~4 hours, even using git bisect
. Thankfully, this issue turned out to be an easy one caused by OpenSSL itself, which meant I only needed to rebuild OpenSSL and use the command line client openssl s_client
; if this was caused by OpenSSL and cURL (or worse, PHP cURL) having issues together, it could have taken days just to find the issue.
Testing every version of cURL and OpenSSL that's available isn't something we can do, which is why it's important for people to test release candidates.
Replying to Kent Brockman:
A well implemented ca-bundle.crt file demonstrated to do the magic. The tricky part is... keeping that .crt file up to date, given the lots of updates that may arise. All in all, is a foolproof solution. I bet that monitoring changes in a couple of git ca bundles (like this one: https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt ) will be enough to keep up with it.
We do monitor changes in the bundle, which is what caused this issue in the first place. The key of the issue is that Mozilla has decided to remove 1024 bit root certificates from their root (that bundle file is generated from it). Mozilla's software and infrastructure uses NSS (a different SSL library) that can handle alternate chains; OpenSSL 1.0.1l was the first version to introduce this to OpenSSL.
Monitoring the upstream changes isn't enough when we also need to have compatibility with years old versions of OpenSSL.
Replying to sneader:
Hosting provider input here: Very standard cPanel shared hosting environment, running CentOS 6.7 OS with yum updates, as well as cPanel updates, every night. OpenSSL reports as version "1.0.1e-fips" but has backported patches (the most recent being June 23, 2015).
It may be worth reporting this to CentOS, since 1024 bit root certificates are going away eventually. This is going to break certificates that rely on alternate chains. We will endeavour to keep supporting this, but I wouldn't be surprised when other software starts breaking too.
Given that this patch is working, recommending for commit in 4.4.1.
This ticket was mentioned in Slack in #core by rmccue. View the logs.
9 years ago
#23
@
9 years ago
By reading about the OpenSSL versions included for CentOS (http://rpmfind.net/linux/rpm2html/search.php?query=openssl), I see 1.0.1e is even the last version on CentOS 7.1. So, even cPanel installs using their recent support for CentOS 7 (https://features.cpanel.net/topic/rhel-7-centos-7-support) will have this issue too, right?
#24
in reply to:
↑ 10
@
9 years ago
Replying to rmccue:
Therefore: I believe we should be able to take our existing bundle, and pull the 1024 bit certificates back in.
Attaching patched version of the CA bundle that adds the 1024 bit certificates back; this fixes resolution for me via OpenSSL on the command line, but needs testing on a site that's broken.
I agree with this direction. With an explicit mention in the commit (and file if possible) specifically explaining why we're including no-longer-trusted-by-browser roots.
If there's any way of automatically generating the crt
file for future maintenance that would also be grand. I'm thinking that even just pulling the latest NSS store + suffixing the 1024bit certs may be enough. Also worth noting is that we now need to monitor the status of these root certs, we've relied upon NSS to do that in the past, but since they no longer trust the certs, if one of them is compromised we'll need to be aware of it somehow.
This ticket was mentioned in Slack in #slackhelp by garubi. View the logs.
9 years ago
#28
@
9 years ago
Got the same errors trying to update some plugins.
I just replaced the ca-bundle.crt with the one provided here and on my server the errors doesn't show up anymore and the plugin updates where all ok.
I can confirm that on my environment this solves the problem.
Here my server's (shared hosting) details:
Server: OS Linux Software Apache Version 32Bit Name ufficiarredati.it Address 89.31.74.213 Port 80 Type Linux www.sosufficiarredati.it 2.6.24-23-server #1 SMP Wed Apr 1 22:22:14 UTC 2009 i686 System: PHP 5.2.4-2ubuntu5.10 Active Plugins 43 Zend 2.2.0 Database: SQL 5.0.51a Build 5.0.51a-3ubuntu5.4 Charset utf8
We get a number of support tickets for this. Will post details as they come in.
Happens on customer's site:
Customer 1:
PHP 5.4.45
cURL 7.38.0
OpenSSL 1.0.1e
Library and Header versions: OpenSSL 1.0.1e-fips 11 Feb 2013
Customer 2:
PHP 5.4.40
cURL 7.38.0
OpenSSL 1.0.1e
Library and Header versions: OpenSSL 1.0.1e-fips 11 Feb 2013
I've asked them to update OpenSSL to f+, will report back as soon as that's done.
Can't reproduce on own server:
PHP 5.5.9-1ubuntu4.14
cURL 7.35.0
OpenSSL 1.0.1f
Library and Header versions: OpenSSL 1.0.1f 6 Jan 2014
Answer from one host about updating:
Also, unless I'm doing something monumentally wrong, neither of these have helped:
Edit 1: added more customer environment data
Edit 2: added answer from host
Edit 3: added info about certs + sslverify
Edit 4: added php version