Let’s Encrypt root cert update catches out many big-name tech firms
6 de outubro de 2021UPDATED The expiration of Let’s Encrypt’s root certificate last week threw up a number of problems, though not perhaps in the areas predicted ahead of the event
A legacy certificate used by the certificate authority – the IdentTrust DST Root CA X3 – expired on September 30
Let’s Encrypt saw this issue coming more than two years ago, repeatedly advising its community and subscribers on how to move over to a new root cert.
BACKGROUND Device ‘breakage’ concerns persist days before Let’s Encrypt root cert expiry
The non-profit delayed the transition earlier this year in order to allow updates for Android to become available, and in the interests of making the changeover as smooth as possible.
Digital SSL certificates that underpin the HTTPS protocol are issued by trusted certificate authorities, with indexes updated through operating system updates.
If these indexes have not been updated, then affected systems will fail to recognize the new Let’s Encrypt root certificate – thereby breaking the chain of trust between a website and a user’s browser.
By way of example, the AddTrust External CA Root expired in May 2020, leaving multiple organizations with problems as a result.
Unchained melody
Last week’s Let’s Encrypt root cert change also proved to be a bumpy road.
Issues arose not so much because of clients running obsolete versions of operating systems, but because the servers of several organizations failed to serve updated certificate chains to clients due to configuration problems.
Web security expert Scott Helme – who posted a detailed blog post warning of potential trouble with the Let’s Encrypt root cert shortly before the event – helped us survey the wreckage.
Ahead of the event, Helme had predicted that slightly older devices and software, particularly systems depending on OpenSSL 1.0.2, could be at risk. Let’s Encrypt previously expressed concerns about older Android devices.
Chain of fools
As it turned out, systems depending on OpenSSL 1.02 (which has been obsolete since December 2019) accounted for the majority of issue with the Let’s Encrypt root cert transition logged so far.
Big companies with an issue included: Palo Alto, Bluecoat, Cisco Umbrella, Catchpoint, Guardian Firewall, Monday.com, PFsense, Google Cloud Platform, Microsoft Azure Application Gateway, OVH, Auth0, Shopify, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, cPanel, Ledger, Netlify, Cloudflare Pages, Sophos, AWS, and DigitalOcean.
OpenBSD issued an alert on the Let’s Encrypt issue
Read more of the latest browser security news
Most of the issues might be categorized as glitches, and the problem is largely in hand. However, anything that affects the fragile chain of trust that underpins e-commerce and much else ought to be a concern.
Issues for Fortinet users are more gnarly than the norm, even though the firm has since offered a workaround, as detailed in an advisory from the network security appliance firm:
The issue being seen by Fortinet customers is due to Fortinet devices validating the full chain of trust and then invalidating the chain when it sees that the CA IdenTrust DST Root CA X3 is expired, even though the cross-signed ISRG Root X1 root is valid for longer.
We have removed the offending expired certificate from the certificate store; however, this still does not solve the problem due to the Authority Information Access – CA Issuers entry.
The networking equipment firm concluded: “Fortinet is working on a longer-term solution to improve certificate validation and add additional intelligence to rebuild missing certificate chains in these cases going forward, and will include this in a future release.”
Unforced error
Many of the issues that Fortinet and customers of other tech firms experience as a result of the Let’s Encrypt root certificate expiry were preventable.
“A lot of the orgs that saw failures was because they depend on software (mainly OpenSSL) that was either old or missing an option being set to allow it to continue working,” Helme told The Daily Swig. “It certainly seems to be the culprit for the Google and AWS ones, probably the Microsoft issue too.”
Helme has created a test site to help identify issues with clients.
The Daily Swig asked Let’s Encrypt for an update on the situation but we’re yet to hear back.
In earlier correspondence, a spokesperson for Let’s Encrypt asked us to reiterate that the organization had been trying to spread the word about the root expiration for more than two years.
“Our first blog post on the topic was posted in April 2019,” the spokesperson said.
In a pinned Tweet, published on September 30, the same day as the root cert expiry, Let’s Encrypt said: “Our cross-signed DST Root CA X3 expired today. If you are hitting an error, check out fixes in our community forum. We’re seeing higher than normal renewals, so you may experience a slowdown in getting your certificates.”
In response to queries from The Daily Swig, Let’s Encrypt said some disruption was inevitable and offered pointers towards useful resources.
The expiration last week did lead to some people having trouble accessing some sites. The internet is complex and many parts of it have been around for decades so it is inevitable that some older clients and devices will be affected when a change like this happens.
We knew that this would be the case, which is why we appreciate you helping to get the word out last week so people would understand and know how to investigate fixing it.
Scott [Helme] has kept a good list of the companies that were affected and we’ve been standing by on our community forum to help any of these companies or others who have come in looking for help on how to update or implement a workaround.
Fonte: The Daily Swig
Resources for Certificate Chaining Help
A blog post from Let’s Encrypt summarizing resources can be found here.
This story has been updated with comment from Let’s Encrypt – Resources for Certificate Chaining Help
Oct 1, 2021 • Jacob Hoffman-Andrews
As planned, the DST Root CA X3 has expired and we’re now using our own ISRG Root X1 for trust. We used a cross-sign with DST Root CA X3 to gain broad trust for our certificates when we were just starting out. Now our own root is widely trusted.
For most websites, it was just another day on the Internet, but inevitably with such a big change some sites and configurations have issues. Our overview of the planned expiration is here. You can read about what we’ve done to make the process smoother. Most problems can be solved by updating the software on the machine that is having trouble.
You may also find these links helpful:
Our certificate compatibility page.
Workarounds for OpenSSL 1.0.2.
Whenever there is a significant change to our API, we post in the API Announcements category in our community forum. Sign in and click the bell for notifications to be sent to your email! If you want to hear even more from Let’s Encrypt and the nonprofit team behind it, subscribe to our newsletter. You’ll only receive a handful of emails each year.
We (and our community) are here for you! If you have any questions about this change, search on our community forum or post on the thread we have to help you with this very topic.
Supporting Let’s Encrypt
As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.
Acesse aqui e saiba tudo sobre TLS, o protocolo de segurança que garante o sigilo das informações e identifica empresas, dispositivos e objetos no mundo eletrônico.