DST Root CA X3 Expires September 30, and Old OpenSSL Is the Real Risk

In ten days a root certificate that has quietly held up a big chunk of the HTTPS web runs out. IdenTrust’s DST Root CA X3 is the root Let’s Encrypt leaned on when it launched, and its self-signed certificate expires at 14:01:15 UTC on September 30, 2021. If you run a normal website with a Let’s Encrypt certificate, you will probably not notice anything. If you run an API, have users on old operating systems, or ship software in containers built on old base images, this is a good week to spend an hour checking a few things.

Why an IdenTrust root matters to Let’s Encrypt

Every new certificate authority hits the same wall. Its own root takes years to get into operating system and browser trust stores, and even longer to reach the devices people actually carry around. When Let’s Encrypt started, it got a cross signature from IdenTrust, whose DST Root CA X3 was already trusted nearly everywhere, and at the same time it created its own root, ISRG Root X1, and applied to the major root programs.

Last November, in Standing on Our Own Two Feet, Let’s Encrypt explained the catch with retiring the IdenTrust path. Android versions before 7.1.1 don’t trust ISRG Root X1, and because of how Android updates flow through manufacturers and carriers, a lot of those phones never will. The suggestion at the time was for those users to install Firefox, which ships its own root store instead of relying on the one baked into the OS.

The Android trick

A month later Let’s Encrypt announced a way around the Android problem. IdenTrust agreed to issue a new cross signature for ISRG Root X1 from DST Root CA X3, and that cross signature is valid well past the expiry of DST Root CA X3’s own self-signed certificate.

It works on Android because of a detail in how trust stores are defined. Root stores technically hold trust anchors, not certificates in the strict sense, and the standards let each implementation decide whether to honor fields like the expiry date on those anchors. Android deliberately ignores the expiry on its trust anchors. Old Android phones never got ISRG Root X1 added, but they also never had DST Root CA X3 removed, so they keep trusting that key and happily validate a chain that runs through the new cross signature. The cross signed certificate itself is good until September 30, 2024, and that’s when Let’s Encrypt expects phones older than 7.1.1 to finally start seeing errors.

What the default chain looks like now

The price of that trick is a longer chain and slightly bigger TLS handshakes. Starting May 4, 2021, Let’s Encrypt switched its default chain for RSA certificates so that it goes from your certificate to the R3 intermediate, then to ISRG Root X1 as cross signed by DST Root CA X3. Any ACME client that uses the chain the API hands back picked this up on its next renewal. There is also an alternate chain that goes from your certificate to R3 and stops at ISRG Root X1, with no mention of the IdenTrust root at all. It’s shorter, and it gives up the old Android compatibility.

After September 30, a client checking the default chain has two possible routes to a root. It can stop at the self-signed ISRG Root X1 in its own trust store, which is valid. Or it can keep following the cross signed certificate up to DST Root CA X3, which is now expired. Which route it takes depends entirely on the software doing the verifying, and that’s where the breakage comes from.

Who actually breaks

Let’s Encrypt’s page on the expiration is refreshingly blunt about it. Devices that don’t trust ISRG Root X1 and aren’t Android will start showing certificate warnings. For anyone who provides an API or supports IoT devices, it spells out two requirements: every client needs to trust ISRG Root X1, not just DST Root CA X3, and clients that use OpenSSL need to be on version 1.1.0 or later.

The first group is mostly old Apple devices and anything with a frozen trust store. The compatibility list puts ISRG Root X1 support at macOS 10.12.1 and iOS 10, so a Mac stuck on an older release or an iPhone that can’t update past iOS 9 is going to start failing on Let’s Encrypt sites. Java keeps its own trust store too, and the same list gives 7u151 and 8u141 as the first updates that include the new root, so an old JRE bundled into an appliance or a forgotten build server is a real candidate.

The second group is sneakier, because it includes machines that already have ISRG Root X1 installed. The OpenSSL project posted a note on September 13 explaining why. When OpenSSL 1.0.2 builds a chain, it prefers the untrusted certificates the server sent over what’s in the local store. Following the server’s certificates leads to DST Root CA X3, which is in the trust store, so 1.0.2 settles on that chain and then reports it as expired. It never tries the shorter route that ends at the ISRG Root X1 it already has. OpenSSL 1.1.0 changed this so trusted certificates are checked first, and the 1.1.1 verify documentation notes that behavior is on by default and can’t be switched off. That’s the whole reason the magic number is 1.1.0.

This one worries me more than old iPhones. OpenSSL 1.0.2 stopped getting public updates at the end of 2019, but it hides in plenty of places that don’t get looked at much: servers on older OS releases, language runtimes compiled against it, and container images nobody has rebuilt in a couple of years.

What to check on the server side

Start by looking at what your server actually sends.

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

The s_client documentation is careful to point out that this list is exactly what the server sent, in the order it sent it, not a verified chain. For each certificate you’ll see a subject line and an issuer line. With the current default chain you should see your certificate issued by R3, then R3 issued by ISRG Root X1, then ISRG Root X1 issued by DST Root CA X3. If you see an old intermediate from a chain file someone copied in by hand, fix that first; Let’s Encrypt’s guidance is to use the intermediate the API returns rather than hardcoding one.

Then think about who your clients are. For a public website the default chain is the sensible choice, since it keeps old Android working and modern browsers don’t care. If you run an API whose consumers you know are stuck on OpenSSL 1.0.2, and you don’t have old Android clients, the OpenSSL note lists serving the alternate chain as a server side fix. Most up to date ACME clients can request it, so check your client’s documentation for the option.

What to check on clients and containers

On anything that calls HTTPS endpoints, check the OpenSSL version your code actually uses, which isn’t always the one the command line tool reports.

openssl version
python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
node -p process.versions.openssl
curl --version

Run these inside your container images, not just on the host. An image carries its own libraries and its own CA bundle, frozen at whatever they were on the day it was built. Updating the ca-certificates package and rebuilding is how an image picks up ISRG Root X1.

You can also test against the future instead of waiting for it. OpenSSL 1.1.1’s s_client accepts verification options including a time to validate at, given in Unix seconds, so you can pretend it’s already October.

openssl s_client -connect api.example.com:443 -servername api.example.com -attime 1633046400 -verify_return_error </dev/null

That timestamp is midnight UTC on October 1, 2021. A verify return code of 0 means that machine’s OpenSSL and trust store will be fine after the expiry, as long as the server’s own certificate is still valid on that date too. The 1.0.2 s_client doesn’t list that option, but its verify command does, so on an older box you can save the certificates from the showcerts output into files and run verify with the same timestamp, passing the intermediates with the untrusted option, to see whether it trips over the expired root.

If you find 1.0.2 clients you can’t upgrade in time, the OpenSSL note gives two client side workarounds. The simplest is to remove DST Root CA X3 from that client’s trust store and make sure ISRG Root X1 is in it. With the expired root gone, 1.0.2 no longer finds the bad route and goes looking for another one that is trusted. The note walks through how that looks on systems that manage certificates with update-ca-trust. The other option is a trusted first flag, but applications have to set it in their own code, so it only helps if you control the source.

The short version

Browsers on reasonably current systems will sail through September 30. The failures, when they come, will mostly be machine to machine: a cron job that can’t reach a webhook, an integration running on an old OS release, a container built from a base image nobody has refreshed. Check the chain your servers send, check the OpenSSL version and CA bundle on every client you’re responsible for, and use the attime trick to see the result now. That beats finding out on October 1 from your logs.