You Patched Heartbleed. Now Finish the Job

A week ago today, the OpenSSL project put out a security advisory for CVE-2014-0160, the bug everyone now calls Heartbleed. Most of us spent the next few days running package updates, which was the right first move, but only the first. If you stop at the upgrade, there’s a real chance your servers are still running the old code, and even when they aren’t, anything that leaked before the patch is still out there in someone else’s hands.

What the bug gives away

A missing bounds check in OpenSSL’s handling of the TLS heartbeat extension lets the other side of a connection read up to 64k of memory, and the wording is “a connected client or server”, so it cuts both ways. Only the 1.0.1 and 1.0.2 beta releases are affected, up to and including 1.0.1f and 1.0.2-beta1, so the older 0.9.8 and 1.0.0 branches are out of scope. The fix shipped in 1.0.1g, with 1.0.2-beta2 promised for the beta line, and anyone who couldn’t upgrade right away was told they could recompile with -DOPENSSL_NO_HEARTBEATS instead.

The heartbeat extension comes from RFC 6520, published in February 2012 as a way to keep a TLS or DTLS connection alive without a renegotiation, and as a building block for path MTU discovery in DTLS. OpenSSL’s NEWS file lists TLS/DTLS heartbeat support among the major changes in 1.0.1, released March 14, 2012. So the vulnerable code has been around for about two years.

The message format is simple. A heartbeat has a one byte type, a 16 bit payload_length, the payload itself, and at least 16 bytes of random padding. Whoever receives a request has to send back an exact copy of the payload. The RFC also says that if payload_length is too large, the message must be silently discarded.

OpenSSL 1.0.1f skipped that last part. In tls1_process_heartbeat it reads the type, reads payload_length, allocates a response big enough for that many bytes, and then copies that many bytes starting where the payload begins. Nothing compares the claimed length with how much data the record really holds. When the claimed length is bigger, the copy keeps going past the end of the real data into whatever happens to sit next to it in the process’s memory, and all of that goes back out in the response. Sixteen bits tops out at 65,535, which is where the 64k figure comes from.

The fix commit that landed in 1.0.1g is small. It makes sure the record is long enough to hold a type, a length and the minimum padding, and then that one byte of type plus two bytes of length plus the payload plus 16 bytes of padding actually fits inside the record. If either check fails, the heartbeat is dropped, with a comment pointing at section 4 of the RFC. Both the TLS code in t1_lib.c and the DTLS code in d1_both.c got these checks, and the DTLS side also got a cap on the response size.

What makes this nasty is what lives in that memory. The US-CERT alert TA14-098A sorts the risk into secret keys, user names and passwords used by the vulnerable service, the protected content itself, and memory addresses that help an attacker get around exploit mitigations. It also points out that an attacker can just keep asking, pulling one 64k chunk after another. The CVE description already says reading private keys has been demonstrated. And the CERT/CC vulnerability note VU#720951 reminds us this isn’t only about port 443, because any service that supports STARTTLS, such as IMAP, SMTP or POP, may be affected too.

Which OpenSSL is actually running?

There are three ways to get the version check wrong.

The first is the version string. Distributions backport the fix into the version they already ship, so a patched system can still say 1.0.1e. Red Hat’s RHSA-2014:0376 fixed RHEL 6.5 in openssl-1.0.1e-16.el6_5.7. Debian’s DSA-2896-1 fixed wheezy in 1.0.1e-2+deb7u5. Ubuntu’s USN-2165-1 fixed 12.04 LTS in libssl1.0.0 version 1.0.1-4ubuntu5.12. Compare the full package version with your distro’s advisory rather than looking for the letter g.

openssl version -a
dpkg -s libssl1.0.0 | grep Version    # Debian and Ubuntu
rpm -q openssl                        # Red Hat, CentOS, Fedora

If you recompiled instead, the compiler: line from openssl version -a shows the flags that the library behind that openssl command was built with, so you can confirm -DOPENSSL_NO_HEARTBEATS made it in.

The second trap is that the openssl command on your path isn’t necessarily the library your services load. A hand built copy under /usr/local can sit next to the system one, and ldd on a server binary won’t always settle it. Apache typically gets OpenSSL through mod_ssl.so, a module loaded at runtime, so ldd httpd can show no libssl at all while the running server has one mapped. It’s more reliable to ask the running process, with lsof -p and its process ID or by reading /proc/<pid>/maps, and see which libssl file it has.

The third is software that brings its own OpenSSL. CERT/CC says reports indicate mod_spdy can keep Apache from using the updated library, because mod_spdy uses its own copy of OpenSSL. Static builds are the same story. A hand built OpenSSL 1.0.1 produces static libraries unless you ask for shared, so programs compiled against it carry the vulnerable code in their own binary and show no libssl file at all. If something speaks TLS and you can’t find a libssl in its process, work out whether it uses another TLS library or has OpenSSL compiled in, because in that case it needs its own rebuild.

Why the upgrade isn’t the end

Restart everything that loaded the old library

A package upgrade replaces the file on disk, but a process that was already running keeps the old library mapped in memory until it exits. Red Hat’s advisory spells it out: every service linked to OpenSSL, like httpd, has to be restarted or the machine rebooted. Debian says to upgrade and restart applications as soon as possible, and Ubuntu’s notice says to reboot after the update.

A reboot is the simplest answer. If you can’t reboot, lsof marks a Linux mapped file that has been deleted as DEL, which is how an old library typically shows up after the package manager replaces it:

sudo lsof -n | grep libssl | grep DEL

Anything that shows up is still running the old code. My advice is to do a full stop and start rather than a graceful reload, since a reload can leave the original parent process, and the library it already has mapped, in place. Then run the check again. Don’t forget long running clients here either, since the bug leaks in both directions.

New private keys, then new certificates

Patching and restarting stops future leaks. It can’t tell you whether your private key already walked out the door, and both US-CERT and CERT/CC note that exploit code is publicly available. CERT/CC’s advice is to regenerate secret keys and passwords on the assumption that an attacker already has them.

The important word is regenerate. Reissuing a certificate for the same old key doesn’t help, because the thing that might have leaked is the key. Generate a fresh one along with a new CSR:

openssl req -new -newkey rsa:2048 -nodes -keyout example.com.key -out example.com.csr

The -nodes flag leaves the key unencrypted on disk, so lock down the file permissions. This step comes after the restart for a reason: install a new key into a process still running 1.0.1f and the new key is just as exposed as the old one.

Revoke the old certificate

A certificate with a leaked key stays valid until it expires, and whoever holds that key can use it to impersonate your site. CERT/CC’s impact section mentions decrypting, spoofing and man in the middle attacks, and its solution section says old keys should be revoked. Once the new certificate is deployed and working, ask your CA to revoke the old one. Doing it in that order keeps you from knocking your own site offline.

US-CERT also suggests considering perfect forward secrecy, which CERT/CC says makes already captured traffic harder to decrypt if a key leaks later. The same note adds a catch: if a TLS session ticket key is leaked, the sessions that use that ticket could be compromised, and ticket keys may only be regenerated when the web server is restarted. That only helps when the server generates its own random ticket key, which is the default in both nginx and Apache. If you set a fixed key file with nginx’s ssl_session_ticket_key or Apache 2.4’s SSLSessionTicketKeyFile, a restart just loads the same key again, so generate a new file by hand.

Kill sessions and cookies

A web server’s memory holds the requests passing through it, and those requests can carry session cookies and submitted passwords. That fits the protected content and secondary key material US-CERT describes, so it’s reasonable to assume session tokens could have leaked too. Invalidate server side sessions, rotate any secret you use to sign cookies, and make people log in again. I’d do this once the old certificate is revoked. A new certificate going live doesn’t stop someone holding the old key from impersonating your site, since the old certificate stays valid until it’s revoked or expires, and even revocation only helps with clients that actually check it. Killing sessions after revocation at least closes the obvious window for grabbing the fresh ones.

Then passwords

Password resets go last, and the order is the whole point. If you ask users to change their passwords while the server still runs the vulnerable library, you’ve just asked them to type new passwords into the thing that leaks memory. Rotate your own credentials at the same point: database passwords, API keys, anything a vulnerable service kept in memory.

The takeaway

Heartbleed is a small mistake with a very long tail. Trusting a length field that came off the network turned into two years of releases that would hand memory to anyone who asked, and no package update can undo a leak that already happened. So treat the upgrade as step one of six, and only ask anyone to pick a new password once the other five are done. It’s tedious, but it’s the difference between having patched and having recovered.