Dirty COW: Patch the Kernel, Then Make Sure You're Running It

Last Wednesday a Linux kernel bug went public with a catchy nickname, which usually makes me roll my eyes a little. This one deserves the attention. CVE-2016-5195, which is being called Dirty COW, lets someone who already has an unprivileged account on a Linux box become root, the vendors say it’s been used in real attacks, and Red Hat says it affects most modern Linux distributions.

The fixes are out for the big distributions. The part I want to spend time on is the gap between installing a fixed kernel and actually running one, because that’s where a lot of servers are going to sit for a while.

What the advisories say it does

Red Hat’s Dirty COW vulnerability page gives October 19, 2016 as the public disclosure date and rates it Important. Its description is a race condition in the way the kernel’s memory subsystem handled copy-on-write breakage of private read-only memory mappings. An unprivileged local user could use it to get write access to mappings that should have been read-only, and from there raise their privileges. Red Hat says an exploit using this technique has been found in the wild and that the flaw affects most modern Linux distributions.

The other vendors describe it the same way in fewer words. Debian’s advisory calls it a race condition in the memory management code that can be used for local privilege escalation, Ubuntu’s notices say a local attacker could use it to gain administrative privileges, and SUSE’s updates describe a local privilege escalation that is reportedly exploited in the wild.

Copy-on-write is the trick the kernel uses so that a private mapping of a file doesn’t get its own copy of a page until something writes to it. When the write happens, the kernel breaks the sharing and hands over a private copy, so the original stays untouched. The bug is a race in that handoff. That’s as deep as I’m going here, because for defenders the useful facts are the ones above: it’s local, it ends in root, and people are using it.

An ancient bug with a history

The upstream fix is commit 19be0eaffa3a, written by Linus Torvalds and dated October 13, and the commit message is refreshingly candid. Torvalds calls it “an ancient bug” and says he tried to fix it once, badly, eleven years ago, but that change was undone because it caused problems on s390. The s390 problem has long since been solved, so this time the fix could stick.

He also explains why an old bug suddenly matters. The VM has become more scalable over the years, and a race that was purely theoretical back then has become easier to trigger. At a high level the fix adds a new internal flag that records that a copy-on-write has already happened, and checks the page table’s dirty bit to make sure that record is still valid. The message also notes that s390 only gained the dirty bit it relies on in 3.9 and that earlier kernels will have to look at the page state instead, which is a hint that the backports to older trees aren’t all identical.

Which kernels have the fix

On October 20 Greg Kroah-Hartman released 4.8.3, 4.7.9 and 4.4.26, and each one carries the backport. The 4.8.3 changelog on kernel.org is short and the fix is the second entry, right under the version bump. The other maintained stable trees followed over the next few days: 3.2.83, 3.12.66, 3.16.38 and 3.10.104 by October 21, then 4.1.35 and 3.18.44 on October 24, and every one of their changelogs on kernel.org lists the same upstream commit.

Most of us don’t run kernel.org kernels on servers, though, so the distribution packages are what count. Debian shipped DSA-3696-1 on October 19, fixing jessie in linux 3.16.36-1+deb8u2. The Debian LTS team followed with DLA-670-1, which fixes wheezy in 3.2.82-1. Ubuntu published its kernel notices on October 20. USN-3106-1 covers 16.04 LTS with 4.4.0-45.66. The companion notices cover 12.04 with 3.2.0-113.155, 14.04 LTS with 3.13.0-100.147 and 16.10 with 4.8.0-26.28.

Red Hat’s updates came a few days later. RHSA-2016:2098 on October 24 fixes RHEL 7 in kernel-3.10.0-327.36.3.el7, and RHSA-2016:2105 on October 25 fixes RHEL 6 in kernel-2.6.32-642.6.2.el6. As of today Red Hat’s page still lists RHEL 5 and several extended support streams as pending, so if you’re on one of those, keep an eye on it. SUSE’s first kernel updates went out on October 21, according to its CVE page. SUSE-SU-2016:2592-1 updates the SUSE Linux Enterprise 12 SP1 kernel to kernel-default 3.12.62-60.64.8.2, and SUSE-SU-2016:2585-1 takes SLES 11 SP4 to 3.0.101-84.1.

Installed is not the same as running

Here’s the trap. Updating the kernel package puts a new kernel image on disk and adds it to the boot loader, and that’s all. The kernel in memory is still the old, vulnerable one until the machine boots the new one. Every vendor says this in its own way. Red Hat says a reboot is required for the kernel update to be applied, Ubuntu’s notices say you need to reboot after a standard system update, and SUSE’s say to reboot after installing. The Securing Debian manual puts it bluntly: if you don’t reboot after a kernel security update, you’re still running the old and vulnerable kernel.

So check what’s actually running, and compare it with what’s installed.

uname -r    # the running kernel release
uname -v    # build string; on Debian it includes the package version
rpm -q kernel                            # installed kernels on RHEL, CentOS
dpkg -l 'linux-image-*' | grep '^ii'     # installed kernels on Debian, Ubuntu

On RHEL and CentOS the release string from uname -r matches the package, so a fixed RHEL 7 box reports 3.10.0-327.36.3.el7 or later. Ubuntu’s notices point out that an ABI change gave these kernels a new version number, so a patched 16.04 server shows 4.4.0-45 or later. Debian is the one that trips people up. The jessie fix is built under the same 3.16.0-4 name as the kernel before it, so uname -r looks identical before and after, and you need the package version in uname -v to tell them apart.

When you can’t reboot yet

If a reboot has to wait for a maintenance window, live patching can close the gap, and a few vendors have already shipped or promised live patches for Dirty COW. Ubuntu published LSN-0012-1, a Kernel Live Patch Security Notice for 16.04 LTS generic and lowlatency kernels on amd64, on October 20. Even there, the notice still tells you to install an updated kernel and reboot at your convenience. SUSE shipped kGraft live patches for SUSE Linux Enterprise 12 kernels on October 25, including SUSE-SU-2016:2630-1. Red Hat’s page says a kpatch will be available for customers on RHEL 7.2 or later through a support case.

I’d treat a live patch as a bridge to a reboot rather than a replacement for one. It only covers the kernel builds the vendor lists, and the next boot should still land on a fixed package.

Why a “local” bug belongs at the top of the list

It’s tempting to push a local privilege escalation down the queue because an attacker needs an account first. I think that’s the wrong read for most servers. Any shared hosting box with customer shell or FTP accounts is exposed directly. So is anything that runs code you didn’t write, like CI runners building pull requests from strangers, or a PHP app with a plugin ecosystem. And a remote code execution bug in a web app usually hands the attacker exactly one thing, a shell as the web server’s unprivileged user, which is the starting point this bug needs.

Containers deserve a special mention. Every container on a host shares that host’s kernel, so a kernel privilege escalation from inside a container is a bug in the host, not in the container image. Rebuilding your images doesn’t fix Dirty COW. Patching and rebooting the hosts does.

If I were ordering the work, I’d go shared hosts and container hosts first, then anything that runs untrusted or user-supplied code, then everything else. CERT/CC’s VU#243144 gives the same basic advice: apply the patches through your distribution’s normal update process.

The takeaway is simple, and it’s easy to get wrong in the rush. A package manager that says the kernel is up to date doesn’t protect you on its own. Check uname on every host after the update, schedule the reboots, and use a live patch where your vendor offers one only to buy time until those reboots happen.