Meltdown and Spectre: What Patching Looks Like One Week In

A week ago today, Google’s Project Zero published details of a processor problem the industry had been quietly working on since June. The short version from Project Zero’s post is that CPU data cache timing can leak information out of speculative execution the processor later throws away, which at worst means reading arbitrary memory across local security boundaries. Project Zero reported it to Intel, AMD and ARM on June 1, 2017, and other researchers reported the same issues too. Their write-ups gave us the names Meltdown and Spectre.

It also went public early. Google’s Security Blog says it posted ahead of a coordinated date of January 9 because of public reports and press speculation, and Intel’s January 3 statement says vendors had planned to disclose the following week, when more updates would be ready. So patches are landing on a compressed schedule. Here’s what the official advisories say, what has shipped, and what to do about servers, laptops and browsers.

What the advisories say these bugs allow

CERT/CC’s VU#584653 describes three variants. Variant 1 is CVE-2017-5753, bounds check bypass, and variant 2 is CVE-2017-5715, branch target injection. Together they’re Spectre. Variant 3 is CVE-2017-5754, rogue data cache load, which is Meltdown. The NVD entries for all three describe unauthorized disclosure of information to an attacker with local user access via side-channel analysis. Intel’s statement says it believes the exploits can’t corrupt, modify or delete data.

CERT explains that when a CPU speculatively runs instructions it shouldn’t have, it discards the results but not indirect side effects like cache changes, and timing memory accesses can reveal values touched during that discarded work. Spectre abuses branch prediction and Meltdown abuses out-of-order execution. CERT says Meltdown lets a user-space process read kernel memory, while Spectre can leak data between processes or between parts of the same process.

Google’s Security Blog says the attacker must first be able to run code on the system. CERT says multi-user and multi-tenant systems, including virtualized and cloud environments, likely face the greatest risk, and systems used to browse arbitrary websites are at risk too, because JavaScript is code from strangers your laptop runs all day. Google adds that its testing showed an attack in one virtual machine could read the host’s physical memory, and through it, another VM’s memory on the same host.

The chip picture isn’t uniform. Red Hat’s kernel advisory says CVE-2017-5754 affects Intel x86-64 processors and not AMD’s. AMD’s January statement said there’s zero AMD vulnerability to variant 3 because of architecture differences, near zero risk for variant 2, and that software updates resolve variant 1. ARM’s table marks only the Cortex-A75 as affected by variant 3.

One fix per variant

Google’s follow-up post on mitigations is the clearest map I’ve found. There’s no single fix; each variant needs its own.

Variant 3 gets fixed in the operating system. On Linux that’s Kernel Page Table Isolation, or KPTI, which Google says it has deployed across its entire fleet of Linux production servers. Debian’s DSA-4078-1 on January 4 brought KPTI to Intel x86-64 in stretch’s 4.9.65-3+deb9u2 kernel, describing it as a near complete separation of kernel and userspace address maps and saying Spectre fixes come later. Red Hat’s RHSA-2018:0007 for RHEL 7 went out January 3 with x86-64 mitigations for all three variants, and says bluntly that the issue is in hardware and can’t be fully fixed by software. Ubuntu’s USN-3522-1 arrived yesterday with Meltdown kernels for 16.04 LTS, and Ubuntu says Spectre fixes come in a later round. Red Hat and Ubuntu both say you need to reboot, and a machine still running the old kernel isn’t protected.

Variant 2 is the hard one. Google says it needs either a microcode update from the CPU vendor, such as Intel’s IBRS microcode, or Retpoline, a software technique Google developed that gets applied to hypervisors, kernels and programs. Google says Retpoline had negligible performance impact on its systems. Microcode takes a longer road: Microsoft notes it goes from the chip vendor to system manufacturers, who decide how to release it. Intel’s January 4 release says it expected to have issued updates for more than 90 percent of processors introduced in the past five years by the end of the following week.

Variant 1 gets handled one binary at a time. Google says that means recompiling so vulnerable code isn’t emitted, and it names operating systems and applications that run untrusted code, like browsers, as examples.

Windows and the antivirus registry key

Microsoft shipped updates on January 3 under ADV180002. A January 9 Microsoft post summarizes the approach: compiler changes and recompiled binaries for variant 1, new CPU instructions for variant 2 that also need a microcode update, and separate kernel and user mode page tables for variant 3.

Microsoft’s KB4072699 says some antivirus products make unsupported calls into Windows kernel memory, which can cause blue screens that leave a device unable to boot. So Microsoft only offers the January security updates to machines whose antivirus vendor has confirmed compatibility by setting a registry key. Windows Defender Antivirus, System Center Endpoint Protection and Microsoft Security Essentials set it, and if you can’t run antivirus at all, the article documents setting the key yourself. If the update isn’t being offered, check your antivirus first.

Servers need more. The original ADV180002 text says Windows Server needs firmware and software updates and the protections must be configured, with details in KB4072698. Microsoft says those in-guest settings matter when a Windows Server instance runs untrusted code. For laptops, Microsoft says you may also need firmware from your device manufacturer.

Browsers are cutting timers

Browser vendors went after the timing half of the attack. Mozilla’s January 3 post says its experiments confirmed web content could read private information across origins this way. It reduced performance.now() resolution to 20 microseconds and disabled SharedArrayBuffer, which can be used to build a high-resolution timer, and an update says both shipped in Firefox 57.0.4 on January 4. Microsoft’s Edge team said the January 3 updates remove SharedArrayBuffer from Edge and cut performance.now() resolution in Edge and Internet Explorer 11 from 5 microseconds to 20, plus up to 20 more of jitter.

The WebKit team’s write-up says the first mitigations shipped January 8 in iOS 11.2.2, the macOS High Sierra 10.13.2 Supplemental Update and Safari 11.0.2, which reduce timer precision to 1 millisecond and disable SharedArrayBuffer.

For Chrome, Google’s product status page says Chrome 63 on desktop has an optional Site Isolation feature that puts websites in separate address spaces, enabled with the chrome://flags/#enable-site-per-process flag or enterprise policy. Google said Chrome 64, due January 23, will add mitigations. If you manage a fleet, that policy is worth testing now.

Cloud providers patched the hosts, not your guests

The AWS bulletin says every EC2 instance is protected from these issues coming from other instances, and still recommends patching instance operating systems. Microsoft’s Azure team says it began automatically rebooting remaining VMs on January 3 for a hypervisor-level fix that needs no change to VM images, while still pointing you to your OS vendor. Google says the infrastructure running Compute Engine is protected and customers must update their VM operating systems against attacks inside the guest.

So the provider stops a neighbor from reading your VM, but your own kernel is your job.

Why the fixes cost performance

Red Hat’s vulnerability article explains the Meltdown cost well. Kernels have long mapped kernel memory into each application’s address space, because programs make system calls constantly and switching address spaces every time means flushing CPU structures like the translation lookaside buffer. KPTI gives up that shortcut, so each trip into the kernel costs more. Google says the impact depends on an application’s rate of system calls, was negligible on most of its workloads, and that microbenchmarks can exaggerate it.

The Red Hat performance team measured 1 to 20 percent in January across the benchmarks it tested, with OLTP databases and other workloads heavy on user and kernel transitions at the top, around 8 to 19 percent, and CPU-bound HPC jobs around 2 to 5. Microsoft’s January 9 post says Windows 10 on 2016-era Skylake or Kaby Lake chips shows single-digit slowdowns most people shouldn’t notice, some Windows 10 users on 2015-era Haswell or older chips will notice, and most Windows 7 and 8 users on those older chips will. Windows Server, especially IO-intensive applications, takes a bigger hit with the in-guest mitigations on. Intel calls the impact workload-dependent, and AWS says it hasn’t seen meaningful impact for the overwhelming majority of EC2 workloads.

My read is that newer laptops will be fine, older Windows 7 machines will feel it, and busy database servers need real testing. Debian notes KPTI can be disabled at boot with pti=off, but I’d save that for isolated machines that never run untrusted code.

What to do this week

Install your OS kernel or Windows update and reboot into it, checking your antivirus first on Windows if the update doesn’t appear. Update every browser. Apply firmware and microcode updates from your system manufacturer as they’re released. In the cloud, patch your guests even though the host is covered. Benchmark your own workload on a patched test machine before rolling out to busy servers.

Then keep checking. CERT’s note warns its content may change as understanding develops, and vendor pages have already been revised several times this week. A hardware flaw that needs OS, firmware and browser changes at once won’t settle in one patch cycle.