Performance Hit: Meltdown and Spectre Patches Slow SystemsIntel, Microsoft and Linux Confirm Slowdowns - Especially for Servers, Older PCs
Security researchers, operating system developers, microprocessor engineers and others are creating patches to help defend against the three memory corruption flaws collectively known as Meltdown and Spectre.
See Also: Privileged Access Threat Report 2019
But the fixes carry a performance cost, in part, because the "side-channel attacks" exploit a physical feature built into modern microprocessors that speed up operations. At least so far, safeguarding against attacks has required sacrificing speed.
On the upside, security researchers say that patches can help block exploits that target Meltdown and Spectre, which were first publicly disclosed on Jan. 3 and which affect millions of modern computing devices based on Intel, ARM and AMD chips. That includes not only PCs and smartphones, but also enterprise appliances and servers used in data centers that process large workloads (see Meltdown and Spectre Forecast: Patch Now and Keep Patching).
Intel, whose chips are most exposed to the flaws, on Tuesday reported that based on its PC benchmarking tests of 8th Generation Core platforms - its most modern microprocessor, introduced last October - with solid state storage, it saw microprocessor performance decrease by 2 percent to 14 percent, with an average 6 percent reduction, after installing patched firmware. The performance degradation for older microprocessors and systems with hard disk drives would likely be more severe.
Intel is already facing multiple U.S. class action lawsuits filed over the flaws.
But with firmware and software updates continuing to roll out, there's no accurate view yet of the performance costs that patching may incur. "It is important to note that many of the benchmarks published so far do not include both OS and silicon updates," Terry Myerson, executive vice president for Microsoft's Windows and devices group, says in a Tuesday blog post.
For maximum assurance against Meltdown and Spectre, organizations could "replace CPU hardware," which was the only solution first offered by Carnegie Mellon University's CERT Coordination Center (see Serious Meltdown and Spectre Flaws Make CPUs Exploitable). But CERT/CC later revised its recommendation to only read "apply updates." CERT/CC didn't immediately respond to a request for comment about what led to its revision.
Regardless, for most organizations, ditching their silicon outside of standard refresh cycles would be prohibitively costly. Furthermore, chips that lack the flaws still lie in the future.
"Obviously it'll need to be designed out in the microarchitecture of future chips, but the interesting technical question is how can they maintain performance without the sort of mechanism that this is exploiting," says Alan Woodward, a professor of computer at the University of Surrey.
Service Providers See Slowdowns
As Intel's tests demonstrate, at least some devices patched against Meltdown and Spectre will experience reduced performance.
Already, some cloud service providers and web services have reported seeing slowdowns. On Friday, U.S. video game company Epic Games, which develops such titles as Unreal, Gears of War and Infinity Blade, blamed "recent login issues and service instability" on its Meltdown patches.
"All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability," it said. "We heavily rely on cloud services to run our backend and we may experience further service issues due to ongoing updates."
Ian Chan, director of engineering for business analytics platform BranchMetrics, also reported seeing a significant performance impact after Spectre patches - not Meltdown patches, as he first suspected - were applied to the company's Amazon Web Services instances.
The #Meltdown patch (presumably) being applied to the underlying AWS EC2 hypervisor on some of our production Kafka brokers [d2.xlarge]. Ranges from 5-20% relative CPU increase. Ooof. pic.twitter.com/fXM0OzfdKx— Ian Chan (@chanian) January 6, 2018
Microsoft Confirms Slowdowns
Benchmark tests remain underway, as do efforts by everyone involved to iteratively design new patches that reduce performance penalties associated with the three vulnerabilities:
- Spectre: Refers to attack variant 1, a bounds check bypass (CVE-2017-5753), as well as variant 2, a branch target injection (CVE-2017-5715), which can be used to take advantage of CPU timing to read kernel memory;
- Meltdown: Refers to variant 3, which is a rogue data cache load (CVE-2017-5754) that can be used to read kernel memory.
"In general, our experience is that variant 1 and variant 3 mitigations have minimal performance impact, while variant 2 remediation, including OS and microcode, has a performance impact," Microsoft's Myerson says.
Benchmarks of Windows 10 running on 2016-era PCs or newer - with Skylake, Kabylake or newer CPUs - have on average seen "single-digit slowdowns" that Microsoft says shouldn't be noticeable. But running Windows 10 on older systems get noticeably slower, he says.
Meanwhile for users of older operating systems and hardware, "we expect most users to notice a decrease in system performance" in part due to operating system design," he says. "Older versions of Windows have a larger performance impact because Windows 7 and Windows 8 have more user-kernel transitions because of legacy design decisions, such as all font rendering taking place in the kernel."
And all Windows Server instances, regardless of how new the underlying hardware might be, will experience a "significant performance impact" if administrators follow Microsoft's Meltdown and Spectre security recommendation to isolate untrusted code within each server instance, if such code might pose a risk, Myerson says. "This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment."
Linux servers face similar challenges.
Linux introduced its first fix for Meltdown and Spectre last November in the form of Kaiser, short for "kernel address isolation to have side-channels efficiently removed." At the time, Jonathan Corbet, a technical advisory board member for The Linux Foundation, said that Kaiser created a "performance penalty," which, in worst-case examples, appeared to slow systems by nearly one-third.
"Kaiser will affect performance for anything that does system calls or interrupts: everything," he said. "Most workloads that we have run show single-digit regressions. Five percent is a good round number for what is typical. The worst we have seen is a roughly 30 percent regression on a loopback networking test that did a ton of syscalls and context switches."
Linux creator Linus Torvalds on Jan. 3 blasted Intel's engineers over the flaws and users' inability to disable Intel's workarounds. Unless Intel committed itself to fixing the problems, he added, "maybe we should start looking towards the ARM64 people more."
Linux Performance Questions
Jon Masters, Red Hat's chief ARM architect, has confirmed that addressing variant 2 creates a "not insignificant" slowdown for processors. "Red Hat's patches will default to implementing the security change and accepting the performance impact, but we've also added system administrators the ability to toggle this - and all the implemented settings - on or off," he says.
Again, benchmark tests are underway. "Actual performance impact numbers will depend on the software and environment in question," Andy Patel, a security researcher for Finnish anti-virus firm F-Secure, says in a Tuesday blog post. Servers will likely see the biggest impact, he says, while home machines may see no noticeable impact at all.
Meanwhile, Linux systems designed to mine cryptocurrencies, such as bitcoin and monero, don't appear to be affected, Patel says. "Mining, whether CPU- or GPU-based, shouldn't be affected - there shouldn't be any syscalls in mining loops," he says. "Monero - a CPU-based miner - network hashrate appears largely unchanged since the patch.
Despite the performance problems, the prevailing information security wisdom is for everyone to begin patching immediately and to keep patching.
Britain's National Cyber Security Center, part of intelligence agency GCHQ, says consumers should install patches as soon as they're available as well as "enable automatic updates so that future security measures are installed for you."
Chips require firmware updates; Intel's chips appear to be most affected. But some older AMD chips have been left unbootable by Microsoft's first security update designed to address the flaws. Microsoft and AMD says they're working on fixes (see Microsoft Pauses Windows Security Updates to AMD Devices).
Apple, Google, Linux, Red Hat and Suse have released operating system updates that begin to address Meltdown and Spectre, as has Microsoft, which says that so far it's shipped patches for 41 of the 45 editions of Windows it supports. All browser makers have also shipped updated software - or plan to soon do so - that is designed to address the flaws.
Intel began sending firmware updates to manufacturers last month. "For Intel CPUs introduced in the past five years, we expect to issue updates for more than 90 percent of them within a week, and the remainder by the end of January," Intel says. "We will continue to issue updates for other products thereafter." Then it will be up to device manufacturers to update the software and distribute it to users. There are no guarantees on when - or in some cases if - that will happen.
Guidance for Enterprise Administrators
As patches and updates do become available on all fronts, however, NCSC recommends organizations install them as quickly as possible. But of course, enterprise administrators must also identify all vulnerable cloud services, data centers and servers, end user devices as well as applications and software used by their organization, and then track what promise to be multiple waves of patches for any given device or service.
Many major enterprise vendors - ranging from Cisco and Dell to IBM and Juniper - among others - have confirmed that at least some of their products are at risk from Meltdown and Spectre and say they don't yet have a full picture.
For all hardware devices, NCSC says, "it's not sufficient just to update the operating system - you will need to check that the underlying firmware is also up to date." While blocking Meltdown only requires software fixes, defending against Spectre requires both firmware and software fixes.