71 comments

[ 4.0 ms ] story [ 111 ms ] thread
No paper up yet, but there's a discussion on seclists

https://seclists.org/oss-sec/2018/q4/127

    01 Oct 2018: Notified Intel Security
    26 Oct 2018: Notified openssl-security
    26 Oct 2018: Notified CERT-FI
    26 Oct 2018: Notified oss-security distros list
    01 Nov 2018: Embargo expired
Why even do an embargo if you give hardware people 1 month and software people 1 week?!
Bad experience in the past probably. Also this probably just leads to the official recommendation to disable Hyper-Threading "for security critical applications" or some PR blurb.
Any number of possible reasons.

OpenSSL shipped a patch for it in that interval. Intel isn't going to fix it faster than OpenSSL ships a patch revealing it unless they already had a convenient killbit for the affected things.

I have no idea what the relevant researcher's policies are, but I would assume we'll hear about it if somebody requested a longer embargo and they refused.

(It also appears to be much harder to reproduce in the presence of dynamic clock speeds, so the impact in most smaller environments is going to be low unless someone does further work to make it reproduce well with that.)

As predicted Spectre/Meltdown were just the first fruit of a major new wave of investigations into these avenues of side-channel attacks in CPUs. Now we're seeing more as it gets more attention, and it's interesting stuff. That said I think the researchers might go too far here:

>"This is the main reason we released the exploit -- to show how reproducible it is," Brumley told us, "and help to kill off the SMT trend in chips."

>"Security and SMT are mutually exclusive concepts," he added. "I hope our work encourages users to disable SMT in the BIOS or choose to spend their money on architectures not featuring SMT."

I don't blame them for being security focused about all else at any cost and any layer, that's their gig. But I think the real response here is likely to be a lot more subtle and interesting. Of course perhaps SMT can in fact be fixed for this without a wholesale tossing in which case it'll just be a universal hardware revision somewhere down the line. This increasing level of public research and awareness of this specific class is still relatively early days after all. But taking it as a given for argument that there really is a fundamental conflict, the fact would remain that SMT can provide significant performance gains, and furthermore that we're still far from the point where SaaS/IaaS is everywhere. Lots of systems are still under single user local control, and in turn attackers being able to co-run their own arbitrary code on the same physical core isn't necessarily part of the threat model at all (and more specifically if attackers get that far least common denominator kicks in, they've already owned what's important). Even if it's desirable to run some risky code as well, hard core affinity for non-secure processes is a brute force solution in a local system context that seems like it shouldn't be a big deal given the a surfeit of physical/logical cores for many work loads.

But perhaps this could be a leading edge of true processor level physical differentiation required between IaaS and more traditional deployments, and that might make for an interesting change to the competitive landscape there. It'd change the cost/benefit in some scenarios, or require more custom processor work to enact harder (and performance costing) boundaries that a traditional setup might take care of with machine isolation. I wonder if that could shift things back away from the Cloud and centralization trend in some instances, or at least create and more dynamic market?

The main problem with the view that a single-user system is safe from this is the point that even single-user systems run untrusted code. For example, your web browser probably by default runs Javascript, NaCl, or WebAsm code, which is supposedly sandboxed sufficiently to no longer be a threat to the other stuff running on your machine. This sort of exploit (which was kind of obviously going to happen at some point) breaks this sandbox to some extent.
>For example, your web browser probably by default runs Javascript, NaCl, or WebAsm code, which is supposedly sandboxed sufficiently to no longer be a threat to the other stuff running on your machine.

I think that would fall under "Even if it's desirable to run some risky code as well" and using hard core affinity wouldn't it? For performance oriented use cases 8-core and higher systems are not exactly super rare at this point, and AMD at least has been pushing the physical core counts pretty hard even on non-super high end chips. In a single owner/trusted dedicated metal case it's a lot easier to just say "core 1 is for running web code and that's it, if every core is needed for something else the web stuff will all be evicted and slept until that's over" which on a percentage basis doesn't incur that big a hit in a high core system. It's not like a IaaS system that's devoted to running dozens/hundreds of commercial client VMs and needs to be able to perform dynamic allocation with no manual intervention and untrusted code is the rule not the exception. IaaS is all about scaling, and that's a hard and interesting problem. But sometimes for individual instances it works fine to just pick a completely non-scalable solution that gets the job done and brute force it.

I'm just thinking that it's worth remembering that most security is an economic equation, and sometimes performance is in fact worth it, or the security issue can be soft mitigated sufficiently in a restricted case. "Only trusted signed code may use SMT" for example, that won't work everywhere and it doesn't mitigate SMT still being a dangerous feature, but it may be plenty good enough in some instances. Or perhaps there is a server where the only truly sensitive piece is the private key, and that simply gets offloaded into an HSM blackbox and thus isn't available to be compromised anyway. Or the auth/signing servers disable it but other boxes do not.

Basically, it just seems like it's too valuable a feature and the threat scenario not universal enough for it to be simply abandoned entirely as the authors suggest given the physical and economic realities we're facing right now with silicon.

Yeah, I don't think SMT is going away. It's one of the tricks CPU vendors have to eke out more performance, now that they can no longer significantly boost clock rates and shrink die sizes. So we will still have supercomputers and such with this technology enabled, but they will be air-gapped and acoustically isolated, or maybe just have their network perimeters beefed up. And then for mission-critical, sensitive server loads SMT will be disabled (or they will use SMTless CPUs).
Does anyone have a tech brief on why SMT is more dangerous than any other type of shared resource? i.e. There are several resources that are shared between cores on a CPU, is there something inherent about SMT that makes the resources it shares more attackable or is this just a bug?
Basically timing attacks are completely unavoidable on SMT because SMT fundamentally depends on what the other thread is doing on the core, if the thread is using few core resources the attack thread gets more, or vice versa. This gives you a very fine grained view into what the other thread is doing timing wise, and is inherent to the concept of SMT (two threads running on a core at once, using resources not utilized by the other).

I suspect this issue is completely impossible to resolve in anything except very constrained situations that negate the benefit of SMT entirely.

Its different than say, a timing attack on the OS scheduler for two reasons

1. OS scheduling is very coarse (A process gets exclusive use of a core for tens of thousands or hundreds of thousands of cycles), with SMT you get a very fine grained near cycle-by-cycle view of core utilization by the other process.

2. you don't get much of a view into what precisely the other thread is doing such as is it doing lots of integer operations, floating point, memory fetches etc, which can be derived in SMT attacks based on how fast such things occur for the evil monitoring thread.

Is this not a problem with, for example, L3 cache that is shared across cores? Is there something inherent about it that makes timing attacks impossible or is it just that we're seeing people go after lower hanging fruit?
Cache timing attacks are definitely a real thing, but cache can at least be isolated/partitioned (even if its not really currently).
SMT is like a shared kitchen. Even if you're unable to see the other person in it, if you go to use the sink, oven, or microwave and have to wait because it's in use, you know some small bit about the other occupant. Knowing how long you have to wait tells you more. Knowing what is already stocked in the fridge from the other occupant that you can immediately use tells you a lot.

Separate cores are more like completely separate kitchens. Sure, the grocery delivery service might take shorter or longer some times, but it's hard to tell much beyond general area business from that. What's in your fridge is what you put there.

The OpenBSD project predicted more was coming after Spectre/Meltdown and disabled SMT by default.
Sure, but OpenBSD has a very specific project goal and use cases (which is absolutely as it should be!) and their stance thus makes sense in context. But it doesn't mean their stance is universally applicable either, anymore then best practices for IaaS all apply everywhere. For an alternate feature example: some OpenBSD committers for years have reasonably rejected OpenZFS [1] requests because it was considered to be just plain too big, and in turn presented more threat surface and higher system resource consumption (big issue for embedded/firewall appliance applications) then was desirable for them. All of which is true and makes total sense there. But not for a lot of big servers and end user systems where I think automatic full data consistency/checks/regeneration, easy full replication, trivial cheap atomic snapshots, portability between many OS, and so on, address much more important threats then the extra load or any unknown potential vulns. No need for a monoculture ecosystem here rather then using the right specific tool for the job right?

-----

1: https://www.tedunangst.com/flak/post/ZFS-on-OpenBSD

His statement is particularly hyperbolic given that no one has shown a working side channel attack on a Ryzen CPU that relied on SMT. The article is littered with repetitions of suspicion, but there hasn't been any relevant code or results published anywhere to my knowledge.
> Of course perhaps SMT can in fact be fixed for this without a wholesale tossing in which case it'll just be a universal hardware revision somewhere down the line.

There's a very simple solution here. Don't schedule two threads with different trust domains on different SMT threads on the same core. No need to change any hardware, no need to disable anything, just accept (as has been known for at least a decade) that there is very likely to be a side channel attack if you look hard enough when SMT is involved.

At what point does the extra complexity and cost of including SMT in the design outweigh the gains is provides if you can only use it in specific situations, and if you don't err on the side of caution is may be a security problem?

If it was clearly outlined as 8 cores that support hyperthreads or 10 cores that don't for around the same price, what do we suppose most people would choose?

Part of the reason for the current status quo might be that hyperthreads are a big differentiator beteen AMD and Intel, and play towards Intel's strength (higher clock speeds at fewer cores).

We have something close to the proposed scenario starting to play out with AMD's offerings that support more cores and no hyperthreading, but it's not a perfect experiment because AMD's cores are also lower clock speed, and there's a lot of brand name loyalty currently.

The point of SMT is that it is quite low overhead. You're not duplicating most of the core, only the physical register file and the TLB units (I think). The execution units and caches--where most of the die of a core goes--remain unreplicated. In terms of area tradeoff, you're not looking at "we'll give you 2 more cores if we disabled SMT." For the Pentium 4 (quickest I could get), adding SMT is a die overhead of 5%. Your realistic trade-off for area constraints is on the terms of SMT, or better out-of-order parallelism scavenging, or more L2 cache.
> For the Pentium 4 (quickest I could get), adding SMT is a die overhead of 5%.

Ah, that's what I was looking for. I'm aware that most the resources are shared, but I also assume there have been design choices in other components to make SMT easier or faster, and possibly that increases their die size a small amount, or in general just complicates their design.

The question (which is ultimately unanswerable) is what would we have had Intel not chosen SMT as the path to pursue? If they had instead invested those resource into other areas (e.g. more cores) and never let SMT concerns enter into the discussion, how would those (theoretical) CPUs compare?

That's the CPU comparison I was alluding to in the prior comment, and why I noted AMD vs Intel (even with all it's other differences) may be the closest match up of that idea we'll see.

It wasn't meant as a rebuttal to anything, it was more a "wondering out loud" type of comment spurred by yours, about what could have been had different paths been taken.

> There's a very simple solution here. Don't schedule two threads with different trust domains on different SMT threads on the same core.

A single thread already has two different trust domains (user mode and kernel mode), so any pair of threads will have different trust domains when one of them is running a system call or an interrupt. So this solution would also have to either pause the SMT sibling while entering the kernel, or run the kernel code in a separate core (an idea like this was briefly suggested on the RISC-V isa-dev mailing list a few months ago: to have a separate core which handles interrupts and kernel mode for a group of user-mode-only cores).

(comment deleted)
Intel's response to this is to cut back on Hyperthreading significantly in the latest generation: https://ark.intel.com/compare/134896,186605,186604
I don't see it as a response to vulnerabilities, it's just new flavours of market segmentation. They have flagship CPU with HT enabled after all.
The i9 is their "performance at any cost, even security" chip. It's for people who want/need the absolute fastest chip and damn the torpedoes. It's so far past the optimal point on the price/performance curve that nobody sensible will be buying it unless they have a very specific need.
I'm no fan of the i9 series at all, but if you have SIMD-heavy workloads (especially anything you're willing to port to AVX-512), those CPUs actually make a lot of sense price/performance wise. It's a fairly specific niche, but not necessarily a tiny one.
It would be suicide to market it in this way. They actually made some security hardware improvements and highlight it. Sure, you could disable hyperthreading and I agree that it might be beneficial for side-channel attack protection, but it's not like Intel is officially accepts that as a valid option. At least I didn't see anything like that.
I think that's mostly a market segmentation measure given the timing, though in a sense you're right since it does protect consumers. They do have SMT enabled on most laptop processors, too. I worry that they'll have a hard time getting all they can out of the full width of Haswell and later chips without SMT.
Just throwing it out there - the researcher said "he strongly suspects that AMD CPUs are also impacted."
I hope someone who isn't as incapable as me will try to make it work there. It seems Intel is always the primary target since it has a much greater market share, but if we want to consider AMD the safer x86 alternative we should actually check their CPUs too :)
I'm also curious if IBM's POWER9 architecture is effected. They actually do 4-way SMT.
So instead of testing it, researcher just speculated.
I have a somewhat naive question. Since this and many similar recent issues are associated with SMT (Spectre, Meltdown), and the concern is about malicious code being run on the same core.

Can IaaS vendors simply restrict VMs to always use whole cores. If you want 3 cores in your VM, you get

    1. Core0 Main thread
    2. Core0 Hyper thread
    3. Core1 Main thread
    Core1 Hyper thread is un-allocated
And then we don't have two actors on one core? Or just only offer 2-core VMs.
It looks like EC2 is already doing this. I don't think they ever shared cores except for t1 and t2.
This is definitely a nit, but isn't calling this a "new vulnerability" incorrect? The vulnerability has existed, but has only recently been discovered. I'd prefer it was referred to as a "new exploit."
It is "new" in the sense that it has not been described before.

Much like a mathematical theorem, which of course has been true all the time since someone formulated the axioms, but when someone proves it for the first time it is "new".

OS could make Hyper threading opt-in per application thread. So chrome could mark its render threads as HT-groupable and they can share a core between them, but otherwise cores are not shared.

There might be different levels of thread opt-in, prime95 might not care about other threads finding out what the 338134th digit of prime is and mark its threads as unrestricted sharable.

This is the right approach, IMHO. Let me run local apps like video rendering and transcoding using all cores, but let me mark anything that has Internet access as only executing on a dedicated core.

Apple's new hardware approach might work well for this: on the A12X chip used by the new iPad Pro, they have 4 high performance cores and 4 low power cores. Let Chrome have a couple low power cores, and that solves a few problems:

1. I get security isolation between Chrome and the rest of my apps. 2. I no longer care very much when poorly written javascript wants to consume 100% of CPU because it's only running on a low power core. 3. My battery life is also better because Chrome can't consume a high power core.

It should be possible to do the same thing on anything that supports CPU frequency scaling. Tell the OS that the core the browser is running on should only run at the lowest supported frequency. Or only allow it to run at higher frequency if it hasn't been active for more than 15 of the last 60 seconds.
Unfortunately this contention is the exact reason this isn't done. Similar workloads (from the same program) will perform poorly in the same SMT groups. You can try this in windows using the core affinity of processes. You'll find that when one application shares both HTs it will perform much worse.
Under optimal conditions (rendering) enabling SMT on Intel gives just ~30% extra performance, ~40% on AMD (usually much less), so of course you can't compare the additional thread provided by SMT to a thread running on another, otherwise idle core.

Scheduling only threads of the same process on a SMT group at a time is the easiest portsmash mitigation while still retaining some of the SMT peformance benefit.

Seems like variations of these bugs are going to wipe out past 10 years of CPU performance advancements... As security research tooling gets more advanced, it can possibly touch even more intrinsic areas of computation, leading to orders of magnitude slowdowns in exchange for minuscule increases in security - when is security finally good enough?
Intel has done nothing it can stand behind since the pentium 3. That chip was amazing. They had to dead end P4 development and go back to a P3 base to develop core. Now I betcha they will now have to dead end core and go back to P3 again. It could be competitive with a die shrink, clock boost (3.5Ghz P3? drooool), a newer instruction set (I'm sure it's fast enough to decode, and intel is good at adding accelerators), and then running a ton of them on a single socket like AMD is doing. Maybe pair it with some optane for instant boot or some gimick. It'd sell.
As much fun as a 4 GHz Tualatin 4- or 8-core CPU sounds, it would probably take several years of development to make something usable by today's standards.
> "This is the main reason we released the exploit -- to show how reproducible it is," Brumley told us, "and help to kill off the SMT trend in chips."

Maybe what we should be killing off instead is exporting everything to the cloud and running untrusted native code willy nilly.

It's not just untrusted native code. A lot of these exploits in general are exploitable through js.

And I know there are folks on hn who think js is an abomination and noscript is the answer to all of life's persistent problems. "My web browser should be exclusively for reading text." But personally I'm not interested in taking us back to the 1990s.

In what ways could JS be limited to preclude such attacks? (E.g. no JIT compilation, cache flush after every opcode, etc.).
No constant addition and removal of new APIs. Lockdown of special purpose APIs with explicit user request to unlock: 99.9999% of sites will never need WebGL, 99.9999% of sites don't need an Audio API. Modern JavaScript has the attack surface of Java and Flash combined and enabled by default, instead of hardening this against attacks browser are stuck in the same feature race they have been in since back when IE6 was still dominant browser.
> cache flush after every opcode

This would make JS outrageously slow, like hundreds of times slower than it is now.

You don't need it to be very fast just to respond dynamically to some event handlers and whatnot. (The basic scripting stuff JS was intended for.)
You also don't need modern wheels, when just pulling a simple carriage wheels were invented for.

Modern JS engines are not written for playing funny tricks with "window.status" or "window.title" as in 1990, but for more complex applications.

Not all uses are useful, but that's to subjective. As with wheels.

I don't need to be in the 1990s, I just wished websites wouldn't need to draw from ~200 untrusted sources and possibly worlds largest malware distribution channel to show some ads from even less trusted sources right next to badly formatted text.
I mean the text is probably from poorly trusted sources as well but that's a different problem.
but text has a limited attack scope compared to say: an entire interpreted language and DOM.
> I just wished websites wouldn't need to draw from ~200 untrusted sources

That's just how it is though. Even if you're using vanilla js.

Don't get me wrong, I know what you mean... it's just that it's really hard to actually find arguments for that... philosophy

At this point, I think it doesn't really matter as long as your site has a good ux.. which is obnoxiously hard to measure

With Noscript enabled I have the power to chose what to trust and not to trust. That means I have gradual control over what runs and doesn't run, and can chose to only allow the bare minimum for getting a specific page to run. It's that simple.

JS is fine as a technology, but blindly executing every single piece of code on every single web page is simply not a good way of doing things.

Yep.

I understand that half the world runs on this stuff nowadays. That doesn't make it any less insane.

Your customer database is in a random DC somewhere halfway across the world running on the same box as some other arbitrary code.

Nice one. It's Web Scale(tm).

Physical containment would work fine. Given the current small footprint of computing modules, we could just run each separate untrusted cloud apps from a node the size of a pi0. All we need is a high throughput interconnect with a well defined interface to exchange information.

... wait, that sounds like plan9's dream

Any cloud provider with sufficient security doesn't allow customers to share SMT cores anyway. For example, AWS hard partitions their physical servers, which is why you can get exactly 16 m4.xlarge instances, 8 m4.2xlarge, 4 m4.4xlarge, 2 m4.8xlarge, or 1 m4.16xlarge instances on a physical box.

I do predict that the VPS providers will see some nasty side-channel attacks in the coming years due to the way they've chosen to share SMT between customers to reduce their costs and overprovision their hardware.

It seems that for 'personal' devices (i.e. laptops/desktops), the biggest vulnerability here is probably actually javascript.

You can be smart about what software you run, but most people don't use the web without JS.

Maybe it's time for javascript to be 'off by default'.

I got heavily downvoted for suggesting that maybe google shouldn't be forcing JS for login yesterday.

I feel somewhat vindicated with this, however the fact remains that the web today doesn't really work without JS. I don't see that changing for any reason other than it offering a better (or more consistent) experience, but that requires web developers to support that. Which I don't see happening any time soon.

My understanding is that best practice for writing crypto libraries these days is that you avoid any data dependent jumps and always execute the same instructions every time, using cmov or similar instructions to make use of one calculation or another. In that case the port usage should be the same regardless of input data and you should be immune to this attack and many classes of timing attack you might be otherwise vulnerable to. You've still got to worry about data dependent power draw but that's hard to attack even if you have the chip on a bench in your lab. So would this attack actually be workable against crypto libraries designed that way? Not that the ones in the wild are necessarily but this might be an impetus.
Hyperthreading is thoroughly done.