Because Intel writes almost all of the x86 arch code for Linux, so AMD gets stuck footing the bill for hardware bugs like this, even if AMD isn’t affected. From Linus’s perspective it’s the same arch, because, well, it is.
Well he does say patches should written that don't treat all CPUs as crap. AMD had to basically say "Wait a moment! This does not affect our CPUs what are you doing Intel?"
Well there is spectre, and meltdown. Spectre is much more limited than meltdown. The patch AMD was referring to was for Meltdown.
Spectre at least does not leak ring-0 memory. Sure it means that some application might be able to read/infer memory values of other user-mode process.
Spectre is much scarier than Meltdown, from my understanding, because it represents a whole class of strategies that lack a single fix, even at the hardware level.
My understanding is that it's not a bug of unintended behaviour, but rather a problem with the specification, all behaviour is as intended, but we need to have different behaviour. Which means that basically we need to get a new ISA -- or a new version of an existing one -- and transition everything to it.
Amd claims that two of the three spectre attacks don't work on their hardware and the last one (variant one) is fixable with negligible performance impact.
Differences in AMD architecture mean there is a near zero risk of exploitation of this variant. Vulnerability to Variant 2 has not been demonstrated on AMD processors to date.
Pretty common in our industry to use apostrophes after an acronym to make it plural. An example is VM's and VMS are two different things. When you get lazy and do vm's and vms you can not always be sure, I have even written an email where I stated "we need multiple VMS VM's".
It's not uncommon for natural languages to have homonyms, so people are used to deal with them. The context allows deciding which is the correct meaning.
> Using an apostrophe is the technically correct way to pluralize acronyms.
No, it's not the “technically correct way”. It's true that some style guides recommend something like this (but always, AFAIK, more limited); e.g., the New York Times style (and Chicago style) uses apostrophes for plurals of abbreviatiins formed with capital letters and periods, and plurals of single letters. [0] AP, on the other hand, only uses apostrophes for plurals of single letters, but not for abbreviations/initialisms/acronyms. Most British styles I've seen don't use apostrophes for plurals at all, IIRC.
As long as the style is consistent, it doesn't really matter.
The general newspaper style is acronyms with periods use apostrophe to separate period from s. Therefore M.D.'s and Ph.D.'s are plural. Where period is not used in the acronym, the apostrophe is not used, so CPU not C.P.U., therefore it's CPUs. Anyway, it's just a convention, the key is consistency.
Is this mentioned in any particular newspaper style guide that a general member of the public can get their hands on? Like the AP style guide for instance?
So is there no way to disable the performance crippling fix? If you ran an air gapped or highly controlled environment wouldn’t you want th option of disabling this to keep performance? How many Giga Terra or Exo FLops will be lost to this fix in super computers? Or is the way to disable it just not merge this patch into your custom kernel code?
This isn't the fix with the 5% quoted performance loss (that one clears all of TLB on every syscall), I don't think this is as costly. But it's certainly not free and it's certainly not clean.
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset. Benchmarks are ordinarily expected with something like this (very hot paths they are changing), and they did have 6 months..
This is worse, far worse --- it basically disables prediction for indirect jumps and calls, plus adds another few extra clock cycles to each one of those for the "retoline" instruction sequence.
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset.
You should be disappointed that they didn't report it. But I'd be willing to bet every last dollar of my net worth that Intel has done very extensive performance testing.
I'd venture that the results are far too embarrassing to report.
Yup this for sure. They probably ran the performance test bench thousands of times, and only then released it with the PR department fully briefed about how bad it was going to be.
So what? IIRC we used to have (and maybe still have) a feature that patched out spinlocks in binary code when it detects it is run on monoprocessor systems. You "just" have to identify all the call sites, and patch the code there during the boot process.
This seems unnecessary hostile. It's quite obvious that these patches had to be completed in a hurry, and in addition to any number of similar patches for other systems. Configurability is currently useless for Intel CPUs, as all of them seem to be affected.
Depending on how their CPU development pipeline works, it's quite likely that even the next generation will still be affected, giving everyone plenty of time later for such niceties.
It's also slightly too harsh to call all of Intel's work "crap" when this bug has apparently existed for the better part of two decades without being noticed.
I agree with Linus though. Not that I've ever worked on a big project like the kernel but things like these will take a backburner.
The kernel maintainers don't have a way to bring these things up in Intel's boardroom like Apple can and it is possible that Intel will never produce a proper fix.
I didn't take the comment as "all Intel CPUs are crap," but that (1) the fix is needed for crap CPUs and (2) the fix is turned on unconditionally on Intel CPUs, even future CPUs. That is, the code itself assumes that all CPUs that could be crap actually are crap. Torvalds wants a flag that could be set appropriately when Intel starts shipping CPUs that don't have this problem.
Considering Intel is working on this for the last 6 months, it is indeed a little telling that they did not have the patch behind a feature flag. It also explains why AMD was so passive aggressive while inserting the exception for their CPUs. Seems like Intel was trying to make this some sort of act of god type of deal instead of writing out the if(intel){horrible mitigation}.
Love it, an LFENCE on every indirect jump in core 32/64bit entry assembler code, not to mention the cost of the trampoline itself. Linus is right to say that this hack should be reserved for configured as broken CPUs..
If I'm reading it correctly, the lfence is never reached for real (only speculatively..)
code invoking the thunk pushes the target address on the stack, then branches to or calls the thunk.
The first instruction of the thunk is a "call" to a nearby address, skipping over an infinite loop (2: lfence; jmp 2b)
the lea n(%xsp),%xsp discards the return address pushed by "call", and the subsequent "ret" "returns" to the branch target rather than to label 2.
perhaps the speculative execution engine doesn't take the stack pointer adjustment into account and so gets stuck in the lfence loop, only being snapped back to its senses when the lea completes..
This would be more convincing if the x86 CPU manufacturer he used to work for were still in business.
I mean, I could sit here and criticize how Linus runs his kernel development project and how so much insecure code gets in, and talk about what a "competent" kernel developer would do, but since I don't run an equally successful, more secure kernel, it's just armchair quarterbacking.
Yes, sort of - I don't think he has the evidence to claim that a competent CPU engineer would have done things in a certain way, because all the really theoretically good CPUs (and I count Transmeta among them!) seem to have not succeeded in the market nearly as well as the messy, awful ones, and a CPU that no longer exists doesn't actually count for anything. A competent CPU engineer is balancing lots of tradeoffs. It might be the case that he's right, but in the absence of evidence I weakly believe he's not, and I strongly believe that he's not in possession of strong enough evidence.
I wonder why it is that when I say "I don't think that X should be doing Y in this situation because Z," I so often get responses of the form "So you think that nobody should be doing generic-version-of-Y ever?" That's probably a flaw in my own ability to convey my points. Let me see if I can convey this one a bit better.
First, I think that there's a distinction between constructive, humble criticism, of the form "Why can't the CPU be designed in such a way that ___", and statements like "Well, a competent CPU designer would have done ___". That distinction is not just politeness (although politeness is of intrinsic worth); that decision leads to better technical outcomes. There might be a good answer, like "It can't be designed like that because of the following constraints" or "We tried it and couldn't get it to work"; there might just be an answer like "That's totally the right thing to do and we just cut corners" or "We never thought of it but our competitor does that and it seems to work for them, yeah." Unlike arguments about how competent people actually are, these discussions contain actual technical content, and help different people understand each other's points of view (which is the whole point of having an online, public, decentralized development community, is it not?).
Second, yes, I think that we should discount, though not by much, criticism from people who say "I could totally do that in a better way" and have not actually tried to do that (or have tried and failed to launch a product). There's a lot of things about building complex systems (like modern CPUs) that aren't apparent on paper, and only become apparent once you try to reach some challenging level of scale or performance. This is not at all to say that there's no place for outside criticism; often outside criticism can offer fresh perspectives, or suggest a way to simplify the system and avoid the emergent problems of complexity. And often there are people who are genuinely qualified and really could build a better system, but happen to be working on something else. But it's still fundamentally outside criticism. The only people who can really say "Yes, you totally could have done this instead" are those who actually did.
Third, in this particular scenario, it happens to be the case that ex-Intel employees with insider knowledge have called out Intel for cutting corners (see the "Update" section of https://danluu.com/cpu-bugs/), which seems like a much more productive basis to have a technical conversation.
And if those stories are to be believed and if they're relevant to this problem, the problem is not the competence of engineers - the problem is the competence (or the confused incentives, really) of management who are not telling their engineers to invest time into doing things the engineers know perfectly well how to do, or are telling their engineers to invest time into building other features that make those things harder. (For instance, quite possibly some part of management got worried about benchmarks of syscall performance and told their engineers to prioritize that over making sure that there aren't side-channel leaks between rings.)
Sorry, I have a weakness for reductio ad absurdum, so fair enough to call me out on that. But in this case I'm glad I gave in to temptation, because your response was really valuable.
I think a kernel developer, let alone a core kernel developer and the guy who started the entire project by himself, is probably one of the BEST qualified to comment on the situation.
That was my point - on kernel design, yes, he's way more qualified to comment than I am. But his claim was about CPU design, not kernel design. He's about as qualified to comment about CPU design as I am on kernel design.
Yes, that's what I said. Transmeta is out of business; all their clever ideas have nothing to show. It is entirely possible that a "competent CPU engineer" needs to make certain things absolutely work with high performance even if the cost is leaky side channels, and someone who worked at a failed CPU company does not know that because they never got to see their product succeed in the market.
Do you seriously think that winning the market equals being right? "The market" is a horribly ill-idealized model full of wrong assumptions. Uninformed customers, unfair practices and outright illegal behaviour need to be expected. "The market" is a very bad indicator as to who was right.
If the claim being made is "well obviously they should have just done foo and it would have worked," there's reason to believe that foo gives you a significantly less competitive product, and the speaker is not giving any evidence for why it's possible to do foo and still succeed in the market, the only thing we have to go on is the speaker's experience in that market.
A rigorous argument for why foo can be done and doesn't kill your product would be fine too, but this email doesn't even attempt to do that. What can be asserted without proof can be dismissed without proof.
Transmeta's IP, and its ideas, were gobbled up and put to good use when the company folded.
It was a clever microarchitecture; they just targeted the wrong consumer segment (breaking into x86 was nigh-impossible that late; chip internals on the other hand... =D)
I'd say having worked on CPU architecture, regardless of the success or failure of it, gives Torvalds even more credibility talking about this sort of thing.
Having said that, though, I think calling it "crap" is a bit much. But that's still a pretty mild/weak insult in Torvalds-speak.
Looking at the patches Linus is commenting on: as an Asm programmer, this is absolutely horrible --- basically every indirect call (one instruction) turns into a seven-instruction sequence that will, due to preventing speculation, result in massive slowdowns:
Unlike the KPTI patches, which only affect things on each system call, this happens on every indirect call and probably bloats the code considerably too. I can see why Linus is not happy.
Edit: To give a bit more background, predicting indirect calls and speculating into them is absolutely critical to getting good performance from OOP-ish code which tends to use them a lot (virtual functions in C++, function pointers in C). The Linux kernel is (thankfully?) not very OOP-ish, but it does rely on indirect calls (function pointers) extensively.
I missed that, actually. But you are right, it's not just the crucial bits of inline assembly they are patching here, they expect your kernel to be built with their special version of GCC (yet unreleased) that mutilates every single freaking indirect jmp/call into what could cost more than a virtual function call..
We use a new compiler option -mindirect-branch=thunk-extern
Every single call rax and jump rax that you see on the right will be replaced with the following:
push rax
jmp .indirect_thunk
And indirect_thunk does an lfence and a call to another function that gets rax from the stack and jumps there in a very roundabout way that interacts terribly with CPU prediction. I can see why they somehow didn't get around to benchmarking this yet, because they are not going to like the results. And it doesn't end with the kernel. Every JIT is going to need the treatment.
Interacting terribly with CPU branch prediction is the whole point -- Spectre attacks work by messing with the branch predictor, to misdirect speculative execution onto paths that reveal information useful to the attacker. So, the point of this whole crazy dance is to effectively disable the branch predictor.
Of course. But it needs spelling out, because branch prediction is essential to good performance and why todays CPUs can be much faster despite not reaching higher clocks.
It's much more than just the cost of a few more instructions.
Apparently the perf impact isn't so bad outside of microbenchmarks actually. 10% for real world programs ... as long as you use profile guided optimisation to devirt as many calls as possible!
(some websites with info on this are DoSd and down right now, including the Arm website and LLVM code review site so switched to a cache link)
Pretty painful for C++ but fixable with build changes. JVM users and other JIT-compiled languages will probably be fixable without any serious impact, as HotSpot and other advanced runtimes devirtualise calls (convert indirect to direct) as much as they can anyway, meaning most calls won't be indirect when finished and once the code has warmed up.
Isn't that patch for the bug that's present on basically every CPU, Intel, AMD, ARM, etc., though? I thought the Intel-specific bug was fixed by "blank out the TLB every sysret".
I'm frankly shocked Linus was anywhere near as restrained as he was upon seeing this monstrosity. Nothing of this sort should ever make it into the kernel without explicit and loudly announced opt-out switches. If anything close to this does make it in, even as an opt-in, this will easily be the biggest performance regression in recent (perhaps all-time?) kernel history.
His restraint and the fact that this stuff is even being considered tells me this is a really really bad bug. I doubt he would be so cordial in any other situation.
Performance at the expense of security is a compromise made daily in every aspect of IT. The only way to make systems perfectly secure is to make them unusable.
Now, you can certainly argue that this particular tradeoff wasn't worth it, and in hindsight it's certainly a questionable one. However, do note that side channel attacks have been known for decades (since out-of-order CPUs became widespread in the 90s) and until this point have rarely presented a significant threat to most systems.
I'd be more inclined to think this was someone who didn't understand the implications implementing a performance improvement. This bug is clearly deep into unk-unk (i.e. well past the known areas of concern) territory and took a great deal of insight to the fundamentals of speculative execution to discover.
This is my understanding, please correct me if wrong:
There's a basic class of bugs (let's call it Spectre-class) that's about exfiltration of memory contents via side channels introduced by speculative execution. Basically every processor that can speculatively execute code (i.e., every modern one in the past two decades or so) is potentially vulnerable.
Of known Spectre-class bugs, one (Spectre) relies on branch predictor tables. All tested CPUs (ARM, AMD, Intel) were vulnerable to Spectre. The other bug publicly announced (Meltdown) relies on when the #GP(0) exception actually gets thrown and is specific to Intel CPUs. The fix for Spectre is to basically banish branch prediction, and the fix for Meltdown to is to unmap kernel pages when in userspace code.
Beyond these two bugs, there may exist other Spectre-class bugs specific to different architectures/vendors that are not yet thoroughly investigated.
96 comments
[ 2.6 ms ] story [ 144 ms ] threadSpectre at least does not leak ring-0 memory. Sure it means that some application might be able to read/infer memory values of other user-mode process.
https://www.amd.com/en/corporate/speculative-execution
Differences in AMD architecture mean there is a near zero risk of exploitation of this variant. Vulnerability to Variant 2 has not been demonstrated on AMD processors to date.
Near-zero is not zero.
No, it's not the “technically correct way”. It's true that some style guides recommend something like this (but always, AFAIK, more limited); e.g., the New York Times style (and Chicago style) uses apostrophes for plurals of abbreviatiins formed with capital letters and periods, and plurals of single letters. [0] AP, on the other hand, only uses apostrophes for plurals of single letters, but not for abbreviations/initialisms/acronyms. Most British styles I've seen don't use apostrophes for plurals at all, IIRC.
[0] https://afterdeadline.blogs.nytimes.com/2010/04/13/faqs-on-s...
The general newspaper style is acronyms with periods use apostrophe to separate period from s. Therefore M.D.'s and Ph.D.'s are plural. Where period is not used in the acronym, the apostrophe is not used, so CPU not C.P.U., therefore it's CPUs. Anyway, it's just a convention, the key is consistency.
I'm disappointed Intel didn't bother to test the performance impact of this and report it in the patchset. Benchmarks are ordinarily expected with something like this (very hot paths they are changing), and they did have 6 months..
You should be disappointed that they didn't report it. But I'd be willing to bet every last dollar of my net worth that Intel has done very extensive performance testing.
I'd venture that the results are far too embarrassing to report.
Additionally: you might update infrequently when it comes to feature releases, but you'll definitely need security patches.
Depending on how their CPU development pipeline works, it's quite likely that even the next generation will still be affected, giving everyone plenty of time later for such niceties.
It's also slightly too harsh to call all of Intel's work "crap" when this bug has apparently existed for the better part of two decades without being noticed.
Not that I disagree with you, but are you familiar with Linus’ other uhhh...critiques?
http://lkml.iu.edu/hypermail/linux/kernel/1510.3/02866.html
The kernel maintainers don't have a way to bring these things up in Intel's boardroom like Apple can and it is possible that Intel will never produce a proper fix.
"We reported this issue to Intel, AMD and ARM on 2017-06-01"
https://googleprojectzero.blogspot.com/2018/01/reading-privi...
code invoking the thunk pushes the target address on the stack, then branches to or calls the thunk.
The first instruction of the thunk is a "call" to a nearby address, skipping over an infinite loop (2: lfence; jmp 2b)
the lea n(%xsp),%xsp discards the return address pushed by "call", and the subsequent "ret" "returns" to the branch target rather than to label 2.
perhaps the speculative execution engine doesn't take the stack pointer adjustment into account and so gets stuck in the lfence loop, only being snapped back to its senses when the lea completes..
I mean, I could sit here and criticize how Linus runs his kernel development project and how so much insecure code gets in, and talk about what a "competent" kernel developer would do, but since I don't run an equally successful, more secure kernel, it's just armchair quarterbacking.
First, I think that there's a distinction between constructive, humble criticism, of the form "Why can't the CPU be designed in such a way that ___", and statements like "Well, a competent CPU designer would have done ___". That distinction is not just politeness (although politeness is of intrinsic worth); that decision leads to better technical outcomes. There might be a good answer, like "It can't be designed like that because of the following constraints" or "We tried it and couldn't get it to work"; there might just be an answer like "That's totally the right thing to do and we just cut corners" or "We never thought of it but our competitor does that and it seems to work for them, yeah." Unlike arguments about how competent people actually are, these discussions contain actual technical content, and help different people understand each other's points of view (which is the whole point of having an online, public, decentralized development community, is it not?).
Second, yes, I think that we should discount, though not by much, criticism from people who say "I could totally do that in a better way" and have not actually tried to do that (or have tried and failed to launch a product). There's a lot of things about building complex systems (like modern CPUs) that aren't apparent on paper, and only become apparent once you try to reach some challenging level of scale or performance. This is not at all to say that there's no place for outside criticism; often outside criticism can offer fresh perspectives, or suggest a way to simplify the system and avoid the emergent problems of complexity. And often there are people who are genuinely qualified and really could build a better system, but happen to be working on something else. But it's still fundamentally outside criticism. The only people who can really say "Yes, you totally could have done this instead" are those who actually did.
Third, in this particular scenario, it happens to be the case that ex-Intel employees with insider knowledge have called out Intel for cutting corners (see the "Update" section of https://danluu.com/cpu-bugs/), which seems like a much more productive basis to have a technical conversation.
And if those stories are to be believed and if they're relevant to this problem, the problem is not the competence of engineers - the problem is the competence (or the confused incentives, really) of management who are not telling their engineers to invest time into doing things the engineers know perfectly well how to do, or are telling their engineers to invest time into building other features that make those things harder. (For instance, quite possibly some part of management got worried about benchmarks of syscall performance and told their engineers to prioritize that over making sure that there aren't side-channel leaks between rings.)
A rigorous argument for why foo can be done and doesn't kill your product would be fine too, but this email doesn't even attempt to do that. What can be asserted without proof can be dismissed without proof.
It was a clever microarchitecture; they just targeted the wrong consumer segment (breaking into x86 was nigh-impossible that late; chip internals on the other hand... =D)
Having said that, though, I think calling it "crap" is a bit much. But that's still a pretty mild/weak insult in Torvalds-speak.
https://lkml.org/lkml/2018/1/3/770
Unlike the KPTI patches, which only affect things on each system call, this happens on every indirect call and probably bloats the code considerably too. I can see why Linus is not happy.
Edit: To give a bit more background, predicting indirect calls and speculating into them is absolutely critical to getting good performance from OOP-ish code which tends to use them a lot (virtual functions in C++, function pointers in C). The Linux kernel is (thankfully?) not very OOP-ish, but it does rely on indirect calls (function pointers) extensively.
We use a new compiler option -mindirect-branch=thunk-extern
https://godbolt.org/g/LPycpY
Every single call rax and jump rax that you see on the right will be replaced with the following:
And indirect_thunk does an lfence and a call to another function that gets rax from the stack and jumps there in a very roundabout way that interacts terribly with CPU prediction. I can see why they somehow didn't get around to benchmarking this yet, because they are not going to like the results. And it doesn't end with the kernel. Every JIT is going to need the treatment.https://support.google.com/faqs/answer/7625886
http://webcache.googleusercontent.com/search?q=cache%3Ahttps...
(some websites with info on this are DoSd and down right now, including the Arm website and LLVM code review site so switched to a cache link)
Pretty painful for C++ but fixable with build changes. JVM users and other JIT-compiled languages will probably be fixable without any serious impact, as HotSpot and other advanced runtimes devirtualise calls (convert indirect to direct) as much as they can anyway, meaning most calls won't be indirect when finished and once the code has warmed up.
https://godbolt.org/g/MPs43x
* https://reviews.llvm.org/D41723
* https://news.ycombinator.com/item?id=16070050
* https://news.ycombinator.com/item?id=16070356
Now, you can certainly argue that this particular tradeoff wasn't worth it, and in hindsight it's certainly a questionable one. However, do note that side channel attacks have been known for decades (since out-of-order CPUs became widespread in the 90s) and until this point have rarely presented a significant threat to most systems.
There's a basic class of bugs (let's call it Spectre-class) that's about exfiltration of memory contents via side channels introduced by speculative execution. Basically every processor that can speculatively execute code (i.e., every modern one in the past two decades or so) is potentially vulnerable.
Of known Spectre-class bugs, one (Spectre) relies on branch predictor tables. All tested CPUs (ARM, AMD, Intel) were vulnerable to Spectre. The other bug publicly announced (Meltdown) relies on when the #GP(0) exception actually gets thrown and is specific to Intel CPUs. The fix for Spectre is to basically banish branch prediction, and the fix for Meltdown to is to unmap kernel pages when in userspace code.
Beyond these two bugs, there may exist other Spectre-class bugs specific to different architectures/vendors that are not yet thoroughly investigated.
What did I get right and wrong?