175 comments

[ 2.6 ms ] story [ 174 ms ] thread
May be time for OpenBSD people ( if they have the resources ) to build a CPU from ground up with RISC-V?
Just crowdsource a few billion $ for R&D and we are good to go.
The first and most important phases of hardware design are a lot like software development. If open source folks can write an OS without much funding, then they can design a CPU.
Arguably, the most important phase of hardware design is actual production of said design.
I somewhat doubt that is still the case, with modern EDA tools.
I think they were maybe making a joke about whether you want to use the chip or not.
I think I mostly meant to hint at the [financial] difficulties of producing (and distributing) computer cpus at scale. And that a product which exists only on paper doesn't really exists.
> If open source folks can write an OS without much funding, then they can design a CPU

Large open source projects tend to have significant amount of funding - think about Chrome, OpenJDK, the Linux Kernel...

I don't know if you're joking, but that may not even be an impossible task if say OpenBSD were to create a cryptocurrency for their OS, as long as they tried to make it as legal as possible and try to see it more like a crypto-asset/stock kind of thing than a "digital ponzi scheme."

Seems to have worked for the IPFS guys (I'm talking about existing projects that only launched a cryptocurrency later on, rather than from the start). Brave's BAT qualifies for that, too.

There are already CPUs available that aren't affected by these bugs. The problem is that most people want the fastest CPU for a given price -- and making a new, slow, expensive CPU isn't going to solve that problem.
Which widely available and affordable CPUs aren't affected by Spectre?
Most of the ARM CPUs are not affected. For instance, the Raspberry Pi beards are not affected.
And Atom circa 2012(?). Well, OK not really easily available, but I suppose what you can find on eBay should be affordable...
OpenBSD has struggled at times to pay their power bills. They don't have the resources to branch off into building new hardware from first principles.
That's a matter of finding them funding then, and if there are people that I would think are capable of pulling something like that off it would be them. All of this assumes that they are willing of course, and it would be rude to assume they are without asking them.
Hopefully there are some tech savvy new crypto millionaires that are willing to help contribute to worthy projects such as this.
Yea, why don't you start?
Given the dismissiveness and tone of "hw designers are morons, we're the only smart people around here" from some people at OpenBSD, I don't think they know what it actually entails or how hard it is
Semiconductor fabrication is ridiculously expensive, OpenBSD people can't afford it unless some generous billionaire helps.
What would that provide? You'd have a processor that exists that would have one (and only one) operating system that can run on it and everyone else (except for some of the 'linux on everything' types) would ignore.

This is the sparc and sun or MIPS and Irix all over again... except this time without any viable business model to keep moving forward and progressing on hardware or software.

So what if they make their processor that is free from all known processor level bugs. Would that mean dropping support for all the other processors? or would they still need to be aware of the processor bugs that exist in other architectures that they'd still be expected to create patches for? Would they have the resources to fix processor bugs in their own design when they are discovered?

My read on this is that it doesn't make any sense for an operating system organization that lacks any financials to try to pursue custom hardware.

Is there not some sort of OS embargo consortium that could be created to resolve this?
they previously broke the KRACK embargo (among others): https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbh...

it's no surprise they're not high up on the list of important people to tell

No one seems to have told the other BSDs, either.
FreeBSD was told in December, which is still a realistic timeline.
Late December, i.e. a week or so before the cat actually got let out of the bag, and probably around a fortnight's notice until the originally planned coördinated announcement date.

* https://news.ycombinator.com/item?id=16074531

That is hardly enough time, considering that it is likely a holiday period for most of the people concerned, to prepare what needed to be prepared. Google and Intel gave themselves six months, in contrast, and they gave Ubuntu since November 2017.

I still haven't nailed down how much advanced notice Matthew Dillion, author of DragonFly BSD, was given; although his apparent level of preparedness with patches on 2018-01-05 can be attributed to the fact that he actually talked publicly about the problem in April 2017, months before Google Project Zero secretly reported it to processor manufacturers in June 2017 in the first place.

* https://news.ycombinator.com/item?id=16084641

> That is hardly enough time, considering that it is likely a holiday period for most of the people concerned, to prepare what needed to be prepared. Google and Intel gave themselves six months, in contrast, and they gave Ubuntu since November 2017.

Correct, but it’s significantly better than OpenBSD’s timeline (which learnt about it from the media). Mitigation for Meltdown is possible within of a week, or two. It requires working through the night, but it’s possible.

Mitigation after it’s in the media is a whole different situation.

No, it is not significantly better. Contrasted with the months that the likes of Microsoft, Ubuntu, and (going by what Paolo Bonzini has already said on Hacker News) RedHat got, FreeBSD and OpenBSD were in practical terms in the same boat. The difference of a week, when that week is Christmastide, is insignificant when put alongside those.
And the guy complains in the same paragraph that there are embargoes and that he wasn't in on the embargo... If he doesn't want embargoes he should be happy to discover the vulnerability with everyone else.
He told them they could go ahead, he then regretted it but that's not OpenBSD's fault.

From https://www.krackattacks.com/#openbsd :

> As a compromise, I allowed them to silently patch the vulnerability.

Receiving permission to patch is the opposite of breaking an embargo.

And it continues:

> As a compromise, I allowed them to silently patch the vulnerability. In hindsight this was a bad decision, since others might rediscover the vulnerability by inspecting their silent patch. To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.

The problem with embargoes is that they tend to run so long that users are put at risk unnecessarily. Most of the reasons for extending embargoes way beyond the time needed to prepare and test patches are just bullshit.

For example, from the conversation on the KRACK embargo:

>Q. What is the rationale on extending the embargo so long? To give vendors a chance to patch? It seems like once people in the know know about a vulnerability, the shorter an embargo time the better.

> A. There’s a couple of different interests at play.

> For instance:

> Researchers want to make a timed media splash.

> Security agencies want to evaluate the problem and make sure they get patched before anyone else.

> Vendors want time to prepare patches, yes, but that alone does not justify such a long delay.

> Reviewing the patch, testing it, and preparing it for commit and publishing erratas took only a couple of hours of my free time.

Apparently Meltdown was discovered back in June, or seven months ago. Only a select few were told about it and have been working on it. It is apparent that some are more equal than others in the "free and open source community," which ironically apparently includes Microsoft, Apple, and the three-letter agencies.

yes im sure security agencies spend as long as possible "evaluating the problem"
I think there is another important reason to extend embargoes aside preparing patches. Vulnerabilities like Meltdown and Spectre are in fact a whole new family of attack methods. It would take time to fully evaluate the extent of the problem. I would not be surprised that we learn new variants of Spectre in the coming future.
These classes of attacks are not new. Timing attack in crypto have been widely used for a long time. And even specifically in CPU's there was stuff like http://www.daemonology.net/papers/htt.pdf and http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190... over ten years ago(!).

To quote that last paper:

> We show how processor architecture features such as simultaneous multithreading, control speculation and shared caches can inadvertently accelerate such covert channels or enable new covert channels and side channels. We first illustrate the reality and severity of this problem by describing concrete attacks.

So not just were side-channel attacks on CPUs documented; specifically the dangers of speculative execution and caching on intel cpus (there: the Itanium) was documented.

It's inconceivable that this comes as a surprise to Intel. They're playing the public for fools.

Of course some of this is intrinsic: speculative loading, caching, and accurate timers are an intrinsic problem: an JIT running in the same process as the JITted code will not be able to keep secrets from that JITted code while all three are in play. But intel specifically also allowed this across memory protection boundaries, which isn't necessary.

Yes, I am aware of them and agree with you. In fact, cache-miss side channel attacks are known by academics for years as well. However, this is the first time someone combining various of previous techniques (timing, cache-miss, speculative loading etc.) together to make this attack work.

Quote from the meltdown paper:

> We expect that Meltdown and Spectre open a new field of research to investigate in what extent performance optimizations change the microarchitectural state, how this state can be translated into an architectural state, and how such attacks can be prevented.

So we won't see the last of Spectre.

> We expect that Meltdown and Spectre open a new field of research to investigate in what extent performance optimizations change the microarchitectural state, how this state can be translated into an architectural state, and how such attacks can be prevented.

Not to be cynical or downplay the research, but that's the kind of thing you (have to?) put into papers to make getting research funding easier in future. Hardly a new field, just very difficult to do research on.

Right, so to stick a label on it: the exploit is novel, the vulnerability is not.

Why does that matter? Because people seem to imply intel isn't to blame here; that since the attacks were unforeseen, it's essentially just bad luck. But that's not the case: the risky bits of architecture were known. It's like finding a memory corruption vulnerability and then deciding not to fix it since you don't also know of an exploit - that's just being careless.

Furthermore; finding holes like this isn't some divine act; although there is luck and skill involved there's no question that others can do so too. Even if an attack were completely new, you have to ask yourself how hard it was to find. Does a supplier have any responsibility to provide secure things? Then they should be looking for these kinds of holes. Obviously to a novice any hack like this looks almost magical - but there's a risk that by interpreting any novel attack as not-their-fault that the vulnerable supplier never bothers trying to make anything secure.

Intel was being careless. They should definitely have known this was exploitable even without knowing how to exploit it. And it's at least conceivable that they could have found this or an exploit like this if they had tried.

Why you single intel on this, ARM and POWER also affected

https://www.ibm.com/blogs/psirt/potential-impact-processors-...

Everyone is affected by Spectre, but my understanding is that Meltdown is a particularly powerful "version" of Spectre that only affects Intel CPUs ?
from what I understand AMD and ARM are also affected by the toy POC, quote

Out-of-order execution is an indispensable performance feature and present in a wide range of modern processors.

quote

However, for both ARM and AMD, the toy example as described in Section 3 works reliably, indicating that out-of-order execution generally occurs and instructions past illegal memory accesses are also performed

end of quote

it just need more work to make it work their

https://meltdownattack.com/meltdown.pdf

I saw some speculation that AMD might raise the kernel memory read exception earlier than Intel which prevents any subsequent instructions from using the value fetched from the kernel address space.
Having just read the Meltdown paper, but not the Spectre paper. My understanding is that yes Meltdown is much easier to exploit and more powerful. I believe Spectre requires the attacker to consistently fool the branch predictor while Meltdown enables reading kernel memory mapped into user space process via access in speculatively executed code and observing the affects on the CPU cache.
Spectre involves finding some code in the kernel that has access to the data, and getting it to access it speculatively before the conditional that tells it not to is evaluated.

Meltdown involves crafting some code yourself that speculatively accesses data it does not have access to before the CPU rejects the access due to permissions.

In other words, Meltdown involves your code accessing the data, but Spectre involves getting the kernel to do it for you. Spectre is a neat clever side-channel attack that is hard to protect against, while Meltdown is a blundering error in the way the CPU works.

My ELI5 attempt:

There are 3 vulnerabilities.

Meltdown is 1 of the 3. Meltdown is pretty much Intel only. Some ARM SoCs are also affected, but these are relatively rare. AMD64 is unaffected by Meltdown.

Spectre are the other 2 vulnerabilities. Spectre affects pretty much everyone.

Meltdown is more severe, and more of a blunder.

For a technical explanation, see [1]. Was recently referred to at HN as well.

[1] https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulne...

Do you have any source on why AMD64 is unaffected? The paper only mentions that they couldn't make their current approach work on AMD, but it doesn't rule out that it could be improved and made work
AMD's microarchitecture does not perform speculative loads that would cause a segfault, according to this AMD engineer:

https://lkml.org/lkml/2017/12/27/2

(comment deleted)
> We also tried to reproduce the Meltdown bug on several ARM and AMD CPUs. However, we did not manage to successfully leak kernel memory with the attack de- scribed in Section 5, neither on ARM nor on AMD. The reasons for this can be manifold. First of all, our im- plementation might simply be too slow and a more opti- mized version might succeed. For instance, a more shal- low out-of-order execution pipeline could tip the race condition towards against the data leakage. Similarly, if the processor lacks certain features, e.g., no re-order buffer, our current implementation might not be able to leak data. However, for both ARM and AMD, the toy example as described in Section 3 works reliably, indi- cating that out-of-order execution generally occurs and instructions past illegal memory accesses are also per- formed.

From the Meltdown paper[0] section 6.4 it would seem that out of order execution referencing illegal memory locations still occurs unless of course I'm misunderstanding something.

0: https://meltdownattack.com/meltdown.pdf

The section 3 toy example is demonstrating that OOO execution can cause instructions after a faulting instruction to be speculatively executed, but in that example the subsequent instructions do not depend on the result of the faulting instruction.

AMD's claim is that the result of the faulting instruction can never end up affecting a later speculated instruction - ie, for Intel the bad access says "here's the value you asked for" and later says "actually you don't have permission, forget you saw that". For AMD the bad access says "you don't have permission, so no result available"

Thanks for the explanation. So it's down to when the access check is performed. I wonder why Intel did it the way they did, performance maybe?
(comment deleted)
In terms of ARM, only some of the ARM CPUs are affected. The CPUs used in the Raspberry Pi range don't have out of order execution, so aren't affected.
Meanwhile I have seen nothing from QNX or Integrity...
I wonder to what extent QNX would be affected. It all depends on whether or not they map their micro kernel into the same address space as the application, for a micro kernel there is absolutely no reason to do that, the only things you might want to re-map are the message buffers and that can be done with some page table trickery.
QNX does map it's kernel onto high end of user VM space.

Due to how i386 TLB works (ie. no ASID) doing that is essentially required to get reasonable performance, micro kernel or not.

If the kernel has no sensitive state (believable for a microkernel), what matters is whether it maps the physical memory in its kernel space.
QNX 6.5 apparently maps first 256MB of physical memory into kernel space at fixed location. Also I believe that QNX's kernel memory contains region that maps to pages of user space memory of other processes (as part of the message passing mechanism, but I'm not exactly sure that I understand correctly how it actually works).
What about on ARM?
Has QNX been ported to ARM? Interesting!

edit: apparently it has been:

http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2F...

I would assume that majority of QNX deployments are on ARM these days and also ARM is the only platform supported by complete QNX "ecosystem" as some parts are not available for x86 (which is the only other currently supported platform).

Wikipedia states that supported platforms historically were i386, PowerPC, MIPS, SuperH and ARM. And I would not be too surprised if there was support for Altera Nios-II with MMU in the times when QNX was owned by Harman (As at least the touch screen part of some versions of Harman made Audi MMI is based on NiosII).

I'm using 6.5.0 SP1 in an ARM-based project. The documentation on their Screen graphics stack leaves something to be desired but otherwise it's a reasonably competent system.
Cleaner IPFS link: https://www.eternum.io/ipfs/QmQU1bCPsg7VY5puKqH6wZfAv1ZWBteK...

By the way, here's a short script that will download a (full) page with wget, add it to IPFS (if you have a running daemon) and copy the Eternum link to clipboard:

https://www.pastery.net/zxkzkc/

I love how that link just completely times out.
Works just fine for me? Which link?
I don’t understand the Intel hate. It’s not like their engineers are dumb or lazy. This exploit is very hard to imagine before now. And it’s there because chip makers were trying to wring out more performance. It’s unfortunate if anythig.
The OpenBSD developers are mad that Intel didn't inform them, or any of the other BSDs. Only Microsoft and Linux developers where included in the information that was under embargo. I don't think they're upset that Intel made a mistake, not more so that most others anyway.
Don‘t they refuse to honor embargoes? That would be an explanation why they got no information privately.
Here's an LWN article that references Theo's position on embargoes: https://lwn.net/Articles/601958/
This article seems to suggest that of all the BSDs, only OpenBSD / Theo takes this position - at least in in the case of the OpenSSL problems, the others got advance notice and participated in the embargo.

It might be argued that the risk of the exploit making it into the wild would have been higher if the BSDs were notified, on account of the sort of accidental premature disclosure that actually happened. This would appear to be self-serving if stated by the embargo insiders, but it may still be valid, especially if my guesses that a) this problem's biggest potential impact is in cloud computing, and b) there is relatively little use of BSDs in cloud computing, are accurate.

To me, one of the interesting bits there is the accidental timing of a comment.

From https://www.krackattacks.com

> We notified OpenBSD of the vulnerability on 15 July 2017, before CERT/CC was involved in the coordination. Quite quickly, Theo de Raadt replied and critiqued the tentative disclosure deadline: “In the open source world, if a person writes a diff and has to sit on it for a month, that is very discouraging”. Note that I wrote and included a suggested diff for OpenBSD already, and that at the time the tentative disclosure deadline was around the end of August. As a compromise, I allowed them to silently patch the vulnerability. In hindsight this was a bad decision, since others might rediscover the vulnerability by inspecting their silent patch. To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.

Note the date there that de Raadt was commenting on the discouragement of sitting on a fix for a month. What is the likelihood that he would be very discouraged to sit on it for six months? What if it was a three month embargo that changed to a six month embargo - when would the fix be released?

I would assume that those are questions that need to be asked prior to notifying a project.

They don't like embargoes but they don't go out of their way to break them. The only embargo I know of that OpenBSD broke was for OpenSSL, and that was an honest mistake, as explained here: https://www.tedunangst.com/flak/post/regarding-embargoes

There was another "incident" with the KRAK embargo, where OpenBSD got permission to silently patch it early and then the researcher who found it regretted giving them permission.

I think people put these two incidents together, combine it with the developers' attitudes towards embargoes and come out with: OpenBSD doesn't honour embargoes!

I guess if they made an explicit statement that they will comply with embargoes going forward, they'll be able to correct the record. But I don't see that that has happened. Lots of egos have to get out of the way, maybe?
They do, they break embargoes and that's why they are now paying the price for it.
This is why embargoes are stupid. Unless you're 'blessed' you get screwed. They only exist to give intangible benefits to large corporations at the expense of everyone else.

It's the technological version of insider trading

I think this is a valid opinion, but it its a bit circular.

'We don't respect embargoes' -> 'Company only releases to individuals who respect embargoes' -> repeat

From my perspective: I've been singing Intel's praises for the last decade as they regularly (for my own tests/usage) beat out AMD in performance, clock for clock if not dollar for dollar.

With this recent news, I feel like part of the reason that Intel has been doing so well is that they have been _cheating_.

Is it really cheating? What they are doing is well known and published, with one side effect that didn’t get rolled back (which seems more like a bug). At least that’s what I’ve understood from what I’ve read.
Implementing a cool optimization that happens to break your processor's security features is cheating, yes, even if you did it unintentionally, like accidentally taking a performance-enhancing drug.
Intent matters, at least to me. This does not seem to be a case like VW's diesel-emissions shenanigans.
The problem still is: one of the units responsible for increasing the processor performance was not working correctly. So the increased performance of the processor was obtained by not working correctly. That qualifies as cheating - if not by intent but then by nature of the design. It remains to be investigated how much of the security implications of their design choices was or could have been known to Intel. The fundamental attack concept was known for many years.
It is working correctly. A side channel is unfortunate, but in no way comparable to a missed page fault.
No, it just doesn’t. You’re diluting the meaning of that term to irrelevance. There are plenty of other words that adequately describe what happened - sloppy, careless, short-sighted. Cheating isn’t one of them.
You can easily cheat, without bad faith, by being sloppy or careless. You can also cheat in bad faith and call it being sloppy or careless. Breaking rules often has nothing to do with intent.
They didn't unintentionally break security features; they knowingly did so - the potential for information disclosure was widely documented. In their defense - the solution isn't trivial in general, and large parts of at least spectre apply in ways a CPU would be hard pressed to prevent (how is a CPU to know that even though memory isn't marked as protected from currently running code in actual fact that's because the JIT and the JIT-tted code are running in the same memory space?). But the intel CPUs are leaking a lot more data than reasonable.
Your parent was talking about Meltdown; that's the one that breaks security features, specifically: memory safety of a higher privilege level from a lower one.
In that case, probably no processors are “clean” as 100% elimination of all bugs is highly unlikely. Just like a bit of growth hormones slipped into your food supply without your knowledge.
Clock speed has been an awful metric for 2 decades now (except for within a single CPU model series).

All processors (including AMD/ARM) since the early 90's have been using instruction pipelining. It's a universal performance improvement, not Intel cheating.

Intel get drastically more instructions per clock cycle than anyone else, which is a very useful metric. Intel processors still dominate in single-threaded workflows. Meltdown makes that performance advantage look slightly suspect - Intel might be cutting corners to squeeze out extra IPC at the expense of security, reliability or accuracy.
Intel is cheating by validating the protection ring of speculative executions asynchronously, aka Meltdown, which makes this timing attack work in kernel space and requires the PTI mitigation. ARM does this on some cores too. AMD insists that they do not.
I don't think this is equivalent to the VW emissions cheating ... I guess we could find out otherwise later but if so it would amaze me that such a large and variable organization kept a secret for 20 years.
Well, there is the ME for one, an omnipotent, mandatory backdoor. Then there are _many_ undocumented opcodes. What does 0f0d00 do? What 0f78c0? What dbe0? I can continue for a very long time. These are just some known unknowns. If you still need more reasons, they are uncooperative when it comes to certain other firmware blobs.

If you are open to arguments, there are many good reasons to take a negative stance towards Intel.

Fun fact: I sent an overview of unknown instructions in compressed text form to a gmail address, and Google rejected it citing potentially malicious content.

To be fair gmail flags basically any code, if it's zipped i've found it only needs to vaguely resemble code. basically gmail is only safe for sending image formats and document formats that it knows of and don't contain macros, then it's just a crapshoot that you don't get matched a false positive. Gmail is far from developer friendly anymore.
> Gmail is far from developer friendly anymore.

Was it ever?

Well, not positively, but there was a time when it didn't block you from zipping up a little snippet of code and sending it to someone or saving it a draft when your in a hurry.
You can't even send encrypted 7zip archives through Gmail. Well, you can, but they'll block the download claiming it's not safe.
Spectre is somewhat reasonable; at least some variants are forgivable. Meltdown isn't. The risks due specifically to speculation and cache timing attacks (and lots of related stuff) were well known and documented for over a decade. They shouldn't have been speculating across memory protection boundaries, and they should have known that - even if it was unclear exactly how one might exploit such information leakage.
If it is so well known why has it taken 20 years to be discovered?
I don't know for sure but i'm guessing it's because this is all proprietary stuff, security researchers working on closed source hardware have to to a lot of inference... turn an already difficult theory into practice on an obscured implementation that is substantially harder to reverse engineer than compiled code.
It didn't take 20 years to be discovered; PoC's for variations on this theme were found over 10 years ago.

The details really matter for making an exploit: so the amount of speculation and what gets speculated, how caches work, how good timing is (and how large the difference between cache and memory is) etc.

Merely knowing that the combination of speculation, caching, and timing have the potential to break through memory protection barriers is a far from enough to actually exploit that weakness.

For comparison: it was widely known that sha1 had weaknesses, yet it took many years for somebody to construct two pdfs that actually demonstrate a hash collision.

I thought this exploit was imagined a decade ago? I could swear I have seen links posted here where concerns were raised as speculative execution entered the picture. But I agree that it’s hard to blame them when it took a decade for something to actually materialize.
Well they did release their Coffee Lake generation chip after they already new about the exploit.

Now i know it would take a lot for a major company to delay/cancel a release like that, but i think it could be argued that they were dishonest if they knew it would have a performance impact.

They would have had to delay it for a year or more.
So the decision was, go ahead and release it even though we know it's vulnerable because we'll make more money that way than we would if we took the time to fix it and make it secure?

If so, I hope the market punishes that decision mercilessly.

Well, Intel sales are going to drop off a cliff, and might not entirely be made up once a "fixed" version comes out.
What? No, that's not even remotely how it works. Not releasing Coffee Lake when it was ready would not have sped up the design of a new CPU which fixes Meltdown in any way. The only way that not shipping Coffee Lake would have helped anyone is by indirectly increasing sales of AMD processors.
To be fair, Coffee Lake is presumably faster/better and not more vulnerable than other chips [0]. If you have to mitigate anyway, and you're desperate to get back to pre-mitigation performance parity, releasing a faster chip that still requires mitigation could be a reasonable option.

[0] Caveat: Skylake and later are extra-vulnerable to Spectre because they have even more aggressive speculative execution, but fortunately, the IBRS microcode update is roughly equally performant as retpoline on Skylake+ only.

> This exploit is very hard to imagine before now.

It's really not. It's the sort of thing you wonder after first learning about out-of-order and speculative execution in a computer architecture class, but your professor assures you that implementors have been very careful to ensure any partial execution is properly flushed and rolled back. Then it turns out that, nope, no one's actually been keeping an eye on this after all.

...but if it is "so obvious" - why has it taken ~20 years to discover it (publically)?
Because like most inventions, its obvious after the fact. But getting there was time-intensive and wrought with many ways of failure.
It’s been discussed for years by people who didn’t need to see the exploit to know that it had to be there. Sane people have been avoiding timesharing for critical work on x86 since Core 2.

Life-safety work has no business in a public cloud.

This is a recurring pattern that just blows my mind: typically some small, often academic, group knows about some problem or potential problem that everybody else is ignoring until it becomes un-ignorable.

Anthopogenic climate change falls into this category.

I have to wonder, first, what sort of obvious stupid self-destructive things am I doing right now but ignoring. And second, how can we build systems that systematically take this into account? Is there some way to short-cut the process and find and fix problems in the early stages? Or better yet, design our systems so that they don't have the problems from the start?

The key thing to take into account here is that for every one of those early warnings that turn out to be true there's also a plethora of speculation that never amounts to anything.
It is not really comparable to climate change - people have been assuming that everything is OK, but there is no well-organized campaign to persuade people that these exploits don't exist now there is good evidence that they do.

There is one attitude that exacerbates those problems that are real - the tacit assumption that concerns do not count for anything until an exploit has been demonstrated.

> but your professor assures you that implementors have been very careful to ensure any partial execution is properly flushed and rolled back.

And, at the time, your professor was correct, to the extent of what he/she defined as "any partial execution".

But his/her definition of "partial execution" only considered state changes to the programmer visible CPU architecture state (i.e., the user level register set and the flags register). Their definition ignored the cache, because at the time the cache was considered simply a transparent optimization system that did not effect the values of contents of the CPU architectural state.

And in a way, that definition is still valid, even after the knowledge of these exploits. The CPU architectural state (registers/flags), even after executing exploit code, is exactly what it would have been had sequential execution happened. The exploit takes advantage of the fact that you can arrange the right set of code to run just the right way to leave a different state in the _cache_ (i.e., that thing believed to have been merely a transparent optimization).

It turns out now that the belief in the cache being transparent and only providing optimizations was the flaw in the belief system at the time.

> But his/her definition of "partial execution" only considered state changes to the programmer visible CPU architecture state (i.e., the user level register set and the flags register). Their definition ignored the cache, because at the time the cache was considered simply a transparent optimization system that did not effect the values of contents of the CPU architectural state.

You're putting words in other people's mouths here.

All other chipmakers are 'immune' to Meltdown precisely because the attack is obvious. Cache timing is hard, but speculating execution before checking permissions is plain dumb.

If it was such a mythical attack why doesn't it work on anyone else's chips? Intel screwed up hard on this and I have no sympathy. Hopefully the incoming lawsuits will make up for the massive amount of money wasted for the performance losses

The most recent ARM chips (cortex A75) are vulnerable.
Didn't know that. The chipmaking business is insular and pretty incestuous, I wonder if the same engineers built both speculative execution units.

I've got a friend in CPU design and he's only got about 50 companies he can work for in the world where he could do the same job he does now

Actually, what I heard through the grapevine is that it became vulnerable after an engineer retired.
There are only so many ways to accelerate a CPU by speculating.

I’ve written RTL that speculatively fetches data from memory in order to avoid bubbles in a pipeline. Not a CPU, but the concept is exactly the same.

If somebody had assigned me to a CPU project without guidance from a security architect and ask me to speculate reads, I’d probably have done the same as Intel.

The chance that the same guy did both CPUs is small. It’s just that it’s not an unreasonable way of doing thing if you’re not familiar with these kind of attack.

In the never ending quest for yet more performance out of memory systems that are ever slower than the CPU's, this is likely exactly how it happened. The speculative reads, because they were speculative, were likely viewed as harmless (because the instruction that generated the read would never be committed if the speculation missed anyway, so CPU state (registers/flags) would not be changed improperly).

And if anyone even considered the results of the reads, they likely saw them as nothing more than free cache pre-fetch instructions that would enhance performance should the speculative path turn out to be the correct path after-all.

And because the push was for yet more performance, free cache pre-fetch operations were likely viewed as a great bonus.

Sadly that doesn't fit OpenBSD's "Intel you suck" narrative.
You may want to read the following from the Meltdown paper.

> We also tried to reproduce the Meltdown bug on several ARM and AMD CPUs. However, we did not manage to successfully leak kernel memory with the attack described in Section 5, neither on ARM nor on AMD. The reasons for this can be manifold. First of all, our implementation might simply be too slow and a more optimized version might succeed. For instance, a more shallow out-of-order execution pipeline could tip the race condition towards against the data leakage. Similarly, if the processor lacks certain features, e.g., no re-order buffer, our current implementation might not be able to leak data. However, for both ARM and AMD, the toy example as described in Section 3 works reliably, indicating that out-of-order execution generally occurs and instructions past illegal memory accesses are also performed.

It's not just OpenBSD. For example, here's an interesting comment from Matt Dillon's commit to DragonflyBSD to mitigate Meltdown [1]:

> I should note that we kernel programmers have spent decades trying to reduce system call overheads, so to be sure, we are all pretty pissed off at Intel right now. Intel's press releases have also been HIGHLY DECEPTIVE. In particular, they are starting to talk up 'microcode updates', but those are mitigations for the Spectre bug, not for the Meltdown bug. Spectre is another bug, far more difficult to exploit than Meltdown, which leaks information from other processes or the kernel based on those other processes or kernel doing speculative reads and executions which are partially managed by the originating user process. Spectre does NOT involve a protection domain violation like Meltdown, so the Meltdown mitigation cannot mitigate Spectre.

> These bugs (both Meltdown and Spectre) really have to be fixed in the CPUs themselves. Meltdown is the 1000 pound gorilla. I won't be buying any new Intel chips that require the mitigation. I'm really pissed off at Intel.

[1] http://lists.dragonflybsd.org/pipermail/users/2018-January/3...

The meltdown paper explains the same side channel race condition vulnerability exists on ARM and AMD CPUs. Their toy project shows all three architectures have the same flaw.

However for unknown reasons it’s worse in Intel. They were not able to exploit the vulnerability to dump the kernel memory on AMD and ARM.

So I find the whole “Intel you suck” really out of line. I would not be surprised someone clever will come and figure out a Meltdown style attacks which works on AMD and ARM CPUs.

As Bruce Schneier says: “attacks always get better they never get worse”

Come on, calling this hate is childish and dismissive.
(comment deleted)
Hi I'm a mostly average Linux user just now learning about these hardware vulnerabilities as I'm planning on building a new computer. What new processor should I buy that has the best chance of being safe when all this dust clears? I first posted this question on a thread about Intel ME. Have processors always been this tricky security-wise or are these low level exploits we're finding a recent phenomena?
You could go with AMD which is seemingly immune to Meltdown, but apparently someone already found a security flaw with their PSP (aka their own Intel ME).
If you need security and it doesn't need to be fast, use a raspberry pi.

No ME, no PSP, no meltdown, and no spectre.

The firmware driving the VC4 on an RPi is non-free and closed, and the VC4 runs with greater privilege than the ARM core, with access to all the RAM. This firmware may not have security bugs like the ME and the PSP have had, and it may not contain a backdoor that we know of, but technically there's nothing stopping it from having one without anyone in the public knowing it.

Still, no Meltdown and no Spectre on any RPi

I've seen rumors that new bios/uefi versions have a toggle to disable PSP.

So AMD looks like a fair choice if this is true.

My opinion is that it doesn't matter which processor you buy because you will have the same important dependencies irregardless. Intel or AMD, you will depend on the expertise of the operating system designers, the chip manufacturer, and the trustworthiness of the code you run. The presence of disclosed vulnerabilities in Intel chips does not reduce the probability of undisclosed or undiscovered vulnerabilities in AMD chips. The number of possible vulnerabilities in AMD chips is a very large number. The number of possible vulnerabilities in Intel chips is a similarly large number.

Basically the choice between Intel and AMD in the context of in-chip vulnerabilities comes down to whether one feels better choosing a chip with known vulnerabilities + significant effort to mitigate or a chip without known vulnerabilities and without significant efforts at mitigation. In both cases, unknown vulnerabilities are probably equal.

To put it another way, building a computer is largely a consumer process not a technical one. The biggest security risk is software you download and run, not hardware flaws (e.g. rowhammer)

Wait a few months if you can.
More like two years.
I meant for the details to settle. But if you mean for non-vulnerable CPUs to come out, yeah...
Is it the right time to call out the fundamental wrongness of Intel(/Apple/Microsoft/etc) outrage narratives?

I mean I get it: large corporations (1) don't necessarily have my interests at heart; (2) are not able to perfectly execute (on extremely large and complicated) products and systems.

um. This is as radical as taking a stand that the sun sets in the west.

I like HN because of the promise that people think just a little bit before just typing something that strokes their feelz-good neurons. Yet, here we are on the front page with "In tel suxxx!" (and "Apple suxxx! in the htop thread, etc., etc. etc.)

sigh I guess it's inevitable.

People just don't like paying for intangible things, so the only way to pay for the internet is clickz/eyeballz. And OUTRGE! clickz/eyeballz are the cheapest and easiest.

I know, I know. There was a bug from vendor X that really irked/irks you and it feelz good to express your outrage.

I just wish there was just some corner of the internet where this wasn't the dominant trend and I had hoped it was here on HN. Well, at least reddit has extremely cute dog and kitten videos, and some chuckle-memes.

>I just wish there was just some corner of the internet where this wasn't the dominant trend and I had hoped it was here on HN

You're not going to find that, because taking the path of least intellectual and emotional resistance is just human nature. There is a positive feedback loop with outrage narratives that feeds people's ego and sense of in-group superiority, where one either feels empowered by agreement, or feels catharsis through disagreement.

The only thing HN has over Reddit in that regard is a culture that puts up the pretense ... sometimes.

> "...taking the path of least intellectual and emotional resistance is fundamental to human nature. There is a positive feedback loop..."

I agree that both of those appear to be fundamental elements of human psychology. Fortunately, we're also have the ability to counteract this, either individually via self-reflection, or more easily, through feedback from others. We can be aware of these tendencies and work to counteract them. If the community has a common goal to do so, to encourage the "better angels of our nature" as it were, we can do better together than we may find it easy to do on our own. And that doesn't have to be a pretense. It can be an explicit, earnest value.

Metafilter seems to be better about this stuff than we are here. At least, whenever I read it, the responses seem more thoughtful.
I agree, but Metafilter is extremely heavily moderated and there is a (small) cost to get privileges to comment/post. It also has a full time staff managing the community. I don't see either of those things changing on HN =/
Well, setting aside outrage, this looks like a primary source documenting a real problem for a marginal but sizable open source community (probably several--the *BSDs.)

It didn't really trigger my outrage button, and did inform me of something that had not actually crossed my mind--who got embargo information or not.

So, with a lot on HNers interested in open source, and using open source software, I see value in raising awareness that we should let these vendors know we value this. Some of us are making purchasing decisions at our companies and can steer our dollars toward good corporate citizens.

I find the outrage rather incidental to the post, regardless of even the email list author's intent in that regard.

While I think the story's title could be more mature, I think that the outrage in the article is justified. Intel didn't notify any of the BSD teams of the vulnerability during and after the embargo period, leaving anyone running a BSD vulnerable for a longer period after the disclosure.

IMO, it's a good thing that this issue gets publicity... if only so that the 'lesser' open source projects don't get ignored in the future.

I might be wrong but wasn't it BSD that a few months ago released information on a critical vulnerability before the embargo was lifted?

I think it was the WPA2 handshake vulnerability.

I remember people saying that BSD would probably no longer receive information about embargoed vulnerabilities because of that.

> I might be wrong but wasn't it BSD that a few months ago released information on a critical vulnerability before the embargo was lifted?

BSD is not an operating system unless your referring to the original Berkley Unix. Are talking about FreeBSD?, OpenBSD?, NetBSD?, DragonflyBSD? Remember they are all different operating systems.

The three big BSDs aren't exactly obscure, they might not have the highest server market share compared to windows and linux but they are the next on the list and often a primary choice for a variety of companies critical infrastructure, the most commonly sighted example these days is netflix.

I think the outrage is justified.

Don't mistake this as OpenBSD's concern for all BSD's. This is just OpenBSD rant because they isolated themselves by not respecting embargoes in the past. Some ARM CPU's are also vulnerable by the way.
That's an interesting point RE OpenBSD, especially if FreeBSD and NetBSD were included (which I do not know), I suspect this was just down to plain irresponsible incompetence though.

...I've no idea what your ARM comment has to do with it though. Meltdown is Intel specific.

A relevant question however might be who coordinated the work between june 2017 and now for Meltdown mitigations? Project zero at google discovered it, but did they hand over the responsibility to Intel or decide who else to inform themselves?

Has FreedBSD/NetBSD made a statement. I know DragonFly has a patch on master.

It's an odd world where Linux is in the "big three" and the BSD counterparts are more on the fringe. It's a far cry from the early world of the 2000s.

The language of the post is incredibly incendiary -- they sound no different than some of the extremists / terrorists talking about their enemies.

At the end of the day, we are all humans and the chip designers screwed up in a big way. But this issue is so complex and obscure that it's been unknown for 20+ years.

That's not their argument, their issue is intel didn't bother informing them so they had adequate time to work on patches (you know like they did for linux and windows kernel devs).
You have a lot of good points. I think it's just as fair to say that if Intel were to foster real conversation and not just fire up the PR monster there'd be less outrage.
I am also increasingly desperate for an internet place to read and discuss things in good faith. When HN started to go downhill I shifted to Twitter, but Twitter is like poison.

Perhaps we can create a new place?

Could you estimate for us when "HN started to go downhill"?
I suppose it's different for everyone, but for me the discussion quality started to fade around 2013.
> I mean I get it: large corporations (1) don't necessarily have my interests at heart; (2) are not able to perfectly execute (on extremely large and complicated) products and systems.

> um. This is as radical as taking a stand that the sun sets in the west.

If all you're picking up from this is the emotional component (aka "outrage"), I think you've missed the point. The larger problem now is that Intel (in this case) consciously chose to ignore serious risks for their customers and is currently doing their best PR job to downplay the impact of those risks now that they have been exposed.

The reality is that these bugs are potentially catastrophic. But maybe we are supposed to simply accept that that is part of the risk of living in this modern world.

When large corporations make serious mistakes that are predictable and that others publicly warned about, they deserve all the criticism that comes their way.

My question is why anyone who doesn't work for the PR department of one of these corporations feels the emotional need to defend them from justly deserved criticism? They didn't include all the responsible stakeholders in their mitigation efforts, that decision put the users of those OSes at risk, and so the developers of those OSes are pissed about it.

Why is that unacceptable to you? Why do you consider that to be "wrong"?

It's OpenBSD, what else to expect? This is the result of them not respecting embargoes. All they have left is to vent.
Is outrage not the way that almost anything moves forward these days? I feel that if not for the internet (the modern version where anyone has the ability to publish something that can be seen by many, regardless of tech knowledge) that we would be utterly powerless. It’s one of the few things that causes large organizations to actually care about anything. It’s almost an obligation these days.
Outrage can move things, but irrational outrage will move things in a mindless direction, which doesn’t help anything.

“Intel sucks” except arm cpus are affected too. “Intel sucks” except there’s a network of entities involved in disclosure. “Intel sucks” except OpenBSD previously opted out of this disclosure mechanism. I don’t see how anything will move in useful direction from this. Perhaps Intel will find a way to write statements with better feelz. That’s nice, but it’s not substantive and doesn’t deal with any of the underlying issues here.

Bleh. It’s the right time, because everyone’s thinking it.

To be fair, you look at those CPU utilization graphs, and the optimization bonus for this sacrifice of security is notable. The benefits are real, for these computation strategies. The reality is that yet again, we must sacrifice convenience in the name of security. This is why we can’t have nice things.

Ultimately, though, your pining for dignified silence and stoicism as the one true lowest common denominator is to be unrequited. Civilization among the civilized is not always a good thing. We all know that somewhere deep within the unknown recesses of some massive social planning department, circuits are being designed to lock us in, and make us pay for stepping out of line. Keep that fact in the front of your mind. Not as a goof. Not as a pacifier or placebo.

It’s all bullshit. It’s a valid gripe. It bears repeating.

People just don't like paying for intangible things, so the only way to pay for the internet is clickz/eyeballz. And OUTRGE! clickz/eyeballz are the cheapest and easiest.

The irony of course is that this was posted on a mailing list (so clicks are meaningless) by a developer of free software who has to spend his time working around these bugs that Intel caused. He has every right to be outraged and vent all he wants. He is actually doing the hard work while you watch kitten videos and chuckle-memes.

> Personally, I do find it....amusing? that public announcements were moved up after the issue was deduced from development discussions and commits to a different open source OS project. Aren't we all glad that this was under embargo and strongly believe in the future value of embargoes?

This is a bogus criticism. The issue was successfully embargoed for something like six months(!) before being published a week early.