If this is process node dependent and has started turning up relatively frequently in places like Facebook and Google, I wonder if it's more common in current-gen AMD CPUs than others... the timing would seem to fit, and they're on the smallest process node of anything currently in datacenter use.
I've seen this for years, it is not new. You have to be running at the scale of tens/hundreds of petabytes of data to experience really quirky anomalies.
The failure rate and failure modes vary from model to model and even from batch to batch. The biggest issue is not that failures exist, but that they're hard to trigger in testing and when they trigger they do not cause a crash. The CPUs that any actually used computation is running on have already been through a lot of testing (manufacturer, cloud provider), the failures have to slip through all of them.
And the failures can be really nasty: imagine some distributed DB server damaging large % of database metadata records because 1 instruction malfunctioned, but didn't crash the process. By the time you know something's wrong you cannot recover 70% of the data because you have no idea which data blocks correspond to which logical rows (remember, the metadata is garbled), except through a manual process that can take weeks or months. Or something like a v-table "miss" where instead of something like Table.Info() the CPU calls Table.Drop() because that function happens to be exactly 64 bytes lower in the v-table and has similar enough signature for the call to succeed. Those are two real examples.
You don't deal with this, other than fixing the damage and removing the bad machines.
The only way forward is expanded testing, that's what the paper from Google is (also) about. I think this issue will always be with us, to a larger or smaller extent.
There's probably a ton of data corruption out there that happens to be in places that doesn't really cause big problems.
Yes, but generally only when human life is on the line, such as aerospace. For just about everything else, the cost of redundancy in dollars and speed is too high. After all, problems are extremely rare, and when the extremely rare problems occur, it is extremely unlikely it matters. Versus a guaranteed cost of more than tripling your hardware, power, data center etc. costs. Maybe you do it in part of your stack, such as redundant mainframes or database nodes. But the most you will see on your appservers is ECC memory in an attempt to avoid cosmic ray bit flips (a similar problem we have been aware of for decades).
The scary part is detecting and reproducing the error reliably is difficult. Haven't read the paper, but I expect tooling on this field will be growing following more awareness.
It is. But being a statistical thing, it's rare for an individual small customer to hit. Probably more rare than running your own servers since they're cared for well.
Plausible. If it is for one of the new AMD instances as someone else in the comments theorized I don't think they're heavily used by Amazon internally.
It's a more prevalent problem at smaller nodes, but it's a general hyperscale problem. Modern datacenter CPUs are simply too complex to fully validate without increasing the cost so much that you're not competitive any more. Hyperscaling is very an exercise in thinking probabilistically. Everything fails without telling you, and that needs to be accounted for. It sort of approaches synthetic biology that way.
I highly doubt its only google or fb who deal with this.
It can even be a problem with a single machine depending on the environment. Like I was messing around with a simple circuit on a bread board. One of the ICs had a schimit trigger and just the EMI from being on a bread board consistently would cause my desktop computer near by to lock up.
There is all sort of spurious noise from cosmic to man made that has the potential to screw up a chips operation
Heck even a dip in power for a few micro seconds could cause issues, and most UPS use a mechanical relay that takes milliseconds to kick in ect... All though any decent PSU should have enough filtering capacitance to filter out such a short event
My rule of thumb is that digital system error rates are about 1 bit per 10^12 to 10^15, depending on the specifics.
That's the typical bit error rate quoted by manufacturers of network devices and hard drives. Physically, something like a PCIe bus is also a network link, and has similar bit error rates.
Of course, some of these are detectable, and some are correctable. But some are uncorrectable, and the worst ones are undetectable.
In my experience, digital signing or encryption for protocols helps noticeably reduce random crashes and weird unreproducible errors. Similarly, disk formats like ZFS, BTRFS, and ReFS that keep cryptographic hashes of the data also regularly report corruption when used at scale.
A rule that I've come up with is:
Security is synonymous with robustness.
Things that improve one almost always improve the other, and neither is really possible without the other. Cryptography is not just there to keep the hackers out, it also ensures that bit flips are caught before they crash your mission-critical application.
What's an example of an error that's undetectable? I'm a little confused about what that means. If an error produces no observable effect, it doesn't seem like it matters very much, unless I'm missing something.
I assume an undetectable error would be one that cannot be detected except by running the computation again and comparing the results, as opposed to a simple check like a checksum. Or maybe it’s e.g. an error that flips 2 bits, with a 1 bit parity check.
Suppose you protect a piece of data with a checksum. The data gets corrupted and the checksum no longer matches. That's a detectable error. But if you're unlucky, the checksum of the corrupted data matches the original checksum. That's an undetectable error.
They mean that single bit errors are detected in storage systems like ecc memory, or ZFS, and will raise a fault (I.e. be detected). Checksums tend not to follow data into the code. So bit flips in the cpu could do anything at runtime: wrong addition, choosing the wrong branch of an “if” block, etc.
There's certainly a large overlap. At the same time you can use insecure hashes for robustness, so the implication doesn't go both ways. That's why I would say security usually implies robustness.
Recently I had my desktop freeze, where it stopped accepting any inputs including ACPI poweroff.
This time I wondered what is it that causes freezes like this?
Is it RAM bitflips, cosmic rays or other things causing a CPU to malfunction, like here, or is there a more mundane explanation for most freezes?
Was this not already understood by the 1980s? At least in IBM? Hence why mainframes are so expensive and why almost every big bank still pays those megabucks. In an industry where a single error at just the wrong place could destroy a multi trillion dollar organization overnight, people are usually more perceptive of even the remotest possibilities.
The entire cloud movement is recapitulating the development of the mainframe: containers, io budgets, metered usage, etc. Our industry has the memory of a bright dog.
So because it's similar to something we had in the past it's supposed to be a bad thing? Or the fact that people did something else in between is supposed to be a bad thing? (Also, obviously many people in the industry know about mainframes, memory is not the issue)
That we let Intel turn ecc into a premium feature to support market segmentation for a generation, and completely ignored data integrity in our storage systems while disk capacities reached terabytes was definitely a bad thing, and that’s on us as an industry.
Reminds me of problem in the early days of Google:
> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1. When Jeff and Sanjay put all the missorted words together, they saw a pattern—the same sort of glitch in every word. Their machines’ memory chips had somehow been corrupted.
> Sanjay looked at Jeff. For months, Google had been experiencing an increasing number of hardware failures. The problem was that, as Google grew, its computing infrastructure also expanded. Computer hardware rarely failed, until you had enough of it—then it failed all the time. Wires wore down, hard drives fell apart, motherboards overheated. Many machines never worked in the first place; some would unaccountably grow slower. Strange environmental factors came into play. When a supernova explodes, the blast wave creates high-energy particles that scatter in every direction; scientists believe there is a minute chance that one of the errant particles, known as a cosmic ray, can hit a computer chip on Earth, flipping a 0 to a 1. The world’s most robust computer systems, at nasa, financial firms, and the like, used special hardware that could tolerate single bit-flips. But Google, which was still operating like a startup, bought cheaper computers that lacked that feature. The company had reached an inflection point. Its computing cluster had grown so big that even unlikely hardware failures were inevitable.
31 comments
[ 0.25 ms ] story [ 74.2 ms ] threadThe failure rate and failure modes vary from model to model and even from batch to batch. The biggest issue is not that failures exist, but that they're hard to trigger in testing and when they trigger they do not cause a crash. The CPUs that any actually used computation is running on have already been through a lot of testing (manufacturer, cloud provider), the failures have to slip through all of them.
And the failures can be really nasty: imagine some distributed DB server damaging large % of database metadata records because 1 instruction malfunctioned, but didn't crash the process. By the time you know something's wrong you cannot recover 70% of the data because you have no idea which data blocks correspond to which logical rows (remember, the metadata is garbled), except through a manual process that can take weeks or months. Or something like a v-table "miss" where instead of something like Table.Info() the CPU calls Table.Drop() because that function happens to be exactly 64 bytes lower in the v-table and has similar enough signature for the call to succeed. Those are two real examples.
The only way forward is expanded testing, that's what the paper from Google is (also) about. I think this issue will always be with us, to a larger or smaller extent.
There's probably a ton of data corruption out there that happens to be in places that doesn't really cause big problems.
The problem is that nobody knows how to write tests that catch all errors.
It can even be a problem with a single machine depending on the environment. Like I was messing around with a simple circuit on a bread board. One of the ICs had a schimit trigger and just the EMI from being on a bread board consistently would cause my desktop computer near by to lock up.
There is all sort of spurious noise from cosmic to man made that has the potential to screw up a chips operation
Heck even a dip in power for a few micro seconds could cause issues, and most UPS use a mechanical relay that takes milliseconds to kick in ect... All though any decent PSU should have enough filtering capacitance to filter out such a short event
That's the typical bit error rate quoted by manufacturers of network devices and hard drives. Physically, something like a PCIe bus is also a network link, and has similar bit error rates.
Of course, some of these are detectable, and some are correctable. But some are uncorrectable, and the worst ones are undetectable.
In my experience, digital signing or encryption for protocols helps noticeably reduce random crashes and weird unreproducible errors. Similarly, disk formats like ZFS, BTRFS, and ReFS that keep cryptographic hashes of the data also regularly report corruption when used at scale.
A rule that I've come up with is:
Security is synonymous with robustness.
Things that improve one almost always improve the other, and neither is really possible without the other. Cryptography is not just there to keep the hackers out, it also ensures that bit flips are caught before they crash your mission-critical application.
There's certainly a large overlap. At the same time you can use insecure hashes for robustness, so the implication doesn't go both ways. That's why I would say security usually implies robustness.
Is it RAM bitflips, cosmic rays or other things causing a CPU to malfunction, like here, or is there a more mundane explanation for most freezes?
> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1. When Jeff and Sanjay put all the missorted words together, they saw a pattern—the same sort of glitch in every word. Their machines’ memory chips had somehow been corrupted.
> Sanjay looked at Jeff. For months, Google had been experiencing an increasing number of hardware failures. The problem was that, as Google grew, its computing infrastructure also expanded. Computer hardware rarely failed, until you had enough of it—then it failed all the time. Wires wore down, hard drives fell apart, motherboards overheated. Many machines never worked in the first place; some would unaccountably grow slower. Strange environmental factors came into play. When a supernova explodes, the blast wave creates high-energy particles that scatter in every direction; scientists believe there is a minute chance that one of the errant particles, known as a cosmic ray, can hit a computer chip on Earth, flipping a 0 to a 1. The world’s most robust computer systems, at nasa, financial firms, and the like, used special hardware that could tolerate single bit-flips. But Google, which was still operating like a startup, bought cheaper computers that lacked that feature. The company had reached an inflection point. Its computing cluster had grown so big that even unlikely hardware failures were inevitable.
https://www.newyorker.com/magazine/2018/12/10/the-friendship...