Cores that don't count = floating-point coprocessor cores? :-)
But seriously, though,
> I think fail-silent CEEs is weaker than the adversary Byzantine failure model.
Of course they're weaker than byzantine failures. There's time locality, and the failure in themselves are not particularly hard to detect if some other core checks the results (although that obviously doesn't happen after every single computation).
wait... consensus algorithms exist?
can they work for human brains?
experiment :
tell a bunch of people that if they form consensus about, say, a color, they all get $5 (or whatever). have them attempt to reach consensus only using the exact mechanisms of a consensus algorithm. (research what they are)
You mean if people perform a successful vote on something they don't care about, you'll give each $5? You don't think anyone could do it in under a minute?
The conclusion is an utter joke:
“Maybe this will lead to abondonment of complex deep-optimizing chipsets like Intel chipsets, and make simpler chipsets, like ARM chipsets, more popular for datacenter deployments.”
Why would one presume that the reason has anything to do with computation error rates, rather than something obvious and mundane like "Graviton instances are more profitable"?
What needs to happen is to bring ECC to all levels of a chips logic to solve this. ARM vs RISC-V vs x64 doesn't address the problem as nothing in them inherently solves the problem. Making adders that add with ECC will.
If all of the logic also operates on ECC with the data, chip yields will also be improved. Say an core of the chip only produces the correct result 99% of the time, currently you have to disable that core. With ECC logic, you can still use it, as it doesn't matter if it has an additional 1% chance of a bit flip, as all of your logic is now immune to single bitflips. For mission critical logic/applications, one can scale up the ECC so its immune to more bitflips before an error is introduced.
What leads us to believe that there is not already fault detection in execution units? We have really no idea what's going on at the gate level in CPUs.
i know Intel and AMD like to hide hardware features - but given the design cost and the non-negligible overhead..one would suppose that they would at least surface a counter in the documentation..and probably even...you know, market it as a feature.
I’m fairly sure you can’t do that. For example, suppose you have logic that uses some inputs to compute an output:
A, B ⇒ C
Add ECC bits to the inputs, and you want
Aa, Bb ⇒ Cc
Now, if you want this to detect errors made by the “⇒” part, you can’t do this as “drop the ECC bits, compute the result, compute the ECC bits of the result”.
So, how do you compute the ECC bits from only the Aa and Bb bits without having to compute the C part?
Depending on the ECC logic chosen, that might be doable for bit shifts, but for addition? For multiplication? For IEEE float square roots?
You would have to develop a ECC that worked similar to homomorphic encryption, where you can do computations on the chipertext (or in this case, the ECC) without knowing the plain text. For this application, plain text would effectively be the int/float you are doing math on.
Since its possible with crypto, I don’t think it’s an insurmountable problem to create an ECC code where
Ecc(a+b)=Ecc(a)+Ecc(b)
It will likely not be as bit efficient as ECC without that property. Checking the ECC would also add a bit of overhead, so at first you would want to only check on a store instruction where you have to wait to select the RAM page anyways.
“I don’t think it’s an insurmountable problem to create an ECC code where Ecc(a+b)=Ecc(a)+Ecc(b)”
It wouldn’t, but that ECC wouldn’t work for multiplication instructions, square roots, etc. Dropping floating point and multiplication instructions will correct that, at the cost of significant speed.
I also think any such ECC effectively would be a copy of computing a+b modulo some constant you can pick.
If so, we’re effectively back at “compute the result twice, trap if the two results aren’t identical”
Maybe we should run the code twice on different cpu architecures and only accept the results if they are identical. I’ve heard of high reliability environments doing this, but maybe there are cases for doing it in web/IT as well.
This would help catch a large variety of possible errors, including but not limited to cpu bit flips, cpu bugs, memory errors.
Maybe, but in most situations you're probably better off adding some more developer time to work on software bugs than you are doubling your computation cost.
Of course they're not the same thing. But they both make things go wrong. There are situations where hardware bugs are important, and there are situations where hardware bugs don't make a meaningful difference.
Triple modular redundancy - explored on the earliest computers, when these kinds of CPU errors were common (but for other reasons). It isn't commonly implemented today for the same reason that ECC RAM isn't in everything by default - people tend to prefer paying less and just crossing their fingers.
Detection is certainly the harder part - but the rest of it is pretty well-trodden ground. Some POWER9 users noticed that linux was under-reporting their core count and further investigation showed that the CPU's RAS had identified some cores that were acting a little funky - so it automatically dropped them into a guard partition and logged the action in the persistent circular buffer that nobody had bothered writing open source code to watch. Of course IBM had already written a ton of software that had it covered in AIX. I've noticed the same thing on a lot of other platforms, I'm looking at an HP MicroServer right now that has low level hardware error logging that never makes it past the bios firmware - and I've been meaning to fix that for about two years now. It might be a good idea to tackle that issue before we start getting too clever with solutions.
Dynamic reconfiguration: Basic building blocks for autonomic computing on IBM pSeries servers
> A deterministic AES mis-computation, which was “self-inverting”: encrypting and decrypting on the same core yielded the identity function, but decryption elsewhere yielded gibberish.
This is such a unlikely thing to happen that it is likely a fault in software that is used to validate this.
33 comments
[ 3.5 ms ] story [ 85.6 ms ] threadBut seriously, though,
> I think fail-silent CEEs is weaker than the adversary Byzantine failure model.
Of course they're weaker than byzantine failures. There's time locality, and the failure in themselves are not particularly hard to detect if some other core checks the results (although that obviously doesn't happen after every single computation).
experiment :
tell a bunch of people that if they form consensus about, say, a color, they all get $5 (or whatever). have them attempt to reach consensus only using the exact mechanisms of a consensus algorithm. (research what they are)
The article even says this is because of reduced feature size, not chip complexity.
If all of the logic also operates on ECC with the data, chip yields will also be improved. Say an core of the chip only produces the correct result 99% of the time, currently you have to disable that core. With ECC logic, you can still use it, as it doesn't matter if it has an additional 1% chance of a bit flip, as all of your logic is now immune to single bitflips. For mission critical logic/applications, one can scale up the ECC so its immune to more bitflips before an error is introduced.
So, how do you compute the ECC bits from only the Aa and Bb bits without having to compute the C part?
Depending on the ECC logic chosen, that might be doable for bit shifts, but for addition? For multiplication? For IEEE float square roots?
Since its possible with crypto, I don’t think it’s an insurmountable problem to create an ECC code where
Ecc(a+b)=Ecc(a)+Ecc(b)
It will likely not be as bit efficient as ECC without that property. Checking the ECC would also add a bit of overhead, so at first you would want to only check on a store instruction where you have to wait to select the RAM page anyways.
It wouldn’t, but that ECC wouldn’t work for multiplication instructions, square roots, etc. Dropping floating point and multiplication instructions will correct that, at the cost of significant speed.
I also think any such ECC effectively would be a copy of computing a+b modulo some constant you can pick.
If so, we’re effectively back at “compute the result twice, trap if the two results aren’t identical”
This would help catch a large variety of possible errors, including but not limited to cpu bit flips, cpu bugs, memory errors.
For 99% of cases it is fine, but when you run into that last 1% it hurts.
Rad-Tolerant Lockstep Quad Core 8051:
https://www.eetimes.com/rad-tolerant-lockstep-quad-core-8051...
Tolerating rads with cold-war mcu softcores. Glorious.
https://news.ycombinator.com/item?id=27378624
https://news.ycombinator.com/item?id=27408398
Dynamic reconfiguration: Basic building blocks for autonomic computing on IBM pSeries servers
https://sci-hub.se/10.1147/sj.421.0029
This is such a unlikely thing to happen that it is likely a fault in software that is used to validate this.