3 comments

[ 3.1 ms ] story [ 20.8 ms ] thread
As systems become more complex, so too do their failure modes. This paper documents over a half a dozen slow performance modes of SSDs with several different internal failure modes caused by the silicon, firmware, etc.

Contrast that with the modes they discovered for CPUs: no internal factors, only external factors such as power delivery or cooling capacity.

That's quite a testament to the construction of the most sophisticated manufactured goods on the planet.

Well, this is just because they were looking for a specific kind of fault (performance regressions); CPUs tend to fail in a different way.

(me: 10y+ Google SRE, 5y in ML and Borg SRE teams working very closely with hardware)

I've seen this first hand, across datacenter providers. The most common one is some sort of power supply failure that causes the CPUs to under-volt and throttle; the machine won't go down entirely, the clock will just go through the floor. It's common enough we set up alerting to watch for it specifically.