HN Explorer

Famous outages along with deep postmortems?

24 points by fizwhiz ↗ HN
I just finished reading the Roblox outage postmortem (trending on HN: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/). I learned a lot reading about it and found myself thinking "surely there's other well-written postmortems that are a doozy". Does anyone know a good resource where I can find a compilation of such outages? If they're classified/tagged then even better :)

12 comments

[ 5.1 ms ] story [ 37.8 ms ] thread
gtsteve
If you'd like to go beyond technical postmortems, I've found the IAEA reports on radiological accidents etc to make very interesting technical reading. Abstractly there is a often something to be learned about having a good playbook to deal with specific events.

https://www.iaea.org/topics/accident-reports

gadders
I did start a site where I was logging what I could at outagereports.net, but I let it lapse when cpanel fees shot up. Dan Luu's site referenced below is a good source [1]. I think also there is a big list on github somewhere of all k8s-caused outages.

[1] https://github.com/danluu/post-mortems

tommydoesntknow
Microsoft was about to spend $500 million on a blitz ad campaign called Five Nines, for 99.999% uptime re NT 5, was it? 2002ish.

They crashed the microsoft.com cluster only days before, sending certain accepted metrics re: uptime from 99.999% to 97.312%.

The cluster crash was caused by some errant out-of-band JavaScript being published to a live MSCOM cluster. The postmortem was not in-depth, it was a cremation. Burn and hide the body.

I thought it odd that all those involved ended up at AWS shortly thereafter, including the executive whose head rolled right out of 1 Microsoft Way.

Those involved owe Dave Cutler an apology, with or without the conspiracy intact.