HN Explorer

Ask HN: Where is the AWS outage post-mortem?

69 points by herodoturtle ↗ HN
I've been checking in on this page [0] frequently since the recent us-east-1 outage.

The outage was significant and had many unexpected knock-on effects for us - despite most of our instances being in a totally separate region - so just wondering what happened.

Any idea how long these post-mortems typically take to be published?

Thanks.

[0] https://aws.amazon.com/premiumsupport/technology/pes/

41 comments

[ 2.7 ms ] story [ 90.7 ms ] thread
thecrumb
Takes awhile to get the creative fictional juices flowing.
cosmotic
I don't know; I thought "impairment of several network devices" was a pretty creative spin
nix23
Nothing happened you can go away now.
exabrial
Pro tip: Never mark your services down on the status page and you don't have to write an RCA!
MrWiffles
Just to chime in here and maybe help explain some of the more snarky responses...

If I were in your shoes, I wouldn't expect one, and if one ever does get published, be suspicious of it. VERY suspicious. Because it's likely missing critical information and/or some portion is likely fictitious to some degree. I mean, this is a company that can't even update its own system status dashboard truthfully, so don't expect any degree of honesty or accountability here.

beebmam
Frankly, why should a company NOT lie? There's no reason to tell the truth if the truth will hurt your business.
yunwal
I'm an infrastructure engineer that has never considered using a comparable cloud provider unless the solution I was looking for wasn't supported by AWS. It used to feel like AWS cared about it's reputation and trustworthiness, and now they clearly do not.

If I was starting a project from scratch now, I'd certainly explore other cloud providers, which is not something I would've said 3 years ago. Their blatant lying is definitely going to cost them in the long run.

ptcrash
When did AWS lie? On their status page? That's the last place to receive updates on service status. It sucks but it's also common knowledge. I'll give you a little insider knowledge: The SHD is updated manually. If it's any consolation, GCP does the exact same thing.

I saw notifications in the AWS Personal Health Dashboard pretty very early into the issue. Did you see something different?

Also, I've seen issues logging into the default console a lot, so I keep one of the backup endpoints bookmarked. Might come in handy for you too!

https://us-east-2.console.aws.amazon.com/

randombits0
If you do not design your cloud resources to be vendor-independent, that’s on you. You absolutely need the ability to cross vendors in short order, whatever stack you’re running.
yuppie_scum
This is easy for the Monday morning armchair QB to say, but the fact of the matter is that most SMBs are not in a position to run multi cloud. Legacy pre container pre kube workloads (tech debt) exist, as do persistent data stores that are too large to replicate between clouds without racking up massive transfer costs. Not even getting into vendor lock-in around networking or things like dynamo or serverless. “Well just move all your stuff to Kube then!” Ok fine, mr CTO, I’ll do it but I’ll need about 3 engineers at 175k and budget a couple mil more in Op Ex. the reality of the matter is that most of us will just gear for MTTR and hope that Amazon takes some lessons from the incident.
ptcrash
Having some authority on the subject because I've seen how both GCP and AWS handle their public status pages internally. They are manually updated by hand and are the last step in raising a large-scale issue.

Automated issue notifications get pushed to the Personal Health Dashboard for AWS and to my knowledge, GCP doesn't have automated status updates available to customer. (But AWS requires a support plan to gain access to their PHD, so it can be considered inaccessible to many as well).

systemvoltage
I wish HN was more professional. It used to be.
[dead] R0b0t1
Professionals recognize professionals lie.
dang
Please don't post unsubstantive and/or flamebait comments to HN. We're trying for something else here.

https://news.ycombinator.com/newsguidelines.html

R0b0t1
Strong disagree. The suggestion that they could be lying is what is under question in this comment chain. And, certainly, they could be, informed not only by general life experience but directly by the threads about Amazon management culture that pop up.

If professionals don't lie then why do doctors get popped for writing bad prescriptions? Why did scientists and executives lie about the harms of tobacco or lead? Etc.

What is being fought here is the suggestion that Amazon and its workers have some drive to tell the truth. Common sense dictates that is false.

Keep in mind an Amazon employee could have flagged my comment out of self interest.

dang
Sorry, but your GP comment was obviously unsubstantive flamebait. This isn't a borderline call. If you have some specific information to add to the discussion, by all means post it (in an informative way, not a provocative way). But definitely please don't post like the above. Even your reply here is nothing but rhetoric.

Edit: I just noticed https://news.ycombinator.com/item?id=29427506. Posting like that ("please consider offing as many CBP officials as possible") is very obviously a bannable offense on HN. I won't ban you for a week-old comment now, but if you do anything like that again, we will have to. Actually that comment is so shocking that I really had to think about this for a while. But you've also posted quite a lot of good comments in the past, so perhaps it will suffice to ask you to please review https://news.ycombinator.com/newsguidelines.html and stick to the rules from now on.

R0b0t1
I don't want to waste your time with this, but:

I can accept that perhaps I should have added more context, but if all I was responding to was "(they aren't lying because) they are professionals" it seems fine to say "professionals can lie."

In the broader discussion happening here it looks like people are mad at the mere suggestion that Amazon could be behaving in a way that is not in the best interests of their customers (and that the moderators are baseless coming to Amazon's defence). I'm open for waiting for more evidence, but the evidence we do have right now seems to justify most of the comments that are not obviously low effort.

You can ban me for whatever you want, but calling for the death of officials that violate rights is protected speech. Calling for the death of a specific individual is usually not. By retaining my comment you are not privy to any illegal behavior.

I've seen what bad cops can do. You can too if you look now. Voting doesn't seem to work. Can't vote when they've already killed you, anyway. Just in my personal experience we had some local cop that was ordering drugs online for who knows what -- could be planting them on innocent people -- and nothing came of it despite the reports filed.

I'd like to remain on the site as I draw great enjoyment from it. Guess I'll stay away from political topics.

1cvmask
Cat got stuck in the server room.
belter
Also waiting to know more details. In the meanwhile...You might want to check these:

"AWS uncertain about true cause of outage" https://news.ycombinator.com/item?id=29492120

"Thousand Eyes AWS Outage Analysis: December 7, 2021 " https://news.ycombinator.com/item?id=29483487

Edit: Using [0] linked above come up with the following statistics...

[0] https://aws.amazon.com/premiumsupport/technology/pes/

15 major events of last 10 years, including the one from 7 Dec.

Percentage distribution across months of the year:

----------------------------------------------

Jan: 0% Apr: 7% Jul: 7% Oct: 7%

Feb: 7% May: 0% Aug: 13% Nov: 13%

Mar: 0% Jun: 13% Sep: 13% Dec: 20%

-----------------------------------------------

Book your vacation for Jan, Mar, and May... :-)

tiahura
So it was either a DNS attack, a squawking NIC, or an ad system bug. That's very helpful.
belter
I made a public bet it was a code update to the Amazon EventBridge service. Waiting to know if I pocket the money...
zeckalpha
Why not all three contributing causes?
dandigangi
Probably within a month. Maybe a small post about it while they get things together more seriously.
[deleted]
(comment deleted)
Bluecobra
My guess is that the the person that broke Facebook for a day got hired at Amazon...
doubled112
What could go wrong? Nobody would ever make a mistake like that twice!

Right?

brycewray
In this case, probably around 4:55 PM Eastern Time on Friday, December 24.
cyberge99
I was thinking tomorrow (Friday, USA) at 4:55 PM.

That’s when the slimy politicians always release bad news.

cyounkins
Conveniently most of AWS' published postmortems do not have publish dates. Some are for the same day, some say "earlier this week".

Realistically, it takes time to fully dig into what happened. In a public postmortem you also want to describe your mitigation efforts, which you also need to think through fully.

I expect something in the next few days.

la6471
It takes 5 business days.
ptcrash
Let's try to remember the guidelines, be kind, and have curious conversation. Hacker News isn't the place for baseless speculation and generalization.

AWS publishes postmortems for major outages here: [1]

This outage was significant enough to expect a public PES but it's typical for it to take at least a few weeks for that page to get updated. At least, that's been the trend for all previous publications. If the PES is anything like previous PESes, it will have a detailed explanation of the root-cause and an explanation of what will change to prevent the issue from happening again but it will still be technologically abstract because cloud providers are very secretive of how they orchestrate resources behind the scenes. Enterprise-tier support customers can ask for RCA's but as I understand it, the account managers don't have any official wording internally yet either. But, it's only been 48 hours, so something of this significance will likely have a large chain of sign-offs it has to go through before official wording is announced.

This type of quietness from AWS before their official wording gets published is standard practice for them. They have a large legal team, PR team, and executive team that will all be interested in controlling the narrative but that's not uncommon for other large companies either.

If I were to take a stab in the dark, I'd say we'll see a PES in ~2 weeks. Maybe sooner, maybe later. If they don't announce a PES, I'll be really shocked because last year's Kinesis outage had arguably smaller impact but ended up getting a publication. [2]

[1] https://aws.amazon.com/premiumsupport/technology/pes/

[2] https://aws.amazon.com/message/11201/

Edit: Typos

herodoturtle
OP here. Just wanted to say thank you for this.
[dead] unbanned
What outage?

Status pages are static OK messages.

gukov
Public post mortems are usually done by growing companies that have something to lose. AWS is too big to “lose” anything.
stunt
It's not that simple. Incident responders and teams that own the pieces have to write the initial draft and review it. Perhaps two different copies. One with more details for internal and another for public. Multiple round of reviews, and then it goes to a separate team to proofread the public material. Meanwhile they also have to measure customer impact and sometimes even inform high tier customers about the incident and its impact.