Obviously it is a coincidence but I have a funny vision in my head of Amazon systems relying on Bezos running a script or similar for the past 27 years and forgetting to have someone else take it over for him. Could probably be an Onion article.
Other possible funny (but probably happened somewhere else) scenarios:
It doesn't rely on him running it, it just runs under his UID.
A widely distributed authentication/security process does an internal healthcheck by looking up 'jbezos' on the datastore. A few hours ago someone disabled the jbezos user.
nah a long time ago they moved everything to run as 'nobody' and that was before the move off of those systems and onto things like ECS.
fun fact jeff's employee id is like 72 or something while still being the lowest as they didn't reset the oracle sequence once they got that system working after testing but before making his user.
I know it's unlikely, but I've grown (even more) sceptical over the years.
I've stopped being surprised when I encounter some configuration file somewhere with a comment: "TODO: Fix this later" and it turns out more than half the business relies on that thing being configured that way.
Even at companies who do have the money and technical resources to do things right, things get forgotten, or pushed back on.
So, while these are problems that are unlikely, I'm sure I'm not the only one wondering... "Did they have something dumb like this, too?"
Ok, so the account still exists, but it's now in a different OU and so the healthcheck that looks for it in the Office of the CEO ou fails...
Again, a joke, but I've seen stuff fail in other dumb ways because of some assumptions that were made about things that never change (until the day they do).
I envision the bunker from Lost where every 30 mins a code must be typed in to keep things running. Now you know why Bezos dind't want to do it any longer.
Product pages on amazon.cn, their website for China, still appear to be working fine. Which demonstrates it's clearly using some separate infrastructure.
Wow, my kid was complaining that he couldn't access a page a couple of hours ago and I dismissed it as him making a mistake somehow. I guess this is must be a pretty huge outage for it to be across all the products!
It’s interesting seeing how their mobile app handles the problem. Instead of showing an error page, you're just mysteriously returned to the Amazon homepage when trying to view a product.
For a huge outage like this, the app’s behavior is a confusing experience, with users likely blaming themselves. The desktop web’s error message at least makes it pretty clear it’s an Amazon issue.
Is that interesting? Seems pretty standard and could even be handled relatively trivially in response to specific HTTP codes of any given individual request... Pretty sure nginx /whatever reverse proxy/load balancer they are using could do this natively without any special logic
You should try hitting one of the potentially non-profitable customer filters. Like buy 5 things with a gift card that all ship to a different address on the same day.
No message, no warning, no email, just your password no longer works and then an endless loop of resetting and verification emails.
This is also the behaviour of the mobile site on iOS. I don't know what the mobile app is like but it sounds like it could just be a webview to the mobile site.
amazon.co.jp seems to be working fine for me. I just accessed it through the iPad app and was able to view product pages and my purchase history with no problem. I’m located in Japan, in case that makes a difference.
"The online retailer’s net income shot up 219.8% to $8.107 billion, exceeding the company’s previous record of $7.222 billion set in the fourth quarter of 2020"
To be honest, I expected it to be larger. 4 million is an insanely big pile of money, but it's nothing that will show up in Amazons budget, even if we're talking about an hour or more of downtime.
"In 2020, online retail platform Amazon reported a net income of 21.33 billion U.S. dollars, up from a 11.6 billion U.S. dollar net income in the previous year. During the same fiscal period, the company's revenue amounted to more than 386 billion U.S. dollars."
Most of Amazon's net income comes from AWS, not their retail sales. This will mostly hurt gross sales, which will have a trickle down effect on net income, of course, but not at the rate you calculated.
Since sales aren't evenly divided hourly, this is misleading. I'd imagine sales are much higher during the 6PM to 12AM EST hours than say, 3AM to 9AM EST hours.
This time on a Sunday night could easily be 3 to 4x that much.
Maybe I'm in the minority but if I wanted to buy something on Amazon and it was down I'd just do something else and then come back later. They wouldn't actually lose my sale.
There was something stupid I tried to buy earlier before seeing this (and couldn't), and now I can't remember what it was. Perhaps a keto snack food I thought of?
For necessity purchases, yes. For impulse purchases, quite possibly no. One of the recommendations to save money is to never buy things the first time you think of buying them.
While Amazon is bad for impulse buying, it can also be a tool to reduce it: I'll put things in my cart and just let it sit there, then maybe once a week I'll complete an order. "Save for later" anything I'm still unsure of. I have a lot of things in my "Save for later" that could have been impulse purchase if not for that.
Also many people just default to looking on Amazon because of Prime and force of (conditioned) habit. In fact, that searches start on Amazon (despite its deliberately crippled search functionality) rather than Google is probably the single biggest heartache for Google as they lose that sweet, sweet time-of-purchase-intent opportunity to influence the purchase decision for advertisers.
If you break that conditioning and shoppers realize prices are actually cheaper on Target, Wal-mart or whatever, they will start routinely looking elsewhere. Now the outage didn’t last long enough to impact a huge proportion of their user base, but if it had it could durably drag on their growth.
I do this now. Mainly because Google shopping results ALWAYS have scam sites in them. My workflow was Google search, click shopping, see one store you've never heard of coming in $50 cheaper...Google store to see no real helpful info, or if you're lucky, one or two links to someone asking if it's legit and being told no. I wish they had better vetting.
More likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.
I d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on
I think peak Christmas time shopping was a couple million per hour back in 2006.
I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).
There was apparently always some kind of mix (AWS did start from internal requirements, though it quickly diverged). The OG services like S3 and DynamoDB power a lot of Amazon for sure. There’s been more bed of Amazon moving to AWS, most recently talk about shifting their main DBs.
I actually tried shopping on Amazon.com from AWS just to see if it was something local to me. Nope! I couldn't see any products or add them to my cart from AWS.
My understanding is that Fastly has a lot more edge locations than AWS, so using their image CDN results in lower latencies for many users than they could achieve with Cloudfront alone.
It's not edge location count that matters, but Cloudfront doesn't use BGP Anycast but rather does a more traditional DNS-based routing and tries to spread the requests across multiple edge locations (even those farther away) for redundancy, intentionally.
When I asked for detail about why they don't use Anycast, the Cloudfront engineering team basically said their customers care more about uptime than latency and that full Anycast was too sketchy. Apparently amazon.com disagrees, at least. I'm also happy getting much lower first page view latency out of Cloudflare.
Right but when they return how do you distinguish between a new impulse and a continuation of the old? This is one of those cases where marketers and businesses love to act like they know everything when in fact it's a huge info blind spot.
Not too surprising I think... for myself and I suspect many/most, the calculus is something like:
A) check back in half an hour when Amazon will probably be working again
or
B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up.
Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.
There's an element to slow shipping that has less effect today, since we're starting the week, it doesn't matter if things turn up tomorrow or Friday if it is for weekend DIY projects.
I wouldn't mind using another retailer today and near enough forget about it until the weekend. If by some chance it doesn't arrive until the following Tuesday, it doesn't matter, I'll find something else to do at the weekend quickly enough!
However, if you need it pronto, I'd rather pay for a courier.
Many people have grown up with Amazon, and in the early day it looked like they were undercutting retail from top to bottom, making the highstreet look like the bad guys. As their warehouses get exposed, and the issues of no union appear, I'm less likely to buy from Amazon as I think there are ethics involved that need consideration.
Do I really want to buy something knowing that a poor soul was probably operating at 98%, just below maximum threshold before collapse to get this item boxed?
Ok so, option C: philosophize for a while, then follow option B. For the vast majority of customers, the calculus during an Amazon outage still favors option A.
The one Amazon property that seems unaffected is their China site (https://amazon.cn). No doubt its running entirely separate infrastructure, unlike every other country's local Amazon site.
yeah, Amazon is past the stage where they had sysadmins. They now have armies of Operations People with SDEs being the fallback in case a runbook does not cover the current failure
This was meant as a friendly nod to the fact that FAANG outages usually lead to a lot of people checking HN and generating insane amounts of load, like the time a few weeks ago when YouTube/Google Auth was down and took HN right with it :)
But overall this site is extremely well run, so please don't take this as more than a joke.
I think it was this [0] event. Some people commented on HN being overloaded [1], including me [2] :-) It was not completely down (iirc), but there were a lot of "Sorry, we cannot serve your request right now".
Honestly, given the headlines of the past few weeks this is a fair joke, but in my experience they do not fire people for outages like this. Anything that can take down retail or AWS at large is considered a design problem, not a human error problem that deserves termination.
1. Even before the past few weeks of articles, anecdotes from people I know have always indicated that Amazon is a pretty brutal company so I am surprised by that.
2. Is it just a matter of delayed impact on you? Would it come up in reviews later?
Since it happened on a weekend, either someone did something stupid on a weekend, a routine operational task went wrong, something bumped into a limit, or there was a complex interaction of systems (all the best outages involve a complex interaction where it's actually hard to assign blame). I'd rally only be worried in the first case.
There must be a silly single source of failure, like a lambda giving the time of day or a marketing tracker pixel.
Yourkit, a java profiler, stopped working after a certain date in 2018 and had to be patched on thousands of users in my company alone. Could be some such time based constraint on a dependency loaded by all components for instance.
You can regionally partition but if the poison pill was inserted years ago and rolled out everywhere with a time trigger...
There are always bottlenecks and single points of failure no matter how resilient the design in. And even if there are multiple regions, the same bit of offending code could have been deployed to all of them.
It could be that all global systems happen to rely on some kind of single point of failure that the team never really realized before. Or if they did realize it they didn't realize the potential impact.
Or it was a code change that was able to be validated and rolled out to all regions but then some totally independent interaction caused it to fail afterward in a unique way.
I'm banking on surprising singe point of failure though. I really hope we get a detailed post mortem here.
Graph on downdetector.com shows AWS (likely) experiencing the same issue, which I guess is no surprise. So, a lot more than just amazon.com affected by this.
I think that is just downdetector users incorrectly reporting Amazon retail website failures against "Amazon Web Services". They should probably name it "AWS" to avoid that.
169 comments
[ 3.3 ms ] story [ 112 ms ] threadIt doesn't rely on him running it, it just runs under his UID.
A widely distributed authentication/security process does an internal healthcheck by looking up 'jbezos' on the datastore. A few hours ago someone disabled the jbezos user.
fun fact jeff's employee id is like 72 or something while still being the lowest as they didn't reset the oracle sequence once they got that system working after testing but before making his user.
I've stopped being surprised when I encounter some configuration file somewhere with a comment: "TODO: Fix this later" and it turns out more than half the business relies on that thing being configured that way.
Even at companies who do have the money and technical resources to do things right, things get forgotten, or pushed back on.
So, while these are problems that are unlikely, I'm sure I'm not the only one wondering... "Did they have something dumb like this, too?"
Again, a joke, but I've seen stuff fail in other dumb ways because of some assumptions that were made about things that never change (until the day they do).
Probably the closest I've seen.
https://www.google.com/amp/s/www.cnbc.com/amp/2018/07/19/ama...
For a huge outage like this, the app’s behavior is a confusing experience, with users likely blaming themselves. The desktop web’s error message at least makes it pretty clear it’s an Amazon issue.
This would be acceptable if it showed an error notification after redirecting you, but it doesn't.
It just very cheerfully dumps you to their homepage.
It's bad enough the mobile app is just the website bundled up in a less functional form, the least they could do is add small things like this to it.
No message, no warning, no email, just your password no longer works and then an endless loop of resetting and verification emails.
"The online retailer’s net income shot up 219.8% to $8.107 billion, exceeding the company’s previous record of $7.222 billion set in the fourth quarter of 2020"
8 Billion * 4 / 365 /24 = $3,652,968/hour.
"In 2020, online retail platform Amazon reported a net income of 21.33 billion U.S. dollars, up from a 11.6 billion U.S. dollar net income in the previous year. During the same fiscal period, the company's revenue amounted to more than 386 billion U.S. dollars."
The total number above does indeed include AWS, which should be subtracted since it's not offline, leaving us with ~$340B in 2020.
This time on a Sunday night could easily be 3 to 4x that much.
Maybe it will come back to me, maybe it won't.
While Amazon is bad for impulse buying, it can also be a tool to reduce it: I'll put things in my cart and just let it sit there, then maybe once a week I'll complete an order. "Save for later" anything I'm still unsure of. I have a lot of things in my "Save for later" that could have been impulse purchase if not for that.
I wonder how many others did something similar, or how many millions of dollars such an outage cost them.
If you break that conditioning and shoppers realize prices are actually cheaper on Target, Wal-mart or whatever, they will start routinely looking elsewhere. Now the outage didn’t last long enough to impact a huge proportion of their user base, but if it had it could durably drag on their growth.
Let's see whether this outage takes out HN with it once again ;-)
I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).
Amazon's internal, last i read, was years behind AWS's infrastructure and they're generally jealous of not having the same tools available.
I tried finding the article, but maybe someone else can help me remember?
https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
https://aws.amazon.com/blogs/aws/prime-day-2021-two-chart-to...
I can also run a WireGuard VPN with AWS as the endpoint. The second way works better on slow connections.
https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
When I asked for detail about why they don't use Anycast, the Cloudfront engineering team basically said their customers care more about uptime than latency and that full Anycast was too sketchy. Apparently amazon.com disagrees, at least. I'm also happy getting much lower first page view latency out of Cloudflare.
https://www.itv.com/news/2021-06-21/amazon-destroying-millio...
A) check back in half an hour when Amazon will probably be working again
or
B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up.
Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.
I wouldn't mind using another retailer today and near enough forget about it until the weekend. If by some chance it doesn't arrive until the following Tuesday, it doesn't matter, I'll find something else to do at the weekend quickly enough!
However, if you need it pronto, I'd rather pay for a courier.
Many people have grown up with Amazon, and in the early day it looked like they were undercutting retail from top to bottom, making the highstreet look like the bad guys. As their warehouses get exposed, and the issues of no union appear, I'm less likely to buy from Amazon as I think there are ethics involved that need consideration.
Do I really want to buy something knowing that a poor soul was probably operating at 98%, just below maximum threshold before collapse to get this item boxed?
Example: https://www.amazon.cn/any-random-string-works-here/dp/032199...
But overall this site is extremely well run, so please don't take this as more than a joke.
> YouTube/Google Auth was down and took HN right with it
Have I forgotten something obvious?
[0] https://news.ycombinator.com/item?id=25415989
[1] https://news.ycombinator.com/item?id=25416390
[2] https://news.ycombinator.com/item?id=25416144
2. Is it just a matter of delayed impact on you? Would it come up in reviews later?
Good luck fellow Amazonian, wherever you are.
Legitimately laughed out loud. A person jumped off of Amazon HQ because of the brutal pressure.
.au app works.
Yourkit, a java profiler, stopped working after a certain date in 2018 and had to be patched on thousands of users in my company alone. Could be some such time based constraint on a dependency loaded by all components for instance.
You can regionally partition but if the poison pill was inserted years ago and rolled out everywhere with a time trigger...
Or it was a code change that was able to be validated and rolled out to all regions but then some totally independent interaction caused it to fail afterward in a unique way.
I'm banking on surprising singe point of failure though. I really hope we get a detailed post mortem here.
Probably not; I don't know specifically for Amazon, but most places avoid weekend code changes.
Its global. Someone out there is updating their resume...