Is that a problem with AWS? I thought people accept paying a premium because they don't have to deal with these issues anymore. If bandwidth is relatively expensive less people are going to abuse it, etc...
You pay a premium because you expect things to work well.
When things don't work, AWS doesn't tell you a reason why it didn't work. Instead they teach you lessons about how to architect your applications for failure by saying:
The EC2 team also recommend that you architect for failure using the white paper linked below:
https://d0.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf
The EC2 has responded, and informed me that they do not provide analysis of individual issues in AWS infrastructure.
They referred to the AWS SLA that states that "AWS will use commercially reasonable efforts to make Amazon EC2 and Amazon EBS each available with a Monthly Uptime Percentage (defined below) of at least 99.95%".
The AWS SLA is available here:
https://aws.amazon.com/ec2/sla/
(The above is a part of the response I received when I attempted to ask AWS about sporadic reboots and outages on one of my instances.)
Another gem from AWS in response to a query that mentioned that every major DNS server around the world had picked up a change made to the A record of a domain except for Amazon DNS (even after 2 hours of the change):
Thanks for that information. It really help us understand the issue faced. As you know changing a A record for a domain can take time to replicate through all the Root DNS servers and then onto the none authoritative servers from there on. When you are editing your DNS Zone file it is highly related to the TTL settings for quicker updates on changes to them.
However it really isn't uncommon to see full DNS replication when making changes to an A record.
..
Also for even more control and perhaps faster DNS resolving times within AWS look into our Route53 service.
https://aws.amazon.com/route53/
I hope this has addressed your questions and please feel free to ask if there is anything else.
It's only partially true though. There are tons of teams that Amazon that do not use AWS. They use legacy internal data centers that do not have an ounce of AWS infrastructure. And for some reason it is nearly impossible for them to transition.
It may have been a while since you worked there. Very few teams are not using AWS (primarily EC2). For all of retail, marketplace, and digital, it was increasingly difficult to get legacy or bare metal hosts. The only teams that I interacted with still on legacy hosts were either operationally deficient or still running their own databases.
> The only teams that I interacted with still on legacy hosts were either operationally deficient or still running their own databases.
There are a ton of these teams. AFT (several hundred devs), for example, is practically on their 10th year of a 3 year mandate to get off of their monolithic Oracle database backends, a constraint which forces them to run in legacy datacenters. You'll find similar issues with plenty of Supply Chain, Operations, and Transportation teams as well. The primary data warehouse clusters (non-redshift) and management interfaces are also in legacy data centers, last I recall.
They were responding to the comment where "9AM EST" was explicitly used. I don't understand the value of your comment as it doesn't serve to clarify - I think most if not all readers would understand the reference.
Yeah, I saw that but I figured both people would see the response if I put it on the child comment since people often only look at comment replies and their children. Conceptually I was replying to both.
One of my responsibilities at Amazon is "call leader". It's a rotation of long-time Amazon folks who facilitate the recovery process whenever there are outage events; call leaders run the conference call (there's always a conference call) and help people focus on the right things in the moment as well as making sure we gather everything we'll need for a rigorous post mortem.
The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful.
Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.
In general, our conference calls are structured and laser focused on recovery; "what can we rollback, undo, failover or change to restore availability?" There's much less emphasis on figuring out the root cause, or the "why" of the issue, in that moment. That's another natural tendency - to want to understand the issue rather than fix it - that can be counterproductive.
In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software.
If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join.
For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.
If you know exactly where the problem lies, then I agree that in some cases a conference call might not help. However, more often than not, part of the challenge in large-impact outages is figuring out what's going on and where the problem actually lies.
In a microservices or SOA architecture, services can layer on top of other services, that call more services, that put messages in a queue for handling by yet another service, etc., belonging to different teams. Unwinding this to identify the problem is something that's much faster with a phone call. "Connecting the dots", especially across teams. A technical discussion within a team is likely to take place by chat.
I've participated in a number of tech conference calls over the years, and I've always been impressed with the professionalism: the discussion is calm and focused on identifying the problem and recovering. I think folks strive for the "cool radio voice", the Houston Center Voice like was described in the SR-71 Ultimate Ground Speed Check story: http://oppositelock.kinja.com/favorite-sr-71-story-107912704...
> Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the " Houston Center voice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the Houston controllers, that all other controllers since then wanted to sound like that, and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.
It's used to search their site for products. If you want to buy a "571B Banana Slicer" the easiest way to find it on amazon's site is to use their search.
The search feature being broken translates to sales being close to zero for that time period.
'For decades I have been trying to come up with an ideal way to slice a banana. "Use a knife!" they say. Well...my parole officer won't allow me to be around knives....'
Also, Amazon is established enough, and has few enough comparable competitors, that anyone whose search breaks is liable to just try again tomorrow.
Still, some people who do go somewhere else, or are less likely to try Amazon in the future. I'd be surprised if this wasn't at least a multi-million dollar mistake.
I never do. Going to their homepage is like seeing the dog's breakfast. It is a complete mess of whatever passed an a/b test of increasing sales. It is like they reached a local maxima but it is on St Crappy's Peak.
If I go to Amazon.com I just go to the search bar and refine from there. So I guess I use their "navigation" in the form of their groupings of features on the left side.
Unless there was deliberate action (skipping procedure in a way that a person has been warned and disciplined about previously) there will be no firing.
Any calculation based on the hours down needs to be reduced to account for people finding products by other means (already saved for later in their basket, direct links, searches via external search engines).
Firing seems like a toxic move, especially without knowing what happened. I'd like to think that Amazon has multiple layers of alerts built into place for situations like this, so this was probably a complex bug (perhaps a perfect storm of faulty configurations like what happened with Google's Cloud offering not too long ago).
I'd say that number is probably a ceiling on their losses, and that the actual number should be much lower. I'm super curious if non-tech news organizations will pick up this story, in which case the biggest hit could be from the PR. That being said, it doesn't seem like such a bad time to go down.
Not as bad as this, but my friend was an SDET contractor at amazon and bugs in his testing framework let a multimillion dollar bug to go to prod. Not only was he not fired, but he got a full-time offer at the end of his contract.
Fire someone who they effectively just spent multiple millions of dollars to train? I doubt it. Thats a huge learning experience there, and if anything, will make him more cautious in the future.
I guarantee they told your execs something completely different because Amazon. SOP is to tell you don't worry be happy while loading you up with blame coming down from your director/VP. If you quit, everything they tried to secretly pin on you now focuses on your managers so they are desperate to keep you from quitting. Once you get to OLR the next year you'll get an unpleasant surprise and probably fired. This is called burn and churn management and Amazon is a master of it.
Changing teams is pointless. Every manager of every team at Amazon has burner employees whether anyone knows it or not. Part of being an employee is trying to determine if you are the burner. People are hired in as burner employees at Amazon. That is unless Amazon stopped doing company-wide stack ranking, which obviously it will never do since it legitimizes pass-the-buck and is so beneficial to the people making the rules.
> That is unless Amazon stopped doing company-wide stack ranking, which obviously it will never do since it legitimizes pass-the-buck and is so beneficial to the people making the rules.
I've got a friend who's a manager at the Amazon Game Studio. That part of the company at least doesn't do stack ranking in the way that it was described in the NYT article. There may be other teams (or even most of the company) that do, though. I think that teams in different locations end up developing their own internal culture, to a certain degree.
I have no idea where I read it, but I think it was about Bezos and a situation just like this. The quote was something like "why would I fire the person that caused the outage? They just learned a very valuable lesson and got great on the job training, why would I fire them now?"
When I read it I remember thinking how true it was. That person would be very unlikely to commit the same error, and might even work to prevent others from making that error.
Also, when I was at Amazon, we were taught that "Person X broke the site" is the wrong way to think about things. There should be enough testing, redundancy, code review and so on to avoid issues given that humans are fallible.
So the system failed, not a particular person, and the solution is to dive deep into why the system failed, and improve it.
Obviously there are exceptions for willful extreme negligence or policy violations.
Definitely true -- if it's possible for a single person's change to break Amazon's search, then something is wrong with the system of testing, code review, deployment, etc. It's never just a single person's fault, even if they wrote the causative line of code.
Well with Amazon's cult-think, every single person is a "leader" yet also, no one is as important as the shitty middle manager above them. So someone will take the fall but it'll be spun into some type of reckoning consisting of: Ownership, Dive Deep, and Earn Trust [1]
As long as you can add to your cult karma, you can do anything at Amazon!
Just wanted to see if it was spun as some type of use of the leadership commandments. Because most things about people or process are spun to somehow derive off of the commandments, at least that was my experience working for them recently.
This is not correct about Amazon/AWS from what I've been told.
I have several friends that work at, or worked at AWS within Amazon. When something goes wrong, I hear it's a blame game. I hear that it's a witch hunt and the person that's responsible gets held accountable, even if they didn't cause it. Example: Employee is in charge of project, project hires subcontractors, subcontractors screw up, Employee is fired.
I used to work at AWS, and this is not at all typical. I was in meetings with various high-level directors and the mantra was always the same: Why did this happen? If the answer is "somebody made a mistake" we need to figure out how we can automate so nobody can make that mistake again.
Now, that's not to say there's not a lot of pressure. People are held accountable. But they are held accountable for fixing the underlying issue and making sure the same mistakes aren't made again. I've never heard of someone getting fired for making a mistake. People get fired when the same expensive mistake is made repeatedly by the same person. (And really, based on my experience, that might not even be enough. I think in order to get fired you probably would have to fail to understand the nature of the mistake when everyone else in the room understands.)
Who ever told you that was incorrect. Amazon's root cause analysis process focuses on understanding the failure and implementing corrective actions. A process with a human in the critical path is a broken process.
This is 100% correct. The average stay at AWS is less than a year. Each employee has color coded badges based on length of stay. You can also use the LDAP tools to fetch each employees years of employment. The average is pretty sad and spun as "we are hiring more than ever. our oldest employees left when their stock became worth a ton"
But of course the real reason is that Amazon's work environment is basically chock full of shitty middle-managers, many who are adopted from Microsoft or salvaged somehow from better tech companies in the valley.
They even have these leadership principles that are basically a bunch of no-brainers that they force down the employees throats at every opportunity. They use them as a criteria for performance reviews, even before actual hard metrics. Shitty middle managers will keep stressing the leadership principles because (1) that's what they are told to do (2) they need to rate their subordinates on them (3) because these principles are qualitative, the only insurance they have is that they spewed them enough to get through.
Amazon, to sum it up, is just, shitty employees from other tech companies. And dumb/average intelligence follow-the-pack type drones - many who can hardly speak English and are on work visas..and they put many of these terrible speakers as managers!! (Amazon loves cheap labor, and will use it for ANY position possible)
When your leadership principles are sold as prolific and something every employee "should study" (Bezos words') you really can't expect to hire intelligent folk who don't want no-brainer basic business and character traits hamfisted to them as if it is the new work of Plato himself.
At Google, the answer is a firm no. Someone else commented on the "we just spent a million dollars training them, why would we fire them now?" quote which is commonly thrown around.
People get fired for leaks and being intentionally malicious, not mistakes. Even expensive ones. [besides, a big outage is usually caused by multiple smaller issues if your systems are sufficiently reliable. Also, read the SRE book about "blameless postmortems"]
I was watching this talk on Google's SRE and the speaker explains in depth why the answer at Google is indeed no https://www.youtube.com/watch?v=H4vMcD7zKM0 She explains the emphasis is more on learning from the problem and post-mortem than pointing fingers.
Does something like this actually cost them a huge amount in sales or do people know enough about the internet today to just come back in a few hours and try again? I know that personally, if I'm buying from Amazon, I would just come back later and try again.
I think it comes down to how you define "a huge amount in sales". Let's optimistically say they lose 1% of sales over a 4 hour period although it could certainly be much higher. Is 1% of Amazon's sales a huge amount for them? Not in the short term. Is that absolute amount a huge amount for me, personally, or most of the other companies in the world? It sure is.
In addition, many search-initiated purchases will be postponed, not abandoned. I'm sure they lose a ton of sales during this period, but I doubt it's as high as 90%.
I seem to recall reading an article or post previously about how they don't lose nearly as much as most people expect for this very reason. They would see a dip in sales when it was down, but it was more or less made up with a slight sales spike when it came back up. If I can find the article I'll be sure to share.
I have a friend that worked in Amazon for a while. He told me that after outages are repaired, there is a spike in sales but that it is never enough to cover the loses. The revenue does just disappear.
could be even worse. On the Internet it's not like some corner store you can check later, as it's still just as convenient next time: everything is a Google search away. So everyone who goes through the trouble of figuring out where else they can buy from - might just end up as those other people's customers for good!
I'd be curious to see the numbers on that sort of thing. I know for some people that is the case, but i almost feel.. locked in to Amazon. With free 2 day shipping, and decent customer service if i have problems/etc, i am not likely to shop elsewhere.
Is it a strange indicator that this feels more like lockin to me, than loyalty? I have no idea what that means.. or how to change the conceptual impression.
> Is it a strange indicator that this feels more like lockin to me, than loyalty?
My hunch would be that your subconcious is telling you loyalty towards a huge faceless corporation feels weird. If it was a local store with people you could connect your positive experience to, my guess is you'd have no issue calling it loyalty, even if that store was part of an equally big corporation.
That may be an accurate hunch. I feel loyalty towards Costco, because i hear good things about their employees and know multiple people who want to work there.
With that said, it's hard to say if that's because i hear good things, or because it's a physical store with people i see.
Compare that to Amazon, and i rarely hear good things about their staff treatment. Despite having a good experience with Amazon. .. to be clear, i just don't hear good things.. not implying that i hear lots of bad.
I'm up to about 65/35 loyalty/lock-in. It's almost like I keep expecting them as a faceless BigCo to fuck something up and piss me off ... but they never do. The very few minor issues I've had with orders they've resolved beyond my expectations. It's .. hard to stay mad.
This is in stark contrast to many other faceless BigCo's that I've stopped using entirely, or only use out of necessity but loathe every interaction with. (eBay, PayPal, Comcast, Verizon, etc.)
Having just recently bought from a couple other places online for the first time in a while, I'm not worried about Amazon losing customers long term. I had forgotten what it was like to have your package take a week and a half to get to you. If I was another e-commerce store, I'd be very, very worried about competing with Amazon's logistics machine.
How many people actually do this though? I rather wait for them to be online instead of registering my card information someplace else. I can buy stuff from almost any site as long as they accept Paypal but many want to store the card information themselves, understandable from their perspective but not from mine.
I wonder if this says a lot about immediacy and consumerism. Do people go elsewhere or does time permit them to realize they don't really need/want what they were about to buy?
Impulse purchases at the click of a mouse have really changed the way I personally consume. Anyone aware of any studies on the impact of instant feedback?
An reason I used to use to stop that behavior in myself was "oh, it will take a few days to get here". It's difficult to use that logic these days. If one day Amazon could get it to me in an hour without extra fees I'm in trouble.
It's too bad they haven't expanded faster - Walmart just barely rolled out their curb side pickup service where I live - if Amazon had made it here with same day delivery even just a few months ago we'd be customers for life, but as it is we'll probably stick with Walmart for groceries even when Amazon does eventually get here. (better the devil you know, etc.)
I find that interesting because Amazon is a "no impulse buy zone" for me. Everything I buy there is meticulously planned. It is probably an effect of living in Hawaiʻi - shipping can kill a good deal very quickly (and Amazon, while I'm at it, you know where I live - stop telling me that the item is free to ship when it isn't!).
Amazon doesn't deliver most physical things to my (current) home, so all of my purchases are Kindle (impulse) buys. I'm guessing that for a lot of international readers the Kindle Store is the most important part of Amazon.
I'm working in Antarctica and everything takes at minimum 3-4 weeks to arrive (often 6) after ordering, so it's really taken the starch out of impulse buys.
So, just for one point on a graph... I didn't really need the Star Trek salt and pepper shakers and Star Trek ice cube trays that I bought last night. If Amazon search had been down last night instead of this morning, it's a safe bet I wouldn't have bought them.
This is a big one, a lot of Amazon is designed around making it really easy to make impulsive purchases. One-click purchases? One-day/same-day delivery? The myriad of suggested/related items on every page between search and checkout? I can see where there'd be a big drop of orders that would never be placed again.
If there is no spike of sales, all hours of sales is worth the same, and assuming Amazon makes 100B per year in online sales, then it seems like a considerable loss, but still a drop in the bucket when compared to their total revenue:
I don't think anyone suggested that 4 hours would make up a sizable part of yearly revenue. That's not the point.
As another poster mentioned, even with the post-outage spike in sales from folks coming back to get what they wanted an hour ago, they lose those impulse sales forever.
but if that 100B annual sales includes periodic down-time, and this down-time is within the normal annual amount, then you are double counting down-times...
But some won't. They'll either be buying on a whim (and the mood passes), or buy from a competitor. Those are lost sales that aren't coming back.
There are people around who never buy on a whim, but there are also a lot of people that buy stuff because the thought crosses their mind, forget about it, then remember when it turns up several days later. These aren't always things you "need" strictly speaking.
As to competition I'd name Jet.com, Target, and Walmart as being major ones. I buy a LOT of stuff from Target these days using the Red Card (effectively 5% pays for the tax, and free delivery on all purchases).
Sort of. If the brand is good (e.g. Amazon), people will come back later to buy.
On the other hand, you can model some buying as periodic, not need-based. I go to the grocery store when I need food, but I go to J Crew once every six months. If I'm traveling this June, I'll delay my trip until July, so while the revenue isn't "lost", it's pushed into the future. My next trip is then in January, not December, and at this point, J Crew has lost one month of my sales forever.
There's a similar effect for many items at Amazon. I only spend so much time shopping, so if I can't do it now, there's some opportunity they miss out on in the future. The aggregate ends up looking like revenue that's lost forever.
I was doing some late night search for a purchase in Amazon and saw the problem. Tried couple times and gave up. Now I lost the mood to continue. Yeah, it does cost loss of sales.
I noticed this while I was up very late working but it only was an issue for some items. For instance when I searched for Item1 and Item2 it failed but Item3 just kept working. Not sure how they have search divided up but found that interesting.
Would love to see something come out about how it affected their users.
Yeah maybe. They would have to be server-side as I was picking random things I have never searched before to see if search was broken for everything or not (I was initially getting different error messages for different products).
Could be due to HTTP/2 switch - e.g in the morning it was working in HTTPS and now it's HTTP only again. BTW, it's interesting how much bloat there is in their HTML code.
It's kind of interesting how being the purchasing agent for an Amazon seller account gives you visibility on Amazon's health. After getting my 9am sales report, I panicked and had our tech team check if our seller software was working properly. Conversely, I knew Bezos was laughing all the way to the bank when people ragged on Prime Day last year before the official numbers came out. We didn't even have a featured deal and could see the insane difference.
163 comments
[ 3.8 ms ] story [ 194 ms ] threadAnd don't forget the CPU throttling, noisy neighbors, IOPS and what not.
For people who use it as a VPS provider, they are going to be hard hit by other tenants who are applying the intended purpose.
When things don't work, AWS doesn't tell you a reason why it didn't work. Instead they teach you lessons about how to architect your applications for failure by saying:
See my earlier comment as well: https://news.ycombinator.com/item?id=11822298Solution proposed by AWS: Use Route53.
There are a ton of these teams. AFT (several hundred devs), for example, is practically on their 10th year of a 3 year mandate to get off of their monolithic Oracle database backends, a constraint which forces them to run in legacy datacenters. You'll find similar issues with plenty of Supply Chain, Operations, and Transportation teams as well. The primary data warehouse clusters (non-redshift) and management interfaces are also in legacy data centers, last I recall.
I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.
There are still areas of Canada (Southhampton Island) and Central America (Panama IIRC) that are on EST year round.
The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful.
Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.
In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software.
If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join.
For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.
In a microservices or SOA architecture, services can layer on top of other services, that call more services, that put messages in a queue for handling by yet another service, etc., belonging to different teams. Unwinding this to identify the problem is something that's much faster with a phone call. "Connecting the dots", especially across teams. A technical discussion within a team is likely to take place by chat.
I've participated in a number of tech conference calls over the years, and I've always been impressed with the professionalism: the discussion is calm and focused on identifying the problem and recovering. I think folks strive for the "cool radio voice", the Houston Center Voice like was described in the SR-71 Ultimate Ground Speed Check story: http://oppositelock.kinja.com/favorite-sr-71-story-107912704...
> Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the " Houston Center voice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the Houston controllers, that all other controllers since then wanted to sound like that, and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.
(Actually what is Amazon search used for?)
The search feature being broken translates to sales being close to zero for that time period.
Wow, turns out that's actually a thing.
You made my day! :-)
tl;dr, Amazon's product search engine actually is handled mainly at A9. They seem to attract decent talent.
Godspeed, Amazon site reliability team...
Although this was longer and more severe.
Last quarter amazon earned $20.58bn from product sales.
So 4 hours of that is $38,000,000.
Still, some people who do go somewhere else, or are less likely to try Amazon in the future. I'd be surprised if this wasn't at least a multi-million dollar mistake.
If I go to Amazon.com I just go to the search bar and refine from there. So I guess I use their "navigation" in the form of their groupings of features on the left side.
Any calculation based on the hours down needs to be reduced to account for people finding products by other means (already saved for later in their basket, direct links, searches via external search engines).
I'd say that number is probably a ceiling on their losses, and that the actual number should be much lower. I'm super curious if non-tech news organizations will pick up this story, in which case the biggest hit could be from the PR. That being said, it doesn't seem like such a bad time to go down.
Mistakes happen. Sometimes they are really expensive.
I've got a friend who's a manager at the Amazon Game Studio. That part of the company at least doesn't do stack ranking in the way that it was described in the NYT article. There may be other teams (or even most of the company) that do, though. I think that teams in different locations end up developing their own internal culture, to a certain degree.
When I read it I remember thinking how true it was. That person would be very unlikely to commit the same error, and might even work to prevent others from making that error.
So the system failed, not a particular person, and the solution is to dive deep into why the system failed, and improve it.
Obviously there are exceptions for willful extreme negligence or policy violations.
As long as you can add to your cult karma, you can do anything at Amazon!
https://www.amazon.jobs/principles [1]
[1] https://www.amazon.jobs/principles
I have several friends that work at, or worked at AWS within Amazon. When something goes wrong, I hear it's a blame game. I hear that it's a witch hunt and the person that's responsible gets held accountable, even if they didn't cause it. Example: Employee is in charge of project, project hires subcontractors, subcontractors screw up, Employee is fired.
Sounds unfair. Sounds unreasonable. Happened.
Now, that's not to say there's not a lot of pressure. People are held accountable. But they are held accountable for fixing the underlying issue and making sure the same mistakes aren't made again. I've never heard of someone getting fired for making a mistake. People get fired when the same expensive mistake is made repeatedly by the same person. (And really, based on my experience, that might not even be enough. I think in order to get fired you probably would have to fail to understand the nature of the mistake when everyone else in the room understands.)
But of course the real reason is that Amazon's work environment is basically chock full of shitty middle-managers, many who are adopted from Microsoft or salvaged somehow from better tech companies in the valley.
They even have these leadership principles that are basically a bunch of no-brainers that they force down the employees throats at every opportunity. They use them as a criteria for performance reviews, even before actual hard metrics. Shitty middle managers will keep stressing the leadership principles because (1) that's what they are told to do (2) they need to rate their subordinates on them (3) because these principles are qualitative, the only insurance they have is that they spewed them enough to get through.
Amazon, to sum it up, is just, shitty employees from other tech companies. And dumb/average intelligence follow-the-pack type drones - many who can hardly speak English and are on work visas..and they put many of these terrible speakers as managers!! (Amazon loves cheap labor, and will use it for ANY position possible)
When your leadership principles are sold as prolific and something every employee "should study" (Bezos words') you really can't expect to hire intelligent folk who don't want no-brainer basic business and character traits hamfisted to them as if it is the new work of Plato himself.
[1] https://www.amazon.jobs/principles
People get fired for leaks and being intentionally malicious, not mistakes. Even expensive ones. [besides, a big outage is usually caused by multiple smaller issues if your systems are sufficiently reliable. Also, read the SRE book about "blameless postmortems"]
[1] https://articles.uie.com/three_hund_million_button/
Is it a strange indicator that this feels more like lockin to me, than loyalty? I have no idea what that means.. or how to change the conceptual impression.
My hunch would be that your subconcious is telling you loyalty towards a huge faceless corporation feels weird. If it was a local store with people you could connect your positive experience to, my guess is you'd have no issue calling it loyalty, even if that store was part of an equally big corporation.
With that said, it's hard to say if that's because i hear good things, or because it's a physical store with people i see.
Compare that to Amazon, and i rarely hear good things about their staff treatment. Despite having a good experience with Amazon. .. to be clear, i just don't hear good things.. not implying that i hear lots of bad.
This is in stark contrast to many other faceless BigCo's that I've stopped using entirely, or only use out of necessity but loathe every interaction with. (eBay, PayPal, Comcast, Verizon, etc.)
Impulse purchases at the click of a mouse have really changed the way I personally consume. Anyone aware of any studies on the impact of instant feedback?
It's too bad they haven't expanded faster - Walmart just barely rolled out their curb side pickup service where I live - if Amazon had made it here with same day delivery even just a few months ago we'd be customers for life, but as it is we'll probably stick with Walmart for groceries even when Amazon does eventually get here. (better the devil you know, etc.)
I'm sure if you totaled up my Amazon purchases and my Steam library it's cost me thousands.
It just happened to me, I was trying to search a book and want to buy it, but since I can't search, I didn't. And now I don't want to buy it anymore
As another poster mentioned, even with the post-outage spike in sales from folks coming back to get what they wanted an hour ago, they lose those impulse sales forever.
But some won't. They'll either be buying on a whim (and the mood passes), or buy from a competitor. Those are lost sales that aren't coming back.
There are people around who never buy on a whim, but there are also a lot of people that buy stuff because the thought crosses their mind, forget about it, then remember when it turns up several days later. These aren't always things you "need" strictly speaking.
As to competition I'd name Jet.com, Target, and Walmart as being major ones. I buy a LOT of stuff from Target these days using the Red Card (effectively 5% pays for the tax, and free delivery on all purchases).
Behavior over 100ms can't be compared to behavior over hours. It's possible that it's true, but it's unlikely.
On the other hand, you can model some buying as periodic, not need-based. I go to the grocery store when I need food, but I go to J Crew once every six months. If I'm traveling this June, I'll delay my trip until July, so while the revenue isn't "lost", it's pushed into the future. My next trip is then in January, not December, and at this point, J Crew has lost one month of my sales forever.
There's a similar effect for many items at Amazon. I only spend so much time shopping, so if I can't do it now, there's some opportunity they miss out on in the future. The aggregate ends up looking like revenue that's lost forever.
Would love to see something come out about how it affected their users.