200 comments

[ 2.3 ms ] story [ 237 ms ] thread
Sorry for the weird query string (?down=yes), but HN already had a amazon.com submission
(comment deleted)
JeffB is for sure getting paged.
What's the guess as to how much revenue Amazon loses for every second of downtime?
I was told varying different numbers at the time I joined. I'd be curious to know as well.
Given their annual revenues of $60B and they were down for ~ 20-25 minutes. So they must have lost ~$3M in revenues during entire downtime.

P.S. Wild guess. No idea about how much sale do they make during peak hours.

Not all abandon sales are a loss for Amazon. Many customers will simply just com back and buy another day because they don't know about the competition.
How many $/seconds do you think the homepage being down costs?
Since Amazon's retail operations are unprofitable, they're actually gaining money.
Is this seriously true?? (If so, where do they make their money?)
Volume, volume, volume!
If each item sells at a loss, more volume certainly doesn't help gain money.
Yes, it does. Maybe not from customers..
They are optimizing for market share and innovation rather than profits. It's a world domination thing.

I think it's awesome. Imagine if Google had run a bunch of low-rent punch-the-monkey display ads early on. It would have killed them. Facebook vs MySpace is another good example of what happens when you focus on long-term value creation versus short-term profit taking.

Does that mean its a big bate and switch model? Get market share with cheap prices, destroy the competition then ramp up your prices.
which has never worked in history. as soon as you raise prices, you lose market share...
With its low profit margin amazon leaves virtually no room for a small size competitor to dislodge them from their share of the market.

Say you want a bite of the tablet market dominated by apple, it's easy, make a somewhat decent tablet for cheap and there you have it.

If you want a bite of an amazon dominated market, well good luck with that, and while at it hope that amazon is not planning to get into the market you're in.

It seems their strategy relies on tiny margins, maybe with a different set or circumstances amazon would change their stance, but I don't think it's currently part of their plans to ramp up prices.

It depends on the product, a lot of cosmetics they seem to be selling at a decent margin and getting undercut by marketplace sellers.

If you are going up against the loss leader kindle though it is going to be a lot harder.

From Amazon's perspective, it's the only way to exist in the long term. If they keep costs and margins low, they don't give their competitors much breathing room to challenge them on price.
More likely the other way around. Destroy the competition and keep prices the same, but push costs down.
I'm sure it's just a typo. But for the record, it's "bait and switch". A 'bate and switch model is something else entirely.
It's true that their profit margins can be vanishingly thin, but that doesn't mean they don't make money.

For some classes of items, they can sell at cost and still make money, because their operations are allegedly so good that they can turn over the inventory before their own payment to the supplier is due.

For example, say Amazon buys a book today and payment is due to the publisher in 30 days. They sell the book tomorrow at cost. Now they get to sit on the full price of the book for the rest of the month. In fact, take that money and buy another book, and sell it right away too. Keep that up, and you have a very big pool of money always sitting in your bank account. Money that can be profitably invested in other activities.

Why would a publisher give them 30 days to pay? Because they're Amazon. It's good to be big.

Net 30 is very common, even if you are not Amazon.
And lots of big companies will take 2X or 3X longer to pay you if they can get away with it.
Without revenue coming in their fixed costs will assure they are losing money.
(comment deleted)
Good one. Apparently, no one else is capable of laughing.
Zero, because people who want to buy from Amazon will just try again in a few minutes/hours when it is back up.
Is there a name for the fallacy of ignoring marginal effects at the tail end of a probability distribution? I see it here incredibly frequently.

There will almost certainly be some number of people who would have stopped by Amazon right now and made some impulse purchases. At the scale Amazon operates, the increase in inconvenience to push off the marginal purchase as a function of inconvenience is almost certainly miniscule (See frequent reports on how milliseconds of page load time affect the likelihood of purchase)

Exactly. People say they lose $X/hour. The next hour when they come up, is the order rate back to $X/hour, or $X + delayed demand / hour?

(Surely there's some loss from being down, but it's not a simple loss = current order rate * downtime argument).

I'd be more concerned about the PPC ads leading to a 500 error.
... or go to costco.com, which is a great site that oftentimes can beat amazon's price.
Not completely zero - I can easily imagine how someone trying to buy a gift during a fifteen-minute break would go to barnesandnoble.com or whatever.
I can attest to this being the case. I mentioned it elsewhere in the thread, but we're in e-commerce and when walmart.com went down around black friday last year we saw a 20megabit jump in traffic until their site came back up... and we're only one e-com provider out of many.
Amazon might be able to measure this effect, by taking comparing to their projections over the next day. Imperfect, but...
Not if that customer moved on to a competitor’s site and made the purchase there.
I will take a stab. Using these numbers: http://www.statista.com/statistics/197099/quarterly-revenue-...

61.1 Billion dollars (yearly revenue) / 31556926 seconds = 1936.18 dollars/second

That's assuming that they're revenue is spread out evenly over every day and every minute. Which is not the case. Think Christmas, deals, weekends, time of the day, etc...
It also assumes that everybody who goes to buy something from Amazon while its down end up buying somewhere else as opposed to simply waiting until later that day
That's less than I expected! Back at the height of their popularity, I remember hearing that AOL could lose a lot of money every second their servers weren't showing ads.
> That's less than I expected!

That's still a huge amount of money ... $7mm an hour?

Back when I worked for them (many years ago) I was told about $100,000 in lost orders per minute.
It's January, so not much is being lost here. We're looking at the slowest shopping season of the year. My unscientific estimate based on previous experience in retail would suggest sales are probably 1/50th peak Thanksgiving/Xmas volume.

Still, downtime is money, even if it isn't a world-changing amount of it.

I don't know, there's valentine's day around the corner and I was actually shopping on amazon today.
I just went to amazon to buy something I wanted overnight shipping on. It is down, so I am probably going to have to get it somewhere else.

On a side note, the first thing I did was google, amazon down, and saw it was (http://www.isitdownrightnow.com/amazon.com.html) then I came here, and I am proud to say, this post was #1.

Depends on whether people buy by going to the home page and searching, or just searching on Google.
Back up @ 19:46 GMT.
... and it's down again @ 19:47GMT.
Still down for me in US-East at 19:49 GMT
Interesting. Never thought I'd see that.

Does anyone have statistics for Amazon homepage uptime? I don't remember the last time I heard about Amazon being down.

And an hour after I read Patrick's (patio11) article on the Rails vulnerabilities. It's a scary day indeed.

It's actually surprising it isn't down more often—internally, everyone has write access to prod and the rule is that if you deploy something to prod you need to be able to roll it back.* Apparently, though, someone has failed on the second item.

* Or so I was told in a job interview with the big A a few years back.

I find this extremely hard to believe. (Not calling you a liar, but I think something must have gotten lost in translation).

The possibility for theft and fraud would be so massive if every dev at Amazon had write access to production that I find it nearly impossible to believe this is true.

I imagine he means the ability to deploy code to prod at will, not full access to their database.
Developers probably have access to most production systems. Credit card processing and source of truth on orders that get shipped are most likely segregated. (actually PCI dictates that physical and data access controls be in place so only essential employees can access card data)
Who cares if I can access the credit card processing system if I can insert random code elsewhere in the system that redirects you to my phishing page whenever you enter credit card information?
Given that you would be an Amazon employee with a solid audit trail leading back to you in that scenario, I'd say it's pretty likely you'd be caught and prosecuted rather quickly.
Yes, I and my coworkers could've sold the realtime trades of a petroleum multinational to the highest bidder, including ones that hadn't happened yet. That would've been easy, and would've been worth 100's of millions to someone. Not getting caught and having your life ruined -- that was scary and would've been hard. Now, if I was working for a sovereign power, like China, and my life was there anyhow, then pulling stuff like that in the US wouldn't be so hard.
When did amazon start selling petroleum futures?
Amazon is not a petroleum multinational. Guess again.
The bits that need high security such as production databases have extra layers of access and tracking. But most devs can push changes to the retail website.
Very often at a lot of companies devs can read production databases and upload the contents offsite.
It's true.

One of the reasons I left Amazon was that I was given the job to deploy code regularly (about weekly) at 1am or so, and one evening, there was a problem due to work of another team, so it escalated and we spent 6 hours dealing with it. We rolled the change back right away, but for contractual reasons their code had to be fixed and deployed and there was an interdependency. Fortunately, it wasn't my team's mistake, but I had to be there to help test it, etc.) So, it's finally working at 7am, and I stuck around for 30 minutes to make sure it kept working before going to sleep around 7:45AM.

I emailed my boss about it, and of course he was getting emails the whole while as the tickets status was changing.

Still, the fact that I showed up at 10:15 for the 10AM meeting that morning was "unacceptable" and I got chewed out. (~2 hours sleep!)

I made the mistake of thinking that my HR rep might be someone to talk to about this, because I wasn't sure how to make it clear to him that it was kinda unreasonable (Especially since I told him I'd be late for the meeting)... and that's when I found out that everything I told her was written up in an email & sent to him.... resulting in getting chewed out yet again for going to HR!

The lesson: as a programmer, never work for a boss who can't program, or at least, be very wary of it!

I have to say, it sounds to me like the lesson isn't about bosses who can't program, so much as "don't have a terrible boos". There are plenty of fields I know nothing about, but if I was managing people in that field, I would expect that on 2 hours sleep they wouldn't be effective, and I also wouldn't expect them to work both night and day shifts. It's common sense.
I have to say, from experience, that you shouldn't count on an HR rep for anything. At all.
I hope we get a nice detailed postmortem on this one.
45 minutes of downtime so far, we're seeing mostly 503 responses with an occasional 200 getting through. We've seen a few other smaller outages for amazon.com in the past but this is definitely the longest in at least the last 3-4 years. Details at http://reports.panopta.com/amazon/server/96291
Unfortunately changing the URL from http://www.amazon.com/?down=yes to http://www.amazon.com/?down=no does not appear to fix the problem.
(comment deleted)
(comment deleted)
This seems to be working for me: http://www.amazon.com/?is_it_jeff=yes
Seems to me that it's probably the caching layer that's borked.
Why is_it_jeff?
Jeff Bezos, CEO of Amazon.
I got that bit thank you :-) but I did not understand why there might be a web page on amazon named after bozos, some sort of test page? Or a joke that has completely passed me by?
It's a joke like if you could add a parameter in the URL stating you're the CEO of the company and you will be granted access no matter what.
Funny. I followed the "down=yes" link and got the "Service unavailable" page, as expected.

Then I clicked on the "down=no" link for fun, and the page partially loaded for me. I refreshed it and got the whole front page loaded. And then one more time and got the "Service unavailable" again...

I'm fairly certain the OP only added that text after the link so that it would be allowed by the duplicate link filter on HN.
Hah, we're losing $3 a minute from lost sales while they're down - the travesty!
Backup for US EAST at 14:58
Been a while since I've seen the amazon homepage down. Wow.

I know from the e-commerce side, when walmart.com went down last year we saw a traffic increase (enough to actually link to to the outage for walmart). I wonder if it'll happen here.

Well this explains some things I had to deal with this morning. Apparently some people had issues with EC2 as well.
The beginning of the end of the world!
Maybe they should consider hosting in the cloud.
I hear this grid-computing stuff is pretty capable. distributed applications are the future! Anyone want to pay me for time-share on my VAX?
They should definitely try AWS. It is ridiculous that a simple online store manages its own infrastructure.
AWS was actually UP all this time, so perhaps that's not a bad suggestion.
Wow. You really don't know why AWS exists. AWS is Amazon.com.

http://www.zdnet.com/how-amazon-exposed-its-guts-the-history...

As for why, it's easy. Taxes. Much like Walmart rents its stores form an LLC it owns to write off the taxes and bring down the liability of the largest revenue sector, Amazon can write off their server costs since they can "rent" them from AWS LLC. While AWS makes a good chunk of change, it has nothing on amazon.com so by making AWS its own entity (and event better for them that its publicly available) they get a gigantic tax write off and AWS makes capex expenditures saving them taxes. All In all, the shell game must save amazon millions just like it does for Walmart

I think you might have missed a joke there.
Your humour radar needs calibrating
(comment deleted)
It's up now! But it's strange that they were down. And don't they run on AWS themselves?
not 100%, but more and more teams are moving over; it's a goal that's pursued aggressively, but not reached yet.
Running on AWS doesn't protect you from problems in the applications you're running on AWS. You can `rm -rf /` an EC2 instance and have plenty of problems.
I remember being in a talk by Dr. Vogels last year and he mentioning that *most of the Amazon.com North America services moved over to AWS in September 2011, many other services outside of NA were yet to move.
But... I can't read my front page letter from Jeff?
Maybe they forgot to pay their EC2 bill
I wonder how many millions they've lost
It appears just to be the homepage, but all deep links are unauthenticated. That is: if you were logged in before the site started misbehaving, and use a deep link, you're not logged in on the page that loads.