If only they had access to some kind of scalable cloud hosting service, they could've completely avoided this sort of outage. :)
Jokes aside, I admire the work of the team(s) responsible for Amazon's web site. I use it so often and encounter glitches so rarely that it really stands out when something does go wrong.
Serious question: I've heard that Bezos's approach with building out commercial units is to break down each part of the vertical into separate commercially-viable components. Idea being if AWS doesn't make sense for 3rd parties to use then it may not be economical to use it internally.
Now the question part: would Amazon ever secretly run Amazon.com in a multi-cloud setup, balancing between AWS, GCE, Azure, etc?
As someone that worked at Amazon a long time ago—back when AWS was just getting started—can confirm (historically). Publicly there were a myriad of AWS services; internally all we could use was S3 (for many years), if we were lucky. AWS being born of Amazon's "spare capacity" is an urban legend.
Nowadays, I hear it's quite different, and much of AWS is more rapidly dogfooded.
I'm a big fan of dogfooding. But given Amazon's ecommerce huge computing needs, maybe it would make sense to not burden the compute availability of your fledgling cloud offering with your ginormous system?
If built well your compute offering can scale with the resources you give it. If you have Amazon’s commerce platform running on a pool of computers, the team running commerce can give those computers to EC2 in exchange for an equivalent amount of EC2 quota. In theory it’s a bookkeeping operation.
There are plenty of good reasons why you wouldn’t run Amazon’s commerce on EC2, but I don’t think cloud availability is one of them.
FWIW 100% of the Amazon retail website is run on EC2 (and iirc it's 100% on spot instances too). (There are of course lots of internal services that still use non-ec2 capacity but fewer and fewer every day.)
Some services (before I left) just did not have the sheer capacity to have Amazon.com as a customer at peak. The service teams just said "nope, sorry, you're going to kill us".
The requirements for prime day/black friday/cyber monday were mind-boggling.
That was quickly adjusted. I think like 3 years ago, Amazon publicly was saying that every day to AWS they were adding the same amount of computing power that was used to run Amazon.com when it was a $10 billion business. AWS is massively greater in scale than Amazon.com at this point.
I cannot confirm or deny that (since my memories are sorta fuzzy), but the real problem was not EC2 capacity. Many other AWS services were involved. :)
At the same time you are also giving the other team more flexibility to solve the problem, as they can optimize it for the Amazon.com use case and are not constrained to a generic solution that is optimized for the masses.
> Now the question part: would Amazon ever secretly run Amazon.com in a multi-cloud setup, balancing between AWS, GCE, Azure, etc?
I'm not sure how secretive it would be. AWS 'bids' on Amazon.com's business and Amazon.com is under no obligation to use AWS as it's cloud service provider.
I can't see Amazon ever using external cloud hosting for anything except the most trivial of tasks. They're absolutely, utterly paranoid about any sort of confidential information, and I think even with encryption the perceived risk would be too high.
I don't see their comment as paranoid because cloud providers generally aee incompetent/steal your data, but because other cloud providers are direct competitors.
Even if they're not sifting through your server data, they can possibly try to get a competitive advantage by analyzing things like usage data as someone above pointed out.
When AT&T was forced by law to allow MCI switching equipment in their facilities, they tended to leave that room's window open, so pigeons could nest in the equipment.
(I might have a detail wrong, this and a hundred other great telecom anecdotes are in The Master Switch.)
Always nice to make sure no parties you depend on have conflicting interests.
So... you save $9k a year in recurring and it will be more than 5 years before you break even due to your $68k up front equipment costs.
And that's assuming you don't have any needs to quickly scale up or down and you are limited to 1 colo instead of the ability to expand to multiple regions like with AWS.
And that's not even taking into account the cost of the brain power to make sure your hardware stays up and running.
Doesn't sound like rolling your own stuff in a colo is a very good idea in this case. But that's job security if you are the sys admin I guess.
> And that's not even taking into account the cost of the brain power to make sure your hardware stays up and running.
Although, as I said upthread, I agree that AWS is very likely ideal for this particular deployment size, let me try to dispell this oft-repeated myth.
Modern server hardware takes almost no "brain power" (or effort of any kind) to keep up and running.
We aren't living in the days of the early dot-com boom where Linux-on-Intel in the datacenter could mean flimsy cases, barely rack-mountable, with nary a redundant part to be seen.
Applying some up front "brain power", one can even choose and configure hardware in such a way as to provide things like server-level redundancy, if that's important and/or preferable to intra-server redundancy (think Hadoop), or the ability to abandon mechanical disks in place instead of ever having to replace one.
This is the main "sweet spot" for AWS (or "cloud" infrastructure in general): small scale.
I am generally a strong proponent of using ones own hardware in a colo or on-premises, instead of or in addition to the cloud (primarily for "base" workload).
However, if the entirety of your needs can fit into a single rack, even I will advocate for AWS, since "convenience" is, perhaps, not strong enough a word.
I do think your server and storage prices are around $25k too high, but that's easy to do buying brand name and/or not negotiating with multiple vendors on price (which is particularly tough at low volume unless you're a startup with a credible growth story). That's assuming such an expensive CPU (in comparison to so little RAM) isn't foolishly profligate, along with the other hardware choices. Of course, this underscores the point (on which we agree) that, as a rule, it's just not worth that much time and effort for so little.
I'll take your word on the AWS pricing, as it's fairly predictable, if very tedious to perform the prediction. The main "gotchas" I've found people run into are forgetting to add in EBS costs for EC2 instance types without (or without comparable) local storage and underestimating data transfer costs.
You'll have to trust me that this examples hardware spec
and requirements are for a basic/base site.
You can thin the profile and increase the # of chassis, compromise on redundancy, etc...but experience has shown that this arrangement is most cost effective. Kinetic event
impact modeling system -w- RT data delivery -- that should answer your conjectures.
No large vendors used in this example - thinkmate or aberdeen supermicro re-brands for due diligence and warranty.
> You can thin the profile and increase the # of chassis, compromise on redundancy, etc
No, I wouldn't suggest more chasses, as that's almost always more expensive (it's tough to break even on that $1k minimum buy-in on a server).
I believe your workload needs the resources you say. It just happens to be a remarkably rare ratio, hence my remark.
> No large vendors used in this example - thinkmate or aberdeen supermicro re-brands for due diligence and warranty.
The vendor doesn't have to be large to jack up the price.. Any re-brand is super suspicious. To me, a large part of the point of a commodity server product is the reliability is predictable (and therefore easy enough to engineer for/around). Paying extra for "diligence", warranty, or hardware support is just flushing money down the toilet.
A fee for custom assembly and/or a basic smoke test is fine, but it had better be a flat rate per server and on the order of $100. Technician labor isn't that expensive.
Larger or "enterprise" vendors are merely the extreme version of this, with upwards of a 10x premium on something like storage arrays, especially if one includes
You seem to be an absolute type of planner. I used to approach IT mgmt and provisioning that way some years ago before being confronted with the realities of small and large business. One size obviously does not fit all and
sometimes you take shortcuts..usually you pay for them later.
I agree with your cautions around supermicro resale but
the warranty support and build diligence are absolutely necessary for a small business. Having a good business relationship with a trusted provider of hardware that
always performs the first time is priceless.
I don't know what an "absolute type of planner" is, but I consider myself an engineer and a pragmatist. I'm well versed with realities. In reality, with business, there's no such thing as "priceless", only risk, and risk is, generally, quantifiable. With enough data, it's easily quantifiable.
I admit that, having an affinity for startups, rather than more traditional small businesses, I have a greater affinity for risk. Ironically, perhaps, I'm usually the voice of risk-aversion with respect to IT infrastucture, so I don't believe it affects my overall understanding.
I recently pointed out to an interviewer who was trying to convince me that it was worth spending half a megabuck on a petabyte from Netapp because it was "business critical" instead of 1/10th that amount for DIY, that, just like the DIY solution, Netapp does not indemnify the business against loss. One isn't buying insurance, only a bunch of technology.
Sure, "works the first time" is worth something. Is it worth the cost of a whole, complete, extra server on a order of qty 6? If the infant-mortality rate on servers is anywhere approaching 1-in-6 and they're being shipped somewhere that the replacement time and/or cost would be prohibitive, I'd still probably rather just order 7 servers instead.
That's my main problem with paying a vendor for "reliability": it's a very fuzzy, hand-wavy assurance. Paying for reliability with more hardware has data and statistics behind it, which is an engineering solution.
That is some serious set up. I can’t see how you get close to this only spending $25K a year on AWS - maybe the price I was quoted for my needs was some sort of suckers price.
One of Amazon Retail's big internal goals this year is "finally get everyone off of Oracle".
Bezos opposed the creation of AWS. Almost everything that was early AWS (EC2, S3) was done over the objection of Seattle leadership. Look where the teams were based.
After they shipped AWS, it took eighty zillion years for Retail to use anything AWS offered beyond S3, and, as mentioned above, it's not like their 100% on DDB or RDS now: they have dependencies on freakin' _Oracle_ all over the place.
I mean, this is not a criticism of Bezos or Amazon at all, but at the end of the day, even if Bezos is a supergenius (and I see no reason to doubt that he is. Although I often wonder how he keeps himself motivated to continue to work so hard on building a Walmart competitor with the precious few years he has left on Earth given that he could do literally anything with his time), Amazon is still a company made of tens of thousands of people. It's not a 4 dimensional chess-ballet. It's got probably 95% of the same chaos and disorder that every big organization made of humans has. It just turns out that cutting it 5% in the right markets has incredible returns.
It is beyond absurd people who sell $AMZN on this news. Amazon users will just keep refreshing until it comes back online, and then continue buying as normal.
Precisely. I wouldn’t doubt there are algorithms that take into account general tone of tweets and news posts (using some sort of NLP) into a trading decision.
Yes, this is done in practice and is a little scary because of its potential as a means of manipulating automated trading. On the other hand the markets can be manipulated already via news read by humans...
It's absurd for long term investors to sell on this news but if your a short term trader or bot then it might be a nice short term profit to trade on this news.
Some rabid fans may, but it's ridiculous to assume that Amazon sees zero missed revenue when their website goes down on their (arguably) biggest sale of the year.
That's similar to all of the locks on all Walmart stores inexplicably getting stuck in the locked state for 2 hours at 6AM on Black Friday.
It's not the same at all, because trying Walmart again later in the day means literally driving back to the physical store. With Amazon, it means getting your phone back out.
I'm sure there is some impact, but it's nowhere near the inconvenience of being locked out of a physical store.
If you already waited overnight, then 2 hours isn't going to cause you to go home and come back. You're going to continue to wait in the line until the store opens back up.
sure, they probably don't get that money back which might slightly influence the revenue for the quarter, but it's not as if the value of the business has changed drastically in the last 3 hours.
This happens every time AWS has an outage as well. Reddit is down, better sell AMZN.
My guess is that they will extend prime day. But they fixed it quickly.
For me currently on amazon.de there is no problem and amazon.com shows a captcha form, so it looks like they were DoSed.
We had about $300-$400 worth of purchases lined up this morning that were going to be made just for the day. Site was down, we couldn’t get through - they aren’t going to happen. A lot of people only shop on the 30-40% of days - so, I’m guessing we’llnwait For digital Monday out in November?
After reading Ryan Holiday's "Trust Me I'm Lying" and knowing how well AWS usually handles surges of traffic, I am not so sure that this isn't a marketing stunt that gets them an amazing amount of press. Cynical?
This made me smile. :) This is very close to to Poe's law as well, because after all, who can tell for sure if something that simply just went-wrong was intentional or not?
There are cheaper ways to get marketing press that don't involve kneecapping one of your two primary revenue generation engines. They've gotten at least two major news cycles just out of vaporware promises to deliver packages by drone, for instance. (It may not be vaporware forever but it certainly isn't coming soon enough to justify breathless press from a few months ago.) It's not like they don't have any idea how to drum up press.
Could also be an A/B test of some new infrastructure design to see if it's ready to deploy in November. I have no idea if they still hold everything together with Perl (via this templating system: http://www.masonhq.com/) but it wouldn't surprise me, and it also wouldn't surprise me that there'd be occasional pushes to replace it with something "modern" (or at least friendlier to the revolving door of new grads).
This would make a lot of sense, they essentially get a "game day" with actual sales, make a lot of money and a few months to prepare for the next wave of seasonal mob-shopping.
Nah, not at all. The vast majority of people don't care about the inner workings of a website, they just want to buy cheap shit and have it all work. Prime Day pretty much markets itself, I just recently got a package from amazon and it was plastered with Prime Day advertising, my co-workers are all talking about it, it's the centerpiece of their website. You'd have to work pretty hard NOT to know about Prime Day today.
Last year they sold $2.4B, and this year I've seen estimates around $3.4B - $4B.
Prime day is 36 hours long, but I bet sales are weighted heavily in the first few hours.
So 3.7B / 36 / 60 / 60 = $28.6K per second, and then maybe double or triple that for the first hour or two after 3PM and that'll give you an idea of the scale.
There's also knock on effects, like reduced trust in Amazon/less orders for the rest of prime week, but also positive effects, like people who will just defer their shopping.
For what it's worth -- my sales are below my 30 day average. Glad I didn't go all out this year in terms of advertising.
It probably doesn't help that the mobile app appears to load a random picture of a cute dog every time I press the "retry" button. So you can guess what I'm doing, trying to get it to load a new "Dogs of Amazon" pic.
Probably should have gone with goatse, reduce the load.
EDIT: do NOT search for "goatse" on your work connection. That alone, even if you've never heard the word, should tell you why I suggested it as an alternative.
Maybe they figure that people who like dogs enough to want to see more will also click the link [1] to find out more about the "Dogs of Amazon", which seems to be unaffected by their problems. There they will find a story about Amazon's dog-friendly offices, 30+ pictures of Amazon dogs, and a video showing many of them. The time to consume all that is time the person is not hitting refresh on the shopping site, so maybe is a net win for Amazon.
Interestingly, despite not being a dog person, I found the first instance of a dog to be a cute, human-relatable inclusion. On later pages I found it frustrating and annoying.
I'm sure there's a fundamental UX principle or two at work there, but I won't pretend to truly know what they are.
Any sources for me to read up on? Googling just gave me the BattleChess duck story, some unrelated UX examples using ducks, and info about how the duck hunt "gun" worked.
I've never heard the term before, but I do recall plenty of people playing duck hunt shooting at the dog when the game said they failed. Although, he kinda earned a laser zapping by laughing at your failures, right?
I’m blown away by the sudden and swift down votes to my original comment.
These engineers work at a world class company and are paid vast sums of money to not fuck things up. They live way better off than the majority of the country and their mere presence makes life more expensive and stressful for communities around them.
To suggest they cannot go a mere 48 hours or less without sleep on one of their company’s most hyped days is out of touch.
Amazon is not world class. The pay is not comparatively great. There is no retail job (including Amazon) where undue stress and sleepless nights are warranted.
You're not saving lives, you're selling books and cat litter on the internet.
Is their pay not comparatively great? Usually when I see this statement, folks are comparing seattle to silicon valley 1:1 which isn't a fair comparison. Seattle is expensive but not that expensive. My friends who work at amzn seem to be compensated in line with everyone else I know, but maybe I'm wrong?
That someone can earnestly suggest that going 48 hours without sleep is somehow an effective way to address an outage is an indictment of a messed up work culture.
People need sleep to think straight, and no amount of money or responsibility is going to change that.
That...just isn't true. Missed sleep costs millions of dollars a day and worse, numerous human lives in sleep related accidents (car, medical and more).
They are pretty good about it, and the "Correction of Error" for this one will be pretty epic.
Generally, the rule at Amazon is that any particular f*-up is forgivable... once. (Especially if you can show that you had preventative measures, documented procedures and redundancy in place.)
That said, there will be finger pointing and blame because you're dealing with human beings.
Funny how this comes on the heels of aggressively expanding their workforce and trying to leverage themselves in a hundred different directions...
Maybe it's just me and my confirmation bias at work, but it seems that the core value proposition that Amazon provided -- high value, low margins on products -- has been eroding before our eyes.
Seems so much like the transition Microsoft made... too much focus on "synergies" and leveraging... not enough on keeping the bilges dry and the engine running.
It's funny... Fred Brooks wrote about this in 1975... and we're still making the same mistakes forty years later. There are real limitations to how quickly any organization can grow. Even awesome companies who are excellent at building organizations -- places like like Amazon and Microsoft -- can't organize this law of software development away.
> Seems so much like the transition Microsoft made... too much focus on "synergies" and leveraging... not enough on keeping the bilges dry and the engine running.
The companies that just keep the bilges dry and the engine running are the ones that we love, but they’re gone because they got made irrelevant. Or they got absorbed into something larger. Microsoft has a bunch of failed initiatives (Windows Phone, Zune) plus a bunch of successful ones (Azure, Xbox, Office 365).
If you’re up for classic books like Brooks check out The Innovator's Dilemma. You have to try to expand in a hundred different directions because you don’t know which one of those hundred directions will be relevant next decade, and you have to be unafraid of cannibalizing your core business because if you don’t eat yourself then someone else will eat you instead.
I think the hard part is to walk the line between stagnation & over-expansion. This is a dilemma we all face in organizations large and small as well as individually.
Building systems and processes (a.k.a Habits) that allow us to assume and integrate some new set of "stuff" without having to think about them (so we can move on to the next new "stuff") are what sets these companies apart; Amazon has been brilliant at this; however, from my perspective, looking at all the high-rise office buildings going up, it seems like maybe Icarus has flown too high...
But again, I could be building a narrative to fit my preconceived ideas... I'm definitely no authority here. Maybe this is just another blip... I think more troubling to me is the overall degradation in quality of the things I have formerly taken for granted in Amazon -- the quality of the products and ratings.
The Innovator's Dilemma is certainly not about synergies between products. By its conclusion, it's worthless to try to diversify within the same structure, you'd better create a new company and turn the old one into a holding.
Why would you think it’s about synergies? I’m not sure why you would think it’s about synergies. Or why it would be “worthless to try to diversify within the same structure.”
Summarizing the book here would be a bit of a disservice—but one of the points of the book is that there are economic reasons why companies focus on their most profitable core products, and there are economic reasons why that kind of focus can result in the company collapsing when the market moves forward. This isn’t some kind of imperative—the book isn’t saying, ”therefore, you should create a new company.” It’s more descriptive, “this is how big, successful tech companies can suddenly fail.”
This seems like a bit of an overreaction. Amazon has had big outages before (particularly at AWS), and they invariably solve the problem and move on. As someone else observed, events like this are mainly notable because of how infrequently they happen.
Kinda wish I still worked there and could see the post-outage writeup.
My bet's on some unforeseen bottleneck that affects search and static pages. Almost everything within Amazon is crazy scaleable, but there are some bits where you scale them up and their behaviour changes radically. For instance, a service's cache misses might skyrocket as customers get distributed over a wider set of servers, causing service response times to increase just a little bit on average, tipping a dependent service over into more frequent timeouts, causing its downstream service to blow a timeout-percentage 'software fuse' and stop using that service... etc etc.
Given that each of those services (and many more possibly-related ones) will have an on-call engineer paged into a conference call when the manure hit the rotating ventilation apparatus, there are going to be a lot of unhappy people cancelling their weekend plans right now. I definitely don't miss that aspect of the job!
my memorial weekend on a few acre olympic rain forest camping trip: regular signal hunting for hn. Oddly found a 6x6 area which after a reboot with provide lte. after idle would have to reboot again to open some more links
I'm on holiday in Japan at the moment and have no idea what day it is. Only checking HN as I'm currently in Osaka waiting for the Shinkansen to Hiroshima. ;)
Hehe. Reminds me of my Murphy's law momemts. Whenever I am on a holiday, a deployment will fail, and I will have to login and check while my family plays volleyball on the beach.
I'd tell your boss to hire someone to replace you. What if instead of on holiday, you were dead? Kinda hard to login and fix their shit when you're dead.
I agree. I quit every job where they don't respect my personal time, especially vacation. Life is just too short. It's bad enough what I've been through with companies and their stingy PTO policies (US).
I was smoked up in colorado for the old "Lets take down Dyn real quick" BS a couple years ago. I know this very well, things like to run for months without issue, and I'm off for an afternoon with the phone ringing.
Not oncall for Amazon but I don't particularly mind it. It sucks to have to be near a computer for the time, but I actually find digging in and finding that needle in a haystack pretty fun. On a good team, the life interruption is minimal.
When I was on-call for a big project, I jumped every time my phone rang. It wasn't a good experience at all.
"We are at the fair, I have 18 customers lined up but I can't take orders. Can you fix this now or should I start taking them on paper?"
You know the joke that atheists don't believe in god until the plane nosedives? That was the moment for me :)
"Oh, I... I need to see the error message before I can say anything. We don't have any errors logged. Hmm... Would you allow me to remote into your device?"
"Can't do that. We lost the internet connection here, sorry."
It wasn't though. I saw an item being listed as a Prime Day deal on the normal Amazon site, then I searched for it through smile.amazon.com and it wasn't there. Went back to non-smile and it was there (but throwing errors so I couldn't click it anyway...)
I have no idea really (I stopped working on the retail websites in 2013) but my gut feel is that that's about a couple of orders of magnitude too low at least.
But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...
Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point.
Don't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.
Not offhand, sorry - I first encountered the notion at a conference some years back, but I've long since gotten out of operations so it's all a bit dusty for me now.
It might also be a momentary blip that you really shouldn't need to wake a human up for. The hard part of monitoring imo is that perfect balance. Asymtotically approaching nirvana but never quite reaching.
All monitoring has that problem :-) But if your rate of rate of change has been positive for some suitable amount of time for the context, it's worth waking a human up over it, because the amount of resource remaining is dwindling exponentially.
This is their official rate limit for the FIFO type SQS. Although I bet the Amazon folks know a couple people around the office who can up that limit :)
I once owned an EAV DB for storing small config values. Someone wrote a great wrapper around it that made it seem like a proper DB with many tables. Since it was for storing configs this library cached the whole thing in memory on startup. Zoom to 5 years later, we have 10+ settings for every customer in this store, and one day all our hosts keep failing over. As it turns out that small table for configs was north of 5 gigs and destroying our heap.
OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems
OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data
OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.
The systems at Amazon are too large to leverage autoscaling. When I was there (in Marketplace) it was generally estimated that Amazon used about 80X the capacity of their entire AWS public cloud.
> Amazon used about 80X the capacity of their entire AWS public cloud
which is probably closer to "the capacity that is available via AWS is a tiny, tiny fraction of their overall computing power.. therefore adding it back in when things are falling over doesn't actually solve any problems."
I did’t understand either. I suspect it’s the semantic difference between ‘public cloud’ and ‘infrastructure’. I don’t know what that difference is really.
I think they are using the term 'capacity' to mean 'spare capacity'. I.e. that Amazon's entire compute usage is 80x the spare capacity, so scaling even a small amount would consume any spare capacity in AWS. Still, it seems hard to believe.
1. he misspoke and meant "80%" of the AWS capacity, which I agree seems implausible.
2. Amazon does not run on AWS because Amazon is 80x more than all of AWS infrastructure. This also seems implausible because of Netflix. In fact, there's an article out there that said AWS exceeded Amazon's capacity within 1 quarter!
I still don't understand what that has to do with autoscaling exactly
I find that implausible. I know they run a huge infrastructure but why would they be using 80X of their AWS public cloud? Thousands of companies if not millions at this stage use AWS and some of them are not insignificant in size.
When I checked it out, the site seemed responsive but search was broken. I searched "usb flash drive" and was presented with 2 results. I pondered this for a moment and then realized that there were 400 pages with 2 results/page (as far as I checked.) Perhaps there is some load shedding algorithm that reduces the result count per page to reduce load. It did discourage me from further searching so I guess it worked. ;)
I'm seeing automatic reloading the page every second or so. Maybe some bad javascript, though it isn't adding entries to the history. Looks like they have a script that is DDoS themselves.
I have completely stopped using Amazon, with the final nail in the coffin being the stories about how they treat their workers. I think it's just a matter of dependence and most of the stuff I used to get, I can find it elsewhere.
There's another way to look at the workers situation. They do have jobs, and let me frankly say that in (for instance) Patterson City, California (aka bumf__k egypt, basically) creating thousands of jobs is not nothing...
Slaves "had jobs" as well. Their masters just paid them in food, water, and shelter.
This whole thing about businesses "giving jobs" is ludicrous. American brainwashing. Businesses NEED workers. Not the other way around. Humans have always existed and survived for hundreds of thousands of years without "jobs". Businesses cannot survive without workers. Businesses would simply cease to exist without workers. People can still grow their own veggies and meat. Businesses can't generate profits without workers.
Better be careful. I was just warned about making obvious and non-threatening jokes on this site by the main headquarters themselves. In fact, this is my last comment ever. I'm done with this site and the group of babies that run it. Take care.
I was only showing the other side of equation... I'm as Libertarian as they come and also believe Karl Marx's theories made tons of sense...(I know that sounds a bit contradictory!)
Amazon needs to share its wealth much more, at least among its workers and also independent authors using it, and so on, and realize that it needs to support the ecosystem that allowed it, including things like more freedom and liberty for people, not less. This vast accumulation of wealth is bad news, even for capitalism itself, when you get right down to it. Bezos has a huge chance to make a stellar example to the world here, but so far...ummm...i mean why not?
Wouldn't it help them to purchase from Amazon today? If there are lots of orders but no one to fulfill it seems like that would be greater pressure on Amazon.
My guess is this is an intentional marketing ploy. Think of how much press this is generating. Frontpage of HN. No way we are talking about this otherwise.
What happens to people responsible for the crash today (infra, culprit services)? Does Amazon take some kind of "action" since Prime Day is a huge, once-a-year event for Amazon?
They will have to write a detailed post-mortem (with many people with titles starting from director watching it every week). Based on what comes out from the post-mortem automation/testing will be implemented to remove/mitigate the failure from the equation in the future.
Unless this is something egregious (ex: a manager not allowing the team to "scale up" in preparation for the event) no one will get fired. Tempers may flare a bit if it is something stupid (it's usually not).
Nothing different from what happened with the (very public) DynamoDB and S3 failures of yesteryear.
I wonder how Alibaba Cloud handles similar events [1], where there are bursts of 256k/s transactions and ~1bil packages being shipped out.
Do they just do brute-force massive scale out?
Amazon's US market is big, but my understanding is that number of online users in China (> 400mil) exceeds the population of the US (~325mil), which makes me wonder if the folks there think about data architecture a little differently than we do.
Oh I included that number to indicate the scale of orders that have to be handled by Alibaba's data systems -- I understand that they don't ship the packages themselves.
Also, I just read that as of 2017, there are 700mil internet users in China, 90% on mobile. The scale there is just staggering.
... Is there anything even good for Prime Day this year? Or the year before? Two years ago I remember seeing at least some Dell Workstations that could be repurposed into cheapo home servers. Most of the stuff seems to be odds, ends, and stuff that the various Chinese product-clone companies couldn't get rid of.
If you aren't being a little sarcastic, you are lucky. The bug saved you money for shit that you obviously are not wishing very hard for. Kids at Christmas time certainly don't need a backup....
I wouldn't blame yourself too hard.. Amazon has strategically made it very easy to add items to your lists (bookmarklets, chrome extensions, etc.) while offering _no_ supported way to get the list back out.
Our team also provides the following additional services.
Catch a cheating significant other by hacking social media, messengers and more
How to remove negative information online and hack search results
How to hack a degree in 30 days
How to hack your credit score
Take back control of your social media
And more
We want to make sure you understand what WE CAN do and what WE CAN NOT do:
We can not: Hack people's personal information or accounts for personal gain
WE CAN: Help create a detailed profile of your subject's online and electronic activity, including location, pictures, chats etc.
We can not: Hack an account and give it to you
WE CAN: Recover or delete stolen accounts and return its control to you.
We can not: Hack your credit report on Experian, Transunion, and Equifax
WE CAN: Work with you and your credit report to remove negative entries, insert positive ones and even help you create an all new credit file.
We can not: Hack your university to get you straight A's
WE CAN: Work with you to get you straight A's in almost any degree, course or certification.
We can not: Hack what's app to help you catch a cheating partner
WE CAN: Help you find out if your significant other is or is not cheating using their online media and electronic footprint.
We can not: Hack Google to get your content removed from search results.
WE CAN: Remove content from its source and help you control your search content so the negative content disappears or is suppressed.
We can not Hack into a system or account you do not own or have LEGAL permission to access.
WE CAN: do penetration tests, phishing attacks, code injections and more to test the security and accessibility of accounts and systems you own or do have Legal permission access.
We get a lot of calls a day so we may not respond to you right away. Paid member brokers get priority access and are called back first.
It's interesting that you can have hired the best talents in the world, but still have major outages. I wonder if there a way to ensure more people = more stability. Sounds a bit stupid, but maybe if each datacenter has its own software team working on the same issue, it will be very redundant work, but maybe it will be more organic in its failure?
380 comments
[ 3.6 ms ] story [ 394 ms ] threadJokes aside, I admire the work of the team(s) responsible for Amazon's web site. I use it so often and encounter glitches so rarely that it really stands out when something does go wrong.
Now the question part: would Amazon ever secretly run Amazon.com in a multi-cloud setup, balancing between AWS, GCE, Azure, etc?
Nowadays, I hear it's quite different, and much of AWS is more rapidly dogfooded.
There are plenty of good reasons why you wouldn’t run Amazon’s commerce on EC2, but I don’t think cloud availability is one of them.
The requirements for prime day/black friday/cyber monday were mind-boggling.
I'm not sure how secretive it would be. AWS 'bids' on Amazon.com's business and Amazon.com is under no obligation to use AWS as it's cloud service provider.
Hell, even us regular joes can do that with services like Packet.Net or Vultr.
I can't see Amazon ever using external cloud hosting for anything except the most trivial of tasks. They're absolutely, utterly paranoid about any sort of confidential information, and I think even with encryption the perceived risk would be too high.
The asymmetry in trust is partly because those cloud providers are huge, and partly because those things are part of their core competency.
Even if they're not sifting through your server data, they can possibly try to get a competitive advantage by analyzing things like usage data as someone above pointed out.
(I might have a detail wrong, this and a hundred other great telecom anecdotes are in The Master Switch.)
Always nice to make sure no parties you depend on have conflicting interests.
In my business I can put a hardware site online with
---6X--- 2 x intel gold 5115 10 core + 64 GB RAM 1 nvme @512G + soft raid1 @4TB magnetic 1 10G, 2 1G ether ~= 42K.
---2X--- storage or NAS with 60TB @RAID5 + 2x quad core low end xeon + 32 GB RAM = 16k
---2X--- 1G edge/core mngd switches + 10G SAN/LAN mngd switches = 5K
---2X--- Endian firewalls + threat appliances = 5K
---1X--- Colo with 2 year lease and 25 amps @208v 1g port speed and committed throughput > 100/mbps = 16K yearly
68K one time cost for depreciating assets we maintain, provision and secure + 16K yearly recurring cost.
Or I can go AWS and modify my processing model, security expectations and service infra and spend 25K a year + 15K 1x migration cost.
And that's assuming you don't have any needs to quickly scale up or down and you are limited to 1 colo instead of the ability to expand to multiple regions like with AWS.
And that's not even taking into account the cost of the brain power to make sure your hardware stays up and running.
Doesn't sound like rolling your own stuff in a colo is a very good idea in this case. But that's job security if you are the sys admin I guess.
Although, as I said upthread, I agree that AWS is very likely ideal for this particular deployment size, let me try to dispell this oft-repeated myth.
Modern server hardware takes almost no "brain power" (or effort of any kind) to keep up and running.
We aren't living in the days of the early dot-com boom where Linux-on-Intel in the datacenter could mean flimsy cases, barely rack-mountable, with nary a redundant part to be seen.
Applying some up front "brain power", one can even choose and configure hardware in such a way as to provide things like server-level redundancy, if that's important and/or preferable to intra-server redundancy (think Hadoop), or the ability to abandon mechanical disks in place instead of ever having to replace one.
I am generally a strong proponent of using ones own hardware in a colo or on-premises, instead of or in addition to the cloud (primarily for "base" workload).
However, if the entirety of your needs can fit into a single rack, even I will advocate for AWS, since "convenience" is, perhaps, not strong enough a word.
I do think your server and storage prices are around $25k too high, but that's easy to do buying brand name and/or not negotiating with multiple vendors on price (which is particularly tough at low volume unless you're a startup with a credible growth story). That's assuming such an expensive CPU (in comparison to so little RAM) isn't foolishly profligate, along with the other hardware choices. Of course, this underscores the point (on which we agree) that, as a rule, it's just not worth that much time and effort for so little.
I'll take your word on the AWS pricing, as it's fairly predictable, if very tedious to perform the prediction. The main "gotchas" I've found people run into are forgetting to add in EBS costs for EC2 instance types without (or without comparable) local storage and underestimating data transfer costs.
No large vendors used in this example - thinkmate or aberdeen supermicro re-brands for due diligence and warranty.
No, I wouldn't suggest more chasses, as that's almost always more expensive (it's tough to break even on that $1k minimum buy-in on a server).
I believe your workload needs the resources you say. It just happens to be a remarkably rare ratio, hence my remark.
> No large vendors used in this example - thinkmate or aberdeen supermicro re-brands for due diligence and warranty.
The vendor doesn't have to be large to jack up the price.. Any re-brand is super suspicious. To me, a large part of the point of a commodity server product is the reliability is predictable (and therefore easy enough to engineer for/around). Paying extra for "diligence", warranty, or hardware support is just flushing money down the toilet.
A fee for custom assembly and/or a basic smoke test is fine, but it had better be a flat rate per server and on the order of $100. Technician labor isn't that expensive.
Larger or "enterprise" vendors are merely the extreme version of this, with upwards of a 10x premium on something like storage arrays, especially if one includes
I agree with your cautions around supermicro resale but the warranty support and build diligence are absolutely necessary for a small business. Having a good business relationship with a trusted provider of hardware that always performs the first time is priceless.
I admit that, having an affinity for startups, rather than more traditional small businesses, I have a greater affinity for risk. Ironically, perhaps, I'm usually the voice of risk-aversion with respect to IT infrastucture, so I don't believe it affects my overall understanding.
I recently pointed out to an interviewer who was trying to convince me that it was worth spending half a megabuck on a petabyte from Netapp because it was "business critical" instead of 1/10th that amount for DIY, that, just like the DIY solution, Netapp does not indemnify the business against loss. One isn't buying insurance, only a bunch of technology.
Sure, "works the first time" is worth something. Is it worth the cost of a whole, complete, extra server on a order of qty 6? If the infant-mortality rate on servers is anywhere approaching 1-in-6 and they're being shipped somewhere that the replacement time and/or cost would be prohibitive, I'd still probably rather just order 7 servers instead.
That's my main problem with paying a vendor for "reliability": it's a very fuzzy, hand-wavy assurance. Paying for reliability with more hardware has data and statistics behind it, which is an engineering solution.
I'd hope for a more substantive reply, if anything.
One of Amazon Retail's big internal goals this year is "finally get everyone off of Oracle".
Bezos opposed the creation of AWS. Almost everything that was early AWS (EC2, S3) was done over the objection of Seattle leadership. Look where the teams were based.
After they shipped AWS, it took eighty zillion years for Retail to use anything AWS offered beyond S3, and, as mentioned above, it's not like their 100% on DDB or RDS now: they have dependencies on freakin' _Oracle_ all over the place.
I mean, this is not a criticism of Bezos or Amazon at all, but at the end of the day, even if Bezos is a supergenius (and I see no reason to doubt that he is. Although I often wonder how he keeps himself motivated to continue to work so hard on building a Walmart competitor with the precious few years he has left on Earth given that he could do literally anything with his time), Amazon is still a company made of tens of thousands of people. It's not a 4 dimensional chess-ballet. It's got probably 95% of the same chaos and disorder that every big organization made of humans has. It just turns out that cutting it 5% in the right markets has incredible returns.
That's similar to all of the locks on all Walmart stores inexplicably getting stuck in the locked state for 2 hours at 6AM on Black Friday.
I'm sure there is some impact, but it's nowhere near the inconvenience of being locked out of a physical store.
This happens every time AWS has an outage as well. Reddit is down, better sell AMZN.
Tons of people that shop sales will just spend their money somewhere else if they miss a sale.
Presumably the price sensitive customer doesn’t just go elsewhere, right?
If you can't even search for products, there's a problem!
That's it, I'm terming this the Klein effect.
[1] https://news.ycombinator.com/item?id=17499447
I don't think they're using Perl anymore however.
https://i.imgur.com/vpIHDpA.jpg?1
alternatively, Amazon rebrands their entire cloud lineup to PAWS
I use this Chrome extension that rewrites all amazon pages to use it: http://www.smilealways.io/
FF equivalent: https://addons.mozilla.org/en-US/firefox/addon/auto-smile/
Prime day is 36 hours long, but I bet sales are weighted heavily in the first few hours.
So 3.7B / 36 / 60 / 60 = $28.6K per second, and then maybe double or triple that for the first hour or two after 3PM and that'll give you an idea of the scale.
There's also knock on effects, like reduced trust in Amazon/less orders for the rest of prime week, but also positive effects, like people who will just defer their shopping.
For what it's worth -- my sales are below my 30 day average. Glad I didn't go all out this year in terms of advertising.
Probably should have gone with goatse, reduce the load.
EDIT: do NOT search for "goatse" on your work connection. That alone, even if you've never heard the word, should tell you why I suggested it as an alternative.
[1] https://blog.aboutamazon.com/working-at-amazon/how-much-does...
I'm sure there's a fundamental UX principle or two at work there, but I won't pretend to truly know what they are.
Warning: turn on your adblocker
(Another "makes you feel old" moment when someone says they had to look up "goatse".)
Not even 30 yet - damn it hurts to put it writing; but clearly old enough to know the old interwebz :)
These engineers work at a world class company and are paid vast sums of money to not fuck things up. They live way better off than the majority of the country and their mere presence makes life more expensive and stressful for communities around them.
To suggest they cannot go a mere 48 hours or less without sleep on one of their company’s most hyped days is out of touch.
You're not saving lives, you're selling books and cat litter on the internet.
Disclaimer: My knowledge is based solely off of public reporting and first hand experiences of SWEs and TAMs no longer at Amazon/AWS.
People need sleep to think straight, and no amount of money or responsibility is going to change that.
I remember people were praising them as such when this outage happened: https://aws.amazon.com/message/41926/
Generally, the rule at Amazon is that any particular f*-up is forgivable... once. (Especially if you can show that you had preventative measures, documented procedures and redundancy in place.)
That said, there will be finger pointing and blame because you're dealing with human beings.
Maybe it's just me and my confirmation bias at work, but it seems that the core value proposition that Amazon provided -- high value, low margins on products -- has been eroding before our eyes.
Seems so much like the transition Microsoft made... too much focus on "synergies" and leveraging... not enough on keeping the bilges dry and the engine running.
It's funny... Fred Brooks wrote about this in 1975... and we're still making the same mistakes forty years later. There are real limitations to how quickly any organization can grow. Even awesome companies who are excellent at building organizations -- places like like Amazon and Microsoft -- can't organize this law of software development away.
The companies that just keep the bilges dry and the engine running are the ones that we love, but they’re gone because they got made irrelevant. Or they got absorbed into something larger. Microsoft has a bunch of failed initiatives (Windows Phone, Zune) plus a bunch of successful ones (Azure, Xbox, Office 365).
If you’re up for classic books like Brooks check out The Innovator's Dilemma. You have to try to expand in a hundred different directions because you don’t know which one of those hundred directions will be relevant next decade, and you have to be unafraid of cannibalizing your core business because if you don’t eat yourself then someone else will eat you instead.
I think the hard part is to walk the line between stagnation & over-expansion. This is a dilemma we all face in organizations large and small as well as individually.
Building systems and processes (a.k.a Habits) that allow us to assume and integrate some new set of "stuff" without having to think about them (so we can move on to the next new "stuff") are what sets these companies apart; Amazon has been brilliant at this; however, from my perspective, looking at all the high-rise office buildings going up, it seems like maybe Icarus has flown too high...
But again, I could be building a narrative to fit my preconceived ideas... I'm definitely no authority here. Maybe this is just another blip... I think more troubling to me is the overall degradation in quality of the things I have formerly taken for granted in Amazon -- the quality of the products and ratings.
Summarizing the book here would be a bit of a disservice—but one of the points of the book is that there are economic reasons why companies focus on their most profitable core products, and there are economic reasons why that kind of focus can result in the company collapsing when the market moves forward. This isn’t some kind of imperative—the book isn’t saying, ”therefore, you should create a new company.” It’s more descriptive, “this is how big, successful tech companies can suddenly fail.”
?
The book has 2 chapters on that single point. And repeats it everywhere else.
On synergies, the OP post was about it, not about disruption.
My bet's on some unforeseen bottleneck that affects search and static pages. Almost everything within Amazon is crazy scaleable, but there are some bits where you scale them up and their behaviour changes radically. For instance, a service's cache misses might skyrocket as customers get distributed over a wider set of servers, causing service response times to increase just a little bit on average, tipping a dependent service over into more frequent timeouts, causing its downstream service to blow a timeout-percentage 'software fuse' and stop using that service... etc etc.
Given that each of those services (and many more possibly-related ones) will have an on-call engineer paged into a conference call when the manure hit the rotating ventilation apparatus, there are going to be a lot of unhappy people cancelling their weekend plans right now. I definitely don't miss that aspect of the job!
> there are going to be a lot of unhappy people cancelling their weekend plans right now.
The week just started... are you saying that you can already anticipate the war room for an event like this to extend through this coming weekend?
"We are at the fair, I have 18 customers lined up but I can't take orders. Can you fix this now or should I start taking them on paper?"
You know the joke that atheists don't believe in god until the plane nosedives? That was the moment for me :)
"Oh, I... I need to see the error message before I can say anything. We don't have any errors logged. Hmm... Would you allow me to remote into your device?"
"Can't do that. We lost the internet connection here, sorry."
Ugh.
https://zwischenzugs.com/2017/04/04/things-i-learned-managin...
But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...
https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/
Does Amazon expect a fix to be deployed ASAP after the immediate crisis is averted?
OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems
OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data
OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.
> Amazon used about 80X the capacity of their entire AWS public cloud
which is probably closer to "the capacity that is available via AWS is a tiny, tiny fraction of their overall computing power.. therefore adding it back in when things are falling over doesn't actually solve any problems."
1. he misspoke and meant "80%" of the AWS capacity, which I agree seems implausible. 2. Amazon does not run on AWS because Amazon is 80x more than all of AWS infrastructure. This also seems implausible because of Netflix. In fact, there's an article out there that said AWS exceeded Amazon's capacity within 1 quarter!
I still don't understand what that has to do with autoscaling exactly
Meanwhile Jeff Bezos' fortune is approximately $1,000 per American household.
This whole thing about businesses "giving jobs" is ludicrous. American brainwashing. Businesses NEED workers. Not the other way around. Humans have always existed and survived for hundreds of thousands of years without "jobs". Businesses cannot survive without workers. Businesses would simply cease to exist without workers. People can still grow their own veggies and meat. Businesses can't generate profits without workers.
Not as an alternative means of even survival-level support if they don't have access to sufficient suitable real estate to do so.
Amazon needs to share its wealth much more, at least among its workers and also independent authors using it, and so on, and realize that it needs to support the ecosystem that allowed it, including things like more freedom and liberty for people, not less. This vast accumulation of wealth is bad news, even for capitalism itself, when you get right down to it. Bezos has a huge chance to make a stellar example to the world here, but so far...ummm...i mean why not?
It can't be that companies have crippled unions so that they can treat wage and working conditions as a one-sided negotiation. Surely not...
just look up province names + strike
"Hey did you know Amazon is unreliable and unusable?" "Really? (checks) Yup, you're right!"
(somewhere in the distance, a marketing guru cackles with delight)
Unless this is something egregious (ex: a manager not allowing the team to "scale up" in preparation for the event) no one will get fired. Tempers may flare a bit if it is something stupid (it's usually not).
Nothing different from what happened with the (very public) DynamoDB and S3 failures of yesteryear.
Do they just do brute-force massive scale out?
Amazon's US market is big, but my understanding is that number of online users in China (> 400mil) exceeds the population of the US (~325mil), which makes me wonder if the folks there think about data architecture a little differently than we do.
[1] https://qz.com/1127087/singles-day-crazy-stats-from-alibabas...
Also, I just read that as of 2017, there are 700mil internet users in China, 90% on mobile. The scale there is just staggering.
https://gph.is/1SuGDrJ
We can not: Hack people's personal information or accounts for personal gain WE CAN: Help create a detailed profile of your subject's online and electronic activity, including location, pictures, chats etc. We can not: Hack an account and give it to you WE CAN: Recover or delete stolen accounts and return its control to you. We can not: Hack your credit report on Experian, Transunion, and Equifax WE CAN: Work with you and your credit report to remove negative entries, insert positive ones and even help you create an all new credit file. We can not: Hack your university to get you straight A's WE CAN: Work with you to get you straight A's in almost any degree, course or certification. We can not: Hack what's app to help you catch a cheating partner WE CAN: Help you find out if your significant other is or is not cheating using their online media and electronic footprint. We can not: Hack Google to get your content removed from search results. WE CAN: Remove content from its source and help you control your search content so the negative content disappears or is suppressed. We can not Hack into a system or account you do not own or have LEGAL permission to access. WE CAN: do penetration tests, phishing attacks, code injections and more to test the security and accessibility of accounts and systems you own or do have Legal permission access. We get a lot of calls a day so we may not respond to you right away. Paid member brokers get priority access and are called back first.
Prosperous Regards,
Contact us on :
Text:786 708 9974
Email:theredhackergroup@gmail.com
You probably want a distributed team though, with maybe fetus being thrown around by pneumatic mail.
It would be better to go with a divisor of 10 so as to assign precisely one finger and one toe to each of the distributed work groups.
(But, I have to admit, you may want yet another group to coordinate and assemble it all.)