HN Explorer

AWS US East is experiencing high error rates on several services

276 points by oliverfriedmann ↗ HN
The services CloudWatch, SES, SNS, SQS, SWS, AutoScale, Cloud Formation, Directory Service, Key Mgmt and Lambda experience very high error rates for about 3 hours now.

Dynamo DB is throttling API access and seems to be having issues with the management of meta data.

183 comments

[ 2.5 ms ] story [ 209 ms ] thread
airza
well, time to find out who has failure tolerance built in to the architecture 8^)
drendorx39
failure tolerance is an alien technology for amazon...
divideby0
that's completely untrue. there are many ways to do fault-tolerance in AWS. it's expensive, but it's possible. netflix even goes as far as simulating the failure of entire aws regions in their simian army testing suite:

http://techblog.netflix.com/2011/07/netflix-simian-army.html

That's why Netflix stays up when us-east or us-west are down.

kordless
It is true. Amazon offloads a decent amount of fault tolerance to the application provider, as you point out here. I will also mention that Netflix does not solely rely on Amazon for running their services. They run their own decentralized caching layer: https://openconnect.netflix.com/deliveryOptions/
crb
"4:52 AM PDT We want to give you more information about what is happening. The root cause began with a portion of our metadata service within DynamoDB. This is an internal sub-service which manages table and partition information. Our recovery efforts are now focused on restoring metadata operations. We will be throttling APIs as we work on recovery."

(http://status.aws.amazon.com/)

[deleted]
(comment deleted)
taf2
The main issue appears to be DynamoDB

Here's a copy from the status page.

3:00 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region.

3:26 AM PDT We are continuing to see increased error rates for all API calls in DynamoDB in US-East-1. We are actively working on resolving the issue.

4:05 AM PDT We have identified the source of the issue. We are working on the recovery.

4:41 AM PDT We continue to work towards recovery of the issue causing increased error rates for the DynamoDB APIs in the US-EAST-1 Region.

4:52 AM PDT We want to give you more information about what is happening. The root cause began with a portion of our metadata service within DynamoDB. This is an internal sub-service which manages table and partition information. Our recovery efforts are now focused on restoring metadata operations. We will be throttling APIs as we work on recovery.

5:22 AM PDT We can confirm that we have now throttled APIs as we continue to work on recovery.

5:42 AM PDT We are seeing increasing stability in the metadata service and continue to work towards a point where we can begin removing throttles.

virtuallynathan
6:19 AM PDT The metadata service is now stable and we are actively working on removing throttles.

7:12 AM PDT We continue to work on removing throttles and restoring API availability but are proceeding cautiously.

7:22 AM PDT We are continuing to remove throttles and enable traffic progressively.

7:40 AM PDT We continue to remove throttles and are starting to see recovery.

7:50 AM PDT We continue to see recovery of read and write operations and continue to work on restoring all other operations.

8:16 AM PDT We are seeing significant recovery of read and write operations and continue to work on restoring all other operations.

[deleted]
(comment deleted)
[deleted]
(comment deleted)
justinholmes
9:12 AM PDT Between 2:13 AM and 8:15 AM PDT we experienced increased error rates for API requests in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.
colinbartlett
This is manifesting itself as downtime for a lot of companies, including Heroku: https://status.heroku.com

If you want alerts on this sort of thing, my side project StatusGator https://statusgator.io will alert you when services post downtime on their status pages. My dashboard blew up this morning with a ton of red and yellow as soon as Amazon started flaking.

Edit: I suppose it's time to invest in a multi-region setup. Since StatusGator is hosted on Heroku in the US-East region, it is in theory affected by this problem though so far is still up.

fidz
From Heroku Status Page:

> Our service provider is still working towards resolution of this issue. We will update when we have news, or in 1 hour.

I wonder why they don't tell that AWS is their service provider. Is it wrong to make the information less obscure?

ukd1
Yes, as it's shifting the blame away from their choice - which was to use AWS.
jonaf
I expect the reason is because Heroku could switch to a new provider in the future and it would be a pain to always update every reference to their provider.

Parent seems to imply there's something wrong with choosing AWS. There is not. (forgive me if I mistook the tone)

ukd1
You did; there is nothing wrong - but it's a choice. I.e. your site being down is your fault ultimately, not AWS's - you choose AWS (instead of many other choices OR making something with multiple, etc).

Blaming it upstream is hiding passing the buck on your decision.

fomb
Indeed. If your site is down, it's your fault - no-one elses. Yes it may be down because someone downstream is down, but ultimately it's your choice where and how you host.
cwilkes
Because it is kind of awkward PR for Salesforce: come build on our platform! Oh by the way we haven't even moved Heroku, a company that we bought 4 years ago, over to it yet.
TheAceOfHearts
Why move it if it works fine as-is? I think Waze was bought by Google in 2013 and AFAIK, they're also still in AWS.
cbhl
The PR is still awkward: "come use Google Cloud Services!" while it's not good/easy enough for your acquisitions to migrate to it from AWS.
anthony_franco
Even Amazon.com isn't fully reliant on AWS. They're not using Route 53 for their DNS (or even for their backup DNS).
watson
They will never be able to move Heroku of AWS. Their customers depend on being on the same cloud as other service providers
elmin
The location of instances could be an option. I imagine if their locally-hosted instances were a bit cheaper they would be the more popular option.
nzadrozny
> I wonder why they don't tell that AWS is their service provider.

It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage.

So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."

cracell
That's just stupid. Heroku rarely gives a proper technical explanation of their outages and under reports the length and severity of them drastically.

I assume this is to maintain their SLA. We really need independent third parties to record uptime for SLAs instead of trusting hosts to do it themselves.

This outage may be the last straw with Heroku for me. They've also previously stated years ago that they would end their dependance on AWS East and yet today shows that obviously hasn't happened.

notatoad
Not that i think there's anything wrong with heroku's messaging here, but "we're down because our service provider is having issues" isn't really a "the buck stops here" sort of update.
jgwmaxwell
Interesting that the outage in us-east-1 is causing huge issues for Heroku in eu-west-1 too - obviously a region for Heroku isn't self sufficient.
watson
Where do you see that it's causing issues for Heroku in eu-west-1? I have a lot of dynos in that region and I don't see any issues
robotfelix
Usually Heroku mentions a specific region in any status updates that don't affect all regions. For example, a status update 11 days ago stated "We're currently seeing increased routing latency in the US region. Investigation is underway."

AFAIK this outage only affected dyno restarting (which may have been triggered by a number of reasons) and creation of new dynos. Perhaps your EU dynos were lucky enough to not have done either of these things during the outage?

georgeott
"Until this incident is resolved, you might be unable to open new support tickets with us."

Maybe they should host the ticket system on a separate provider?

nasalgoat
This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in.

At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

jroid
You still depend on some provider. Or, are you talking about multi cloud installations ?
alexbilbie
DynamoDB now support cross-region replication [0] so you can build more resilient applications with it

[0] http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

andrewchilds
I don't think cross-region replication would've helped in this case:

"The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."

abalone
You could at least fall back to a read-only mode. That could be very helpful, compared to going down completely.
dialtone
You can relatively trivially build multi-master cross-region replication in DynamoDB by using kinesis and writing to kinesis instead of DynamoDB directly. On the consuming end of Kinesis you then fan out to all the DynamoDB (or whatever other database you want to use) regions[1]. Admittedly this only works with some relatively relaxed constraints on the latency you can see, intra region latency can go up to 1 second although rarely, while cross-region is around 3 seconds. An important role is also played by the structure of your objects and how accepting they are of concurrent updates coming from different regions (which is the main reason why the default replication in DynamoDB is not multi-master).

[1]: http://tech.adroll.com/blog/data/2015/06/26/kinesis.html

rue
Alternatively, run a fallback data store.
njharman
Source for service with better uptime than Amazon?
mattkrea
I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc.

This prevents you from getting too locked in and losing control over your application.

I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff.

Edit: Looks like Netflix is back online (10:53AM EST)

nasalgoat
Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services.

I mean, if even Netflix can't stay up during this, what hope does a startup have?

ukd1
Netflix -IS- up, and working fine. Pretty amazing, imho.
[deleted]
(comment deleted)
georgeott
Pocket just came back up also. Issue might be resolved?
ams6110
I was watching Netflix this morning and a number of titles had troubles loading repeatedly. They did eventually work though.
stickfigure
This is silly. Do you think you and your staff will maintain your homebrew RabbitMQ installation with better uptime than Amazon's team? And how much time/expense/manpower are you willing to devote to setting it up and monitoring it? And is that going to make your startup more successful than if you spent those resources developing the features your customers are asking for?
xbasic
Yes, competent people with good planning can easyly do a lot better than that.

We do serious hosting for government and company and if we had a outage like amazon had yesterday, we would loose at least half of our clients and pay heavy cash penality.

I don't understand why people give so much a break to amazon. They have one of the worst uptime.

acdha
I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having to maintain more of your infrastructure directly.
speleding
In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)
acdha
AWS sign-in appears to be working but in general it's true that you need some sort of automatic failover. However, isn't that true for every option other than accepting downtime? It doesn't seem like an AWS-specific challenge other except at the implementation level.
sitharus
One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently.

You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.

zurn
There are many high profile companies down (airbnb, IMDb, tinder, ...) so apparently this is not so straightforward.
peterjancelis
I don't see airbnb or IMDB being down. If there's some advice on what small time apps (like mine) can do to get thing up again sooner, please let me know.
cguess
Explains the weird 500's i was getting when trying to do a review this morning.
gshx
Running multi-region services has associated costs. It's also non-trivial engineering effort to convert most services to a multi-region setup. It's definitely worth the investment though. Another option is to run multi-region on the same provider (like AWS) without using proprietary services like DynamoDb but maybe something similar like Cassandra.
mark_l_watson
I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions.

Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also.

To be fare to Amazon, they tell you you architect and built across availability zones.

hox
dynamo db is affected in the entire region, however, not just a single AZ.
DinkyG
That's why parent post mentioned multiple regions. No other AWS region was affected...
jbeda
Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html

This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.

onetom
I would highly recommend http://www.datomic.com as a database for multiple reasons, but one of them is that fact that it's totally portable between DynamoDB, PostgreSQL, MySQL, etc.

You can create a Datomic backup from any of these databases and restore them into a different one with the exact same semantics.

jsprogrammer
Isn't datomic closed source?
nadams
That's what it looks like. And after 10 seconds of reading I couldn't tell what it actually does.
onetom
It is closed source, just like DynamoDB.

However - unlike DynamoDB - it has a quite usable free version and an even better starter version, which are good enough for production use. (The DDB local version is really only good for development) By production use, I mean for example a non-replicated MongoDB/PgSQL/MySQL, which is what I see from many smaller companies... They would be already really better of with Datomic.

If you really need more features, like high availability and transparent query caching with memcached, then the cheapest paid version has a one-time cost of $3000: http://www.datomic.com/pricing.html

You can easily save that amount of money by not spending 3-6 months developing tons of unnecessary queries, try to maintain that code and try to optimize its performance for scale...

On the "open-sourceness" note, I feel obliged to mention https://github.com/tonsky/datascript which IS open-source, and has a Datomic-like interface, but it's "just" an in-memory implementation.

But we are getting off-topic here I guess...

dialtone
This really is the first major failure of DynamoDB since when I started to use it, when it was in beta over 4 years ago. Not sure many companies would be able to run a multi-million QPS scale database with such a great availability record. In any case we just moved traffic away from us-east-1 and into us-west-2 and all is fine still so you should design at least your critical systems for regional failures.
faizalzakaria
Kinda agree, same goes to AWS tools, Opsworks screwed us up back then due to AWS outage, and decided to decouple as much as possible from Opsworks.
cle
How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it makes a lot more sense to embrace whatever cloud provider you're using (provided you made a good choice) so that you can focus your efforts on your product and not on maintaining infrastructure.
kordless
> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure

No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon).

I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.

rdtsc
> wouldn't it take you that long to switch providers anyway?

It might, or might not, depends how you built it.

> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure.

And it still was out for hours... You might see it as a great success, but you can also interpret it as "what good is it, even if with so many people behind the scenes they still failed for hours...".

The problem is that because that services ends up being embedded in so many services and products, half the internet ends being down. Even Amazon dogfoods their own stuff so services rely on each other. Dynamo is down, maybe SQS will be down or analytics as well.

[deleted]
(comment deleted)
DinkyG
DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.
drendorx39
DynamoDB is literally a garbage. That's why Amazon does not provide any SLA for the service... Even cheap Azure Storage provides cross-region failover.
Ixiaus
Garbage you say? Dynamodb blazed a trail for many open source, eventually consistent kv databases. Certainly not garbage.
drendorx39
The only thing DynamoDB can do good is simplicity. Except for this, even MongoDB has tons more features than DynamoDB and the new version resolved performance problems existed in the previous versions.
icefall
Apples and oranges. People who call tools like this garbage fail to evaluate trade-offs at their required complexity space. Distributed systems have extreme trade-offs. More features => more bugs.
istvan__
MongoDB really? You are comparing a software product to a multi-region global data service? This is just not a great comparison. If you build a distributed global data service on MongoDB you could compare that to DynamoDB.
halayli
I have a feeling you have no clue what you are talking about, nor read dynamo db paper or tried to use it in a production environment.
mdnormy
I hate the fact that most people(including me apparently) still assume AWS is not in their "downtime" equation. Just spend the last 30min troubleshooting SMTP auth problem.

Not funny when its Sunday.

antaviana
I also noticed the random auth problem in SMTP and my knee jerk reaction was to google "status aws"...
interesting_att
You're not alone buddy. Been wasting hours of my life looking at this stuff too :)
bsdpython
Between AWS, Google, Apple, Facebook, Twitter and like I doubt I am alone in spending a ton of my time working around their various issues to run my tech stack.
janson0
SQS is the specific service giving me a ton of trouble right now. Hope they resolve this quickly. Had rayguns about sqs all night heh.

So are they saying they are throttling SQS because of the DynamoDB issue?

oliverfriedmann
I'm not sure. I think many of the other services mentioned probably rely internally on SQS, so resolving the SQS issues might resolve most of the other issues as well.

Not completely sure though whether DynamoDB would benefit from relying internally on SQS.

janson0
Yeah good point. It's something I forget sometimes that AWS uses AWS... and that even if I don't rely on a particular service specifically, a service I rely on may, in fact, rely on that service.

Hopefully there is a relatively fast recovery on this.

Can anyone even log into their aws console right now?

grhmc
I can log in.

I wonder if SQS uses DynamoDB, not the other way around.

janson0
Here's the SQS Error Log right now:

3:14 AM PDT We are investigating increased error rates in the US-EAST-1 Region.

4:06 AM PDT We can confirm increased error rates for CreateQueue, SendMessage and ReceiveMessage API calls in the US-EAST-1 Region and continue to work towards resolution.

5:07 AM PDT We can confirm increased error rates for CreateQueue, SendMessage and ReceiveMessage API calls in the US-EAST-1 Region. As we work towards recovery, error rates may temporarily increase.

6:06 AM PDT We can confirm significantly increased error rates for CreateQueue, SendMessage and ReceiveMessage API calls in the US-EAST-1 Region. As we work towards recovery, error rates may temporarily increase in error rates.

chaosmonkey
I have seen multi-hour SQS outages recently. I'm thinking of options for how we can go about preventing an application failure if this happens again.

* If adding to SQS fails, temporarily store the item on disk or S3, then add to SQS when it's back up?

* any other options?

drendorx39
Amazon CTO: We designed DynamoDB to operate with at least 99.999% availability :D
ryanfitz
I've been using DynamoDB since it was released. In over 3 and 1/2 years of use, this is the first time I've experienced DynamoDB being down.
awscat
If down for longer than 18 minutes then they missed "5 9s" availability (.3 hours / 3.5 years). Not that it is supposed to work that way.
yeukhon
Software bug caused downtime vs infrastructure / hardware availability uptime to me are a different guarantee. I am pretty sure someone did something recently to DynamoDB.
toomuchtodo
Infrastructure guy here doing this for 14 years. Downtime is downtime. You get a pass if its "scheduled maintenance" you've notified your customers about to allow them to be prepared, but if you silently perform maintenance and it goes to shit, you've just counted against your metrics.
yeukhon
Nope. I still disagree. No service can guarantee 99.999999% unless you discount software upgrade. You just cannot. If you think those nines include software upgrades, you are probably over optimistic.
toomuchtodo
> No service can guarantee 99.999999%

Don't advertise it if you can't offer it then.

> If you think those nines include software upgrades, you are probably over optimistic.

If you advertise a product with a specific SLA, and you can't meet that SLA, you're a liar. Don't try to blame the victim because of inaccurate/untruthful marketing or engineering.

yeukhon
Maybe you are the one who needs to understand their SLA and fine print.
antod
SLAs are just contractual thresholds for getting some specified redress if not met. They are not promises.

Not meeting a SLA is not lying.

toomuchtodo
I used SLA to communicate an advertised/marketed level of service. In this case, I agree, that SLA is the wrong term as there is no contractual agreement.
dankohn1
I noticed this because I was unable to checkout on Amazon Prime Now just now.
pgrote
I noticed it when I couldn't stream something. The player begged it off as Silverlight issue even when the flash option is chosen. lol
JOnAgain
Le sigh. This is impacting AirBnB and I need to check in somewhere in LA later today. Good thing all the details are in the AirBnB messaging history with the host. Time for them to just go back to email.
neals
I'm somewhat in the same situation. I want to check-in a movie I just watched and give it appropriate rating, but IMDB is down. I guess we'll just have wait, right?
clebio
Any knowledge or evidence that IMDB runs on AWS, and that the two are thus correlated?
[deleted]
(comment deleted)
nhumrich
Well, Amazon owns IMDB, so it's probably a reasonable assumption.
jspaetzel
Quick lookup says yes, # host imdb.com imdb.com has address 207.171.166.22 imdb.com has address 72.21.210.29 imdb.com has address 72.21.206.80 # host 207.171.166.22 22.166.171.207.in-addr.arpa domain name pointer 166-22.amazon.com.
pavel_lishin
Doesn't AirBNB also send an email copy of correspondence? I've always received emails (and text) when a host contacts me.
Hughlon
Videos and Alexa ia also down
eatonphil
This appears to have completely blown away all Alexa data. Even searching for google.com returns nothing.
vreauobere
other down sites: medium.com, getpocket.com, idonethis.com
[dead] vreauobere
other down sites: medium.com, getpocket.com, idonethis.com
geertj
Address verification on amazon.com doesn't work for me at the moment, blocking me from making any orders.

Not sure if this is related.

SoulMan
Nothing should be effected in non- USEast regions as per the status page
kureikain
Whoever uses autoscaling and, especially lifecycle notification with SQS will be in trouble now(I'am).

The morning is going to be started. Traffic will be ramped up, and not sure if new sevrers will be launched because CloudWatch is failed. Polling SQS to find lifecycle notification message fail too.

tcas
We were in the middle of a large infrastructure change starting at 4:30am this morning, including taking our application offline. I'm very thankful that we did dry runs along with timing how long certain operations like RDS restores should take and planned for abort steps in case something goes wrong.

We noticed that RDS and ElastiCache backup and restores were taking much longer than expected, and once the first set of errors about Dynamo DB came in we decided to abort and try it again at a different date. An hour later we got notifications that RDS was having issues as well. I'm disappointed that it takes so long to update the AWS status page when things aren't working properly.

confiq
similar story here... They status.aws page has serious delays
tnolet
docker, wercker, travis.ci are also affected. Can't login or stuff is really sluggish.
crypt1d
Audible seems affected by this as well. I've made some purchases with my credits but the books are still not showing up in my library...and the checking out process is very slow.
edanm
FYI happened to me as well, but is resolved now.
sauere
Tinder is down due to this, now my life is pointless.