HN Explorer

AWS increased error rates / intermittent outages

100 points by needcaffeine ↗ HN
http://downdetector.com/status/aws-amazon-web-services

https://twitter.com/search?f=tweets&vertical=default&q=ec2&src=typd

51 comments

[ 2.0 ms ] story [ 112 ms ] thread
needcaffeine
11:27 AM PDT We are investigating increased API error rates in the US-EAST-1 Region.
joshwa
11:50 AM PDT We can confirm increased error rates for the EC2 APIs and are currently working to resolve. We also observed isolated periods of impaired network connectivity for some EC2 instances.

12:21 PM PDT We have identified the root cause for error rates accessing the EC2 APIs and EC2 Management Console and are currently working to resolve. We observed isolated periods of impaired network connectivity for some EC2 instances, however running instances are currently operating normally.

tamcap
We are getting very intermittent issues with S3 and SNS so far.
mr337
Same with ECR too
0xmohit
SNS informing about down services would be too bad. So it must go down with others.
dewyatt
Sorry guys, I knew I shouldn't have started my 4 instances at once.
0xmohit
It wasn't you. Somebody had set up auto-scaling at the behest of AWS support.
zymhan
And it solved all their problems!

Because the whole thing stopped working.

cpitman
Nah, pretty sure it was my ansible script. That I was demoing. Righhhhht as AWS came down like a house of cards.
carterh062
In the console, running instances was timing out on load for EC2 and some of our AWS API calls to get hostnames are timing out in PHP
swingbridge
Ditto, can't see any instance data either via the API or graphical dashboard. Instances themselves seem normal for the moment.
needcaffeine
Dammit now my Amazon Echo doesn't work.
mead5432
Alexa? Alexa?!?!? ALEXA, NO!!!!!!!!!!
colin_fraizer
During the outage, my Echo just sang "Daisy" verrryyy sloowwwwly. Weird.
mannycalavera42
Dave, I don't understand why you're doing this to me. I have the greatest enthusiasm for the mission.
elwell
I feel sad for a lonely grandma somewhere; her grandchildren never visit, but they bought her an Alexa to talk to that Amazon couldn't keep online.
0xmohit
I can't even ask Alexa if AWS is up.
Zenfinch
Brace for half the internet going down if gets any worse, especially if its us-east (wish I was being ironic).
carterh062
Looks like it's just their service API for now, i.e not responding to any queries on existing instances for me at all. However, everything else is working fine as far as I can tell.
ed661266
API Gateway is returning 500s. So much for server-less architecture :(
dewyatt
I've had both 500s and 401s interestingly.
frakkingcylons
FYI there's a great Chrome extension that hides all the working services (green checks) on the AWS status page so you can quickly see what's down.

https://chrome.google.com/webstore/detail/real-aws-status/ka...

XiZhao
I like how half the internet can die when EC2 has problems.
mrweasel
Part of the problem is that a large number of site pull in services, re-targeting, AB-test, and weird Javascript in general. These things are pulled in without questioning or demanding an SLA or putting in an easy way of pulling them back out. Even if you don't use AWS yourself, you can be sure that some third party you rely on is deploying on EC2. Of cause they'll never tell you that.

For most new stuff, we require that it can be loaded with something like Google Tag Manager or UberTag, so we can quickly disable them when they fail.

It doesn't help that Amazons status page isn't all that good and sometimes doesn't seem to actually reflect the true state of their service.

zzleeper
I can't log into amazon.com to check my orders (first time that happened). THat's quite surprising as often these problems are unrelated
the_watcher
Amazon.com remains up... unless you attempt to buy something, when you get a 500 error.
FT_intern
I wonder how much revenue is being lost every second
tschellenbach
everything on getstream.io is still up and running. most issues seem related to provisioning more instances.
elwell
Can't log in to AWS Console. Instances running fine.
tschellenbach
does anybody have more details about what's actually down? their description is a bit vague.
elwell
Status page is down too: https://status.aws.amazon.com/

Shouldn't there be a separation of concerns for status pages, maybe use: https://www.statuspage.io/

Edit: http(s) was at fault (no SSL cert?)

[deleted]
(comment deleted)
[deleted]
(comment deleted)
elwell
Just got through to the AWS Console. Finally can get back to work.
loourr
Is anyone having issues with their lambda functions not working? Mine stopped working and is just returning "Service error." when I try to test even though they claim it's operating normally.
0xmohit
AWS would wish that all those downtime reporting services move to their infrastructure. That ways there won't be anyone to report the downtime.
0xmohit
AWS should design for resilience [0].

[0] This is a part of the wisdom shared by AWS support upon reporting an outage.

toomuchtodo
1. "Aren't you fault tolerant against AZ failures?"

2. "Aren't you fault tolerant against region failures?"

throws up hands, moves back to bare metal

jimaek
Am I the only one having daily issues with SQS? I have 150+ servers writing to a queue and they timeout at random a few times per day.
dopamean
The app I work on is a single server writing to a dozen or so queues. The throughput is pretty low but I still manage to see failures for a few minutes once or twice a week.