HN Explorer

Amazon EC2 EU-West is down

109 points by dabeeeenster ↗ HN
Nothing yet on the AWS service status but Twitter is going a bit crazy.

UPDATE 19.15 BST: now being reported as network issues by Amazon

UPDATE 19.24 BST: Can now see my ec2 instances again from the management console but can't ssh into most of them.

UPDATE 19.27 BST: From Amazon: EC2 APIs in the EU-WEST-1 region are currently impaired. We are working to restore full service. We are also investigating instance connectivity that we believe to be limited to a single Availability Zone.

UPDATE 19.34 BST: RDS and Route 53 also affected.

UPDATE 20.10 BST: From Amazon: The issues with the affected Availability Zone are the result of a power failure in that zone. We are currently recovering power and anticipate that instances in the effected available zone will start to recover within the next 30-60 minutes.

Well, that sucks.

37 comments

[ 2.9 ms ] story [ 28.1 ms ] thread
dabeeeenster
I can SSH to some of my newer instances but not the older ones.
[deleted]
(comment deleted)
pimeys
I can see my server from the management console but I can't ssh to it. Damn, my IRC is down :(
sipefree
Clear skies over Dublin, not a cloud in sight. ;)
sdepablos
I see all my servers on the management console, but I'm unable to ssh any of them, all in the same availability zone, eu-west-1b
dialtone
CloudFormation is also affected now.
stephank
The specific availability zone they're talking about is eu-west-1a. Servers we have there are reported as 'running', but are unresponsive.

Other zones seem fine, but starting new servers doesn't seem to be working.

amock
Availability zones are different for different accounts, so for me it's eu-west-1b. You can log into the console and under the EC2 tab it will show you the status of each of your availability zones.
stephank
Ah, good to know, thanks for the pointer.

Another update on the status page, btw. It appears to be a power failure.

sdepablos
I've tried to create a machine in another zone with no success. I suppose there's a lot of people doing the same :(
[deleted]
(comment deleted)
jread
I run a site that monitors over 100 different cloud services. Our monitoring node in EC2 EU West went down at 10:48:51 PDT. Our monitored application with PHP PaaS cloudControl, also hosted in EC2 EU West, went down at the same time. For the past year, we've gotten 99.978% availability in this EC2 region: http://cloudharmony.com/status
[deleted]
(comment deleted)
jimmyjim
Power outage?

Interesting -- Linode also went down today because of power outage: http://status.linode.com/2011/08/outage-in-fremont-facility....

bradly
Is there any chance these power failures have something to do with the solar storms we are experiencing right now?
whakojacko
This is what, the 3rd? 4th? power outage in that facility in the last year? I realize that it's not directly Linode's fault, since this is Hurricane Electric's facility that they lease from, but its still pretty frustrating. I have a personal VPS there which I use to do an hourly data grab, so I missed a few this morning. This is just a personal pet project so no huge deal, but if I was running some actual production-level stuff there I would be very wary of using that facility.
getsat
Linode doesn't notify their customers affected by outages, or am I missing something?
Macha
I've been notified for what amounted to two minutes of downtime before (my VPS is in the London data centre). Are you in the Fremont data centre? That is where this outage was.
getsat
Yes, I have a machine in Fremont. How were you notified? Email?
Macha
Yes. An email about unexpected downtime, then another two minutes later saying the downtime was over.
getsat
Hmm, I don't see any way in my Linode account settings to disable these emails, and I never received any! That is disconcerting. :(
[deleted]
(comment deleted)
puredemo
So that's what interrupted my screen session.
bambax
THANK YOU HN!!! I couldn't connect to my machine and couldn't understand what was going on... good thing I turn to HN as soon as I get online...

You'd think AWS could afford to send emails to keep us customers up to date with the problem?

rwolf
If they sent you an email every time there was downtime, then you would know every time there was downtime. Any time your servers are not available but you don't notice == win.
bambax
> Any time your servers are not available but you don't notice

Can't happen. My servers are monitored by external services that (to the best of my knowledge) don't use AWS.

It's best to play the open card in any case.

spartango
Emails will be sent out when the problem is more fully handled, as usual. HN is a decent place to get some info, but you should consider posting to the AWS developer forum as well, b/c there are Amazon people constantly checking on things and answering questions:

https://forums.aws.amazon.com/forum.jspa?forumID=30

Edit: I should add, they can look up your specific instances to get the exact problem.

[dead] trdlo
Amazons need to put some solar panels on their roof
chaz
A must-follow Twitter feed for anyone using Amazon AWS is @ylastic: http://twitter.com/#!/ylastic

At the very least, even if you don't use Twitter, it's a good resource to check for up-to-the-minute info if you're seeing an issue and want to verify.

calpaterson
If anyone is interested, ServerDensity thinks that our machine went down between 6:37 and 6:42 BST.
rmc
As an AWS customer affected by this, that's the time our servers went dark.
zizzler
Has anyone gotten any timeframe when the rest of the RDS is going to start responding? Not much info on the status page of AWS.