This started a little before 01:00 PDT (08:00 UTC), so we're approaching the 3 hour mark now. FWIW, that's about 1h15m before there was any sort of indication of a problem on status.aws.amazon.com
We are going to set this up for our app, however there's a limit to how useful it is when you are reliant on services such as MongoHQ/PubNub (in our case) that are also affected!
Well, it's not going to be cool or best practice to use cloud services for your startup for the next week or two. Fortunately people on HN will then forget until the next AWS outage and your setup will be fine again.
Duplicating across regions is not well supported by AWS unfortunately. Any idea doing that without completely rebuild the infrastructure in an other region?
I assume you are using the AWS dashboard to manually deploy instances and setting them up by hand? First step is to get your infrastucture to the point where it deploys and scales up and down by itself. Once you manage that, moving to or keeping hot spares in another region is pretty easy.
Look into automated deployment tools like Foreman and configuration management tools like Puppet.
We are learning the hard way that the EC2 Availability Zone separation is not enough, and EC2 is lacking some key features offering multi region tools.
Thanks for the hints, I'm going to check them out.
I echo this... although I don't personally use Foreman. You can do scaling type stuff without the tool and I just use POP (plain old puppet). For me personally it took longer to get on the config-management bandwagon but once I did I have never looked back.
Learning to use Puppet, Chef, Salt or similar will only bring benefits!
Have scheduled tasks to create AMIs of your infrastructure. Create scripts or redeploy by hand into another zone when an extended issue has occurred, or deploy and leave powered down as part of a 'DR' plan or something.
How would you recommend performing that failover, when Elastic Load Balancers only cover a single region? If you host your own load balancer it'll have to be hosted somewhere, and DNS failover seems unpopular [1].
DNS failover is fine, even if its manually triggered. We are almost two and a half hours into this network event, so even if you set a 5 minute TTL on your records and it takes you 15 minutes to respond, you are still way ahead of the game.
The idea that some DNS resolver somewhere caches records longer than TTL is mostly a myth. This is the case rarely on some mobile networks and networks behind satellite uplinks, but they generally have larger network issues anyway.
This got me wondering for quite a long time: I'm pretty sure almost all EC2 issues happen on the US east region -- why is that? Is it because it's the most used region?
Think so... it's the cheapest, and is the default choice. Sounds like it would have an order of magnitude more usage (and therefore problems) than the other regions.
Some of my us-east-1 servers are affected, others are happy so far, so it's not the entire datacenter. However, connectivity on affected servers has gone from just flaky to completely gone.
Had a bunch of timeout alerts. Some machines of an application server array have the packet drop issue. ELB says that everything is peachy. Folks, we're experiencing yet another "EC2 flavored SNAFU™".
Initial though was: f#*k, I ran out of network I/O, since EC2 simply states "low, mid, high" as performance specs, therefore some proper planning is out. Turned out that with load avg under 0.02 on all machines, the I/O wait wasn't to blame. Average response time, per Pingdom, went up from 170ms to 450ms. New Relic isn't happy either. I guess we should all thank Amazon. Again.
I see the same thing. Pingdom reports higher response-time, but no downtime (meaning no alert). Also no alerts from AWS Cloudwatch. I first became aware of the issue when internal api-tests started failing at 9:56am CET. I see users accessing the site, but I don't know how many it's failing for.
No issues in the internal EC2 network. At least, none that I could find. I guess that's the reason why ELB doesn't shift any traffic. The whole issue seems to be on the Internet facing network. Failing routers, maybe.
Pingdom still claims 100% uptime, but New Relic (which includes an equivalent pinging service) reports downtime from time to time. Around 25 timeout alerts into the last couple of hours.
Health Dashboard updated the status to "Resolved":
5:17 AM PDT Between 12:51 AM and 4:52 AM PDT we experienced elevated packet loss affecting instances in the US-EAST-1 region. Some of our APIs also experienced increased error rates and latencies. The issue has been resolved and the service is currently operating normally.
It does, but not for nefarious reasons -- it (until the more recent history) the cheapest region in the world for AWS -- it was only right before Oregon rolled out that Ireland, US-EAST and US-WEST-2 all became the same price point, but for the 4 years prior to that, it was always the cheapest so that is where most customers rolled out most of their infra.
Now that the prices has normalized I think the load is distributing more evenly, but for historical reasons I think that region sees a lot more churn (starting/stopping/deploying/etc.) -- just more grinding on the hardware at that region that others.
rkalla's answer is good, but there's also the issue of the US east coast being a good place to deploy applications that need decent performance to the majority of the english-speaking world. Dropping your servers there puts them within reach of Europe without as much RTT as the US west coast.
Obviously the right answer is to deploy applications into multiple AWS regions, but that's not appropriate for every service's architecture unless built that way from the start or modified specifically to do so.
In addition to the other two excellent answers, there's another reason for the apparently higher issue count:
US-EAST-1 is enormous. It's now made up of 10 data centers with tens of thousands of machines (and associated infrastructure). There are thus more moving parts, and more points of failure. While Amazon builds a bunch of redundancy into its systems, "smaller" issues will tend to impact a larger number of users in US-EAST-1.
32 comments
[ 3.1 ms ] story [ 54.9 ms ] threadLook into automated deployment tools like Foreman and configuration management tools like Puppet.
We are learning the hard way that the EC2 Availability Zone separation is not enough, and EC2 is lacking some key features offering multi region tools.
Thanks for the hints, I'm going to check them out.
Learning to use Puppet, Chef, Salt or similar will only bring benefits!
[1] http://serverfault.com/questions/60553/why-is-dns-failover-n...
The idea that some DNS resolver somewhere caches records longer than TTL is mostly a myth. This is the case rarely on some mobile networks and networks behind satellite uplinks, but they generally have larger network issues anyway.
Initial though was: f#*k, I ran out of network I/O, since EC2 simply states "low, mid, high" as performance specs, therefore some proper planning is out. Turned out that with load avg under 0.02 on all machines, the I/O wait wasn't to blame. Average response time, per Pingdom, went up from 170ms to 450ms. New Relic isn't happy either. I guess we should all thank Amazon. Again.
Pingdom still claims 100% uptime, but New Relic (which includes an equivalent pinging service) reports downtime from time to time. Around 25 timeout alerts into the last couple of hours.
5:17 AM PDT Between 12:51 AM and 4:52 AM PDT we experienced elevated packet loss affecting instances in the US-EAST-1 region. Some of our APIs also experienced increased error rates and latencies. The issue has been resolved and the service is currently operating normally.
Now that the prices has normalized I think the load is distributing more evenly, but for historical reasons I think that region sees a lot more churn (starting/stopping/deploying/etc.) -- just more grinding on the hardware at that region that others.
Obviously the right answer is to deploy applications into multiple AWS regions, but that's not appropriate for every service's architecture unless built that way from the start or modified specifically to do so.
US-EAST-1 is enormous. It's now made up of 10 data centers with tens of thousands of machines (and associated infrastructure). There are thus more moving parts, and more points of failure. While Amazon builds a bunch of redundancy into its systems, "smaller" issues will tend to impact a larger number of users in US-EAST-1.