Ask HN: Best setups to avoid availability outages on AWS

I don't think they show how to do ELB across Regions, or diversity against single-ELB problems (although I haven't seen ELB fail yet). You'd probably have to build this yourself.

Anyhow, sorry I don't have a simple answer - I'm not sure a simple answer exists.

I'm in one of the us-east zones and I haven't had a failure in at least a year. They retired one machine I was using and dealing with that was as simple as starting and stopping -- at a time I chose.

With five zones in U.S. East, the probability of a zone failure affecting a single zone systems is 1 in 5.

If you're a busybody who spreads your system across five zones, the probability of a failure affecting you becomes 1.

You're spending more money, and dealing with a lot more complexity, all to increase the probability that hardware failures will affect you.

Now, you're hoping that a zone-distributed system will be able to recover from failures, but that's tricky to do and it's quite unlikely that this will work if you haven't tested it. Add the fact that all the other "cool kids" will be trying to recover their systems at this time and make AMZN's control plane go down.

In the meantime, with probability 4/5 I'm sleeping through the disaster and the first time I hear about it is on hacker news.

1) Has AutoScaling Groups & Elastic Load Balancers in two regions (and only two availability zones; let's keep front-end instances in the same AZ as your local/region-specific DB)

2) Has Databases in two regions and uses Master Master replication

3) Instances talk to their local DB. If they detect their local DB is down, they failover to the remote DB (ie, the far region). If they failover, they notify you.

4) DNS does geographic load balancing (pre-ELB). You'll need to use a provider like DynDNS or UltraDNS to give you Geo Load Balancing & Failover. Or, you could pair a monitoring service like CatchPoint with Route53

5) Application caching (Memcache, Redis, etc). Let's not put more load on the DB's than necessary.

That's a good start, at least.

PHPFog are on AWS and I pay them to make sure they have the redundancy worked out. If they don't, I would yell at them until I got some money back.

I am considering configuring RDS for Multi A-Z, but need to research it a little more first. From what I can tell you just click a button to turn it on, but there were a lot of people complaining yesterday that the fail-over didn't work at all when it was supposed to.

I also have a bunch of EC2 VMs that do back-end processing and have a load of CRON jobs on them that need to run once every 24 hours. If these go down for a couple of hours then there is no noticeable impact to my customers, they can still log into my service and access their historical data.

I have considered spreading across multiple regions etc but at the end of the day it's just too expensive for the small increase in reliability.

Richard,

AWS is an amazing tool and you have a few options here, but the downside is the more options you use to be highly-available (HA), the more expensive AWS gets (as you would imagine).

Your first option is to be HA across a SINGLE region; to do this you make use of the elastic load balancers (ELB) + auto-scaling. You setup auto-scale rules to launch more instances in different availability zones (AZ) either in response to demand or in response to failures (e.g. "always keep at least 3 instances running").

You compliment that with an ELB to load-balance incoming requests automatically across those instances in the different AZs. This is all fairly straight forward through the web console (except auto-scaling is still done via CLI for some reason)

If you want to be HA ACROSS regions you can't just use ELBs anymore, you have some added complexity and an additional AWS feature you will likely want to use: Route 53.

Route 53 is Amazon's DNS service which offers a lot of slick DNS-features like removing dead points from DNS rotation, latency-based routings, etc. There are also something like 29 deployments of Route53 (and CloudFront) around the globe so you'll hopefully never have Route53 become a point of failure for you even disaster strikes.

In this scenario you would setup the HA configuration for a single-region as mentioned above, but you would do it in multiple regions. Put another way, 2+ servers in multiple AZs in each AWS region. Then a Route53 DNS configuration setup to point to each ELB in each region representing those individual pockets of servers.

Ontop of that you would use Route53 to manage all routing of client requests into your entire domain; you can leverage the new "latency-based routing" (effectively why everyone was asking for GeoDNS for years, but even better) and monitor capability to ensure you aren't routing anyone to a dead region.

SIMPLIFICATION

--------------

Here is what I would recommend given the size of your budget and need to stay up in the AWS cloud, in-order of expense:

1. Launch a single instance in a region with acceptable latency that has never had an outage before (e.g. Oregon has never completely gone down but Virginia has -- yes yes I know VA is older, but you understand my point). This solution will be cheaper than multiple instances in any region.

2. Launch multiple instances using the web console, in multiple AZs in US-EAST (cheapest option for multi-instances) and front them with an ELB. You skip any auto-scaling complexity here but you need to keep an eye on your servers. I think ELB fixed the issue where it would effectively route traffic into the void if all the instances in an AZ went down.

OPTIONAL: If you didn't mind spending a few $ more, you could do this strategy in the region that has never gone down for added piece of mind.

3. Launch single instances in multiple REGIONS and front them with Route53. This isn't really a recommended setup as entire regions will disappear if you lose a single instance, BUT I said I would list possibilities in order of price, so there you go. You could mitigate this by setting up auto-scaling policies to replace any dead instances quickly in the off chance you wanted to do exactly this but not babysit the web console all day.

4. Launch multiple instances in each region, across multiple AZs fronted by ELBs and then the entire collection fronted by Route53.

NOTE: The real cost comes from the additional instances and not from Route53 or the ELB; so if you can use smaller instances to help keep costs down (or reserved instances also) that might allow you to provide a larger HA setup.

What about my data?

-------------------------

Yes, yes... this is an issue that someone already touched on (data locality below).

You will have to decide on a single region to hold your data; in this case I would recommend using DB services that aren't based on EC2 and have never experienced outages (or rarely) -- this includes S3, SimpleDB and/or Dyna...

Sorry to be fatalist, but it's a hard problem. This last outage was more than just an AZ failure. Region-wide API usage was affected, so operations like static IP reassignment and ELB changes were not taking effect. This means you are hanging out in the wind should there be something unusual that requires manual intervention (as was the case with us).

Route 53 is a good service but I don't know how its control plane works, and it could be that problems in a single region would disable the ability to update DNS records (I would guess that DNS reads are a lot more available than writes). And in any case DNS is not a very good failover mechanism due to upstream caching.

Unless your business model requires higher reliability than Instagram, Netflix, and Pinterest, I'd suggest going multi-AZ, crossing your fingers, and doing everything else right.

Going beyond that at a cost of slow response times when trying to access a downed backend, you could deploy back end web services to two different hosting providers, perhaps running something like CouchDB replicated on each provider. The Javascript on your UI could switch to an alternative back end after a timeout. For "one page" style apps, you could maintain the state information that a backend host is down in the browser.

1. Use Rightscale. You can get away with the free edition, but for $500/month the basic paid edition will allow you access to arrays and all the excellent scripts available on the marketplace.

2. The front end. I would strongly suggest moving away from ELB. We are using it and are about to get rid of it. The main problem is what exactly happened last night. If a whole AZ goes down, the ELB for that zone can get screwed and the DNS was not updating the CNAME to remove the bad zone. Instead of ELB, we have our own LB solution we are going to roll out that will use Rightscale server arrays and will handle the updating of the DNS names itself. We also aren't going to use Route53, because we learned last night that the API for that can go down and you can get stuck with bad DNS records.

3. Application servers. Use at least 3 AZ and have them evenly spaced. This is easy to do in Rightscale with sever arrays. Make sure your voting ration for scaling isn't 50% because you might not scale correctly if you loose 2 AZ. Keep the vote to 30% and you will be happy (if one zone votes to grow, let it grow).

4. Database. This is the fun one. We have been using MongoDB with pretty good success. Our multi-shard DB has 3 servers per replica set and has them distributed equally between AZs. We use 4 EBS drive RAID-0 drives for storage which have had problems in the past due to the outages that EBS sometimes has. Our best bet has been a watcher process that will kill the mongod process if there's any problems writing to the drive array. By doing this, the replica set will automatically failover to the next server and we won't get stuck with a primary node that can't write back to disk. For backups, we just freeze the writes on the secondaries and do EBS snapshots even 15 minutes. Rightscale has some great EBS tools for managing this for you. If you loose a server, we can deploy a new server in a matter of minutes and it will rebuild the RAID array from the last backup so we have a warm spare.

5. Monitor, monitor, monitor. Rightscale has some great tools for monitoring everything. Use them, and use more monitoring on other infrastructure (ie Pingdom)

Doing something like this will cost a lot more that just sticking to a single AZ, but you should be able to survive one, if not two complete datacenter outages.