Tornadoes in the midwest and earthquakes in the west. You're doomed.
Seriously though, as horrible as downtime is, I think most internet users aren't terribly surprised when they can't go to a specific website for a short period of time.
I'd be interested in seeing someone polish this turd. How would you spin amazon being down into good press when people are looking for reliability. Only thing I can think of is people will be writing about you and that should increase your page rank but I doubt amazon is concerned about higher page rank.
Anyone else care to speculate how this could be good press?
It could be an opportunity to explain what reliability actually is (it's not "pack one availability zone with all your stuff") and how AWS helps you achieve that.
For ecommerce, people will just got buy somewhere else. So not to big a deal for them but a really big deal to your bottom line. Especially when you are still paying for traffic acquisition, sure you can turn Adwords off in minutes but other traffic sources may take 24 hours.
If you're willing to go outside AWS...Denver. We occasionally see tornadoes out here, but Denver is outside "Tornado Alley" [1], so they're pretty rare. On the WikiPedia tornado page they set the odds of a tornado hitting the downtown area of a big city at around one in a thousand per year.
If you want to be extra paranoid, you could always pick a secondary data center location where it would be safer in the summer, since Denver would be safe from tornadoes in the winter. [2] Or you could just put one on the West Coast and hope that there's not a 1-in-a-thousand tornado within a week of a big earthquake.
I wonder how much of Google's reliability comes from service failover and redundancy and how much from very reliable datacenters etc. I'd find it hard to believe that their platform and DCs aren't better than what we have seen from AWS, which could make compute engine a very attractive product.
A lot of it comes from automatic failover. Individual data centers have issues frequently, but if you're using High Replication Datastore then you won't notice it much, apart from occasionally seeing all your instances getting killed and restarted in a new datacenter, which also results in memcache getting reset.
I can only talk about my experience, but I've had zero downtime with GAE since we migrated to the HR datastore. Development is x1000 harder, but then everything Just Works.
App Engine's reliability massively improved with the HR datastore, and has gotten even better since the pricing change / SLA guarantee. It's actually remarkably good now, I recommend taking another look.
The status page seems to really underplay the severity of the situation. Netflix and Heroku are down, yet these are just side effects of 'performance issues' instead of a 'service disruption'. I wonder what it would take to cross that threshold.
AWS has historically been both slow to update and heavily optimistic with their status page.
When I got the frantic texts when EC2 first dropped offline, sure enough, the AWS status page was all green, but twitter was alight with people talking about it.
I suspect a service disruption would have to be Godzilla.
When our EC2 hosted website seems offline and AWS status page is all green, the first thing I do is check twitter. Usually you can see people asking about outage before amazon acknoledge it.
Is this not a 'service disruption' situation? At the bottom of the page, the yellow icon is associated with 'performance issues'.
If there's one thing that's shocked me about AWS, it's the total failure to acknowledge the severity of service disruptions. Like the above case, or the fact that a 3-hour loss of connectivity is displayed on the service history as a green tick with a small 'i' box: http://oi46.tinypic.com/x5qtch.jpg
Or that there is absolutely no way to deep link to an ongoing outage, and users must reload, then expand the link every single time, or subscribe to an RSS feed.
AWS needs to blantantly copy Heroku's status system, which is worlds better for people needing fast updates on their infrastructure.
What's the deal with the % measuring? I know going below three nines is seen as "bad service", but three hours of outage in a month gives you 99.5%, not 99.97%
They've reporting a power issue with a single US East availability zone. There are four EC2 availability zones in US East. Strange that they would cite orange as "performance issues", but it's certainly more appropriate than suggesting a complete service disruption.
It is absolutely affecting more than one availability zone. Just because people didn't lose boxes outside of one AZ doesn't mean there weren't service disruptions outside of it. I know multiple people, myself included, who STILL cannot launch new instances anywhere in US-East. The EC2 web console and API have also been down intermittently and throwing errors for hours.
1 out of my 6 instances in the region was affected. It took me about an hour to get it back up (which was me clicking once and waiting). I wonder how much of that is fallout from instances being sent to other availability zones and bottle-necking.
If one availability zone is down, there will be suddenly be ton of extra people trying to start new instances in the other availability zones. Could there just be a shortage of available instances in the other availability zones? Perhaps Amazon don't keep enough spare capacity for this eventuality (nor would I expect them to)?
This certainly can happen. You can't rely on being able to launch enough replacement instances if a zone fails.
You can at least mitigate it by keeping utilization of your instances below what would be needed to handle load of one zone failing. I suspect to save money, many people are pushing utilization as high as possible.
If an outage in one zone causes a brownout in the other zones, then the aren't independent "availability" zones in any logical sense. They may as well be on the same power supply.
For embarrassing outages, sure, but have you ever had to try and find out why some of your instances have disappeared? Good luck with that. Gold support, which you'd think would be good, just prioritizes your emails, apparently little more.
Linode, in comparison, has always been straight-forward, personal, and as honest as a company in that space can be.
What's perhaps odd is Amazon's customer service for their material goods is usually superb.
They do have a red icon if an event impacts an entire region. If a customer is correctly utilizing multiple availability zones, a failure in one zone should only impact the customer until they can fail over (Should be within minutes if they are automated).
Aren't multiple availability zones impacted though? I don't get how Heroku can be so widely impacted if its not for the fact that multiple Availabiltiy Zones have been impacted.
The power issue only struck 1 of the 5 zones in US East (Virgina)
I can't speak to why Heroku was impacted as much as it was. It could be that they have single points of failure, or run at a utilization level that makes losing a single zone difficult.
But the icon is meant to display AWS status, not the customer's. Friday's outage did warrant a red icon - unreachable instances, corrupted EBS volumes, failed API calls.
We had 0 downtime. The only thing that's screwed up is a read replica of a multi-AZ MySQL on RDS deployment. Amazon did not send any notification. Kinda annoying.
There's another comment thread going over at (http://news.ycombinator.com/item?id=4180339), if, like me, you got extremely lucky and picked today's lucky availability zones, and have time to read HN instead of scramble to get things back up.
unfuddle is down also, can't access any of my repos. I was going to spend the day working too - oh well time for a long lunch.
This isn't the first time this has happened to AWS - we moved our app to linode last year after this happened to us and it seems to affect AWS more than any other hosting i've ever used, i'd be interested to know how their infrastructure is set up because it doesn't seem particularly robust.
Be careful. Nothing is as it seems right now. Do not trust any API output, nor should you do any API operations that are non-recoverable. Things are up that are reported down and vice-versa.
Wait for the dust to settle. We're all just going to be a bunch of Fonzies here.
EDIT: Looks like API access has been restored, so I'm cautiously optimistic about things working. Note though that some instances may have rebooted or be otherwise impacted so check your error logs.
EDIT2: Nope, ELB is still hosed. Continue to be skeptical.
Site failover is one of the exercises that are left to the user. Considering how relatively bare-metal an experience AWS provides, this has never surprised me.
Are you suggesting that AWS should operate out of more than one location?
It's a wonderful idea!
I even have a list of possible locations they should look into. Beyond the Virginia site, they should be looking at DCs in Oregon, California, Ireland, Singapore, Tokyo and even Sao Paulo. What do you think?
I got a crazy idea, what if I build my own cloud on top of AWS to abstract away all this extra complexity of building redundant sites across geographically distributed datacenters? That way developers don't have to think about hardware at all!
Latency to Europe kind of is, if you want to be in a single site with acceptable ping times to Europe, US East, and US West. But I'd rather just solve replication and be in each.
Because distributing sites across multiple regions is hard -- high latency between your front end and database sucks, and replicating a database over high latency is also hard. If you can do it, great, but if you can't, then pick the best single region.
US East or US West plus Direct Connect to your own colo space, with AWS for the burst capacity, and your own redundancy for the database servers, might be the best plan if you can't do wide area "over the Internet" database replication. (I might get an extra 10G DC (since I need 1G DC myself) and then have some colo for sale with it in US East/US West later this year.)
My appologies. I still believe it's wrong to call it that way because it assumes too many things. Another way to word it is Force Majeure. http://en.m.wikipedia.org/wiki/Force_majeure
No, it isn't another way to word it. Force majeure includes acts of nature _and_ acts of man. Acts of God is only acts of nature.
It's somewhat important to the original spirit of the comment since Acts of God might indeed happen anywhere (1) whereas acts of man might not. For example, disruption caused by on going war in Syria wouldn't be covered by an Act of God clause.
(1) I think that's BS though...there are definitely some places where nature is considerably more stable than others.
EU-West had similar levels of outage last year due to a lightening strike. Twas out for several hours and took a few days for everything to be back to normal.
I commented along the same lines during the last AWS/Heroku outage, but Rackspace still is giving me amazing value and uptime, and every time I try to move away (as I did this week with my lastest project, on Heroku) I get hit with a massive service disruption that pushes me back to Rackspace.
Complete curiosity. I love Rackspace, but am always looking to try new things - new languages, new databases, new methods of hosting, new DNS, new foods, new music - I'm just a curious person. It's not problem with Rackspace; all of my mission-critical stuff is still hosted with you. But I use random projects of mine to to explore in multiple ways (including Black Sands by Bonobo, which I discovered last week - possibly my favorite coding album now).
Not an album for coding per se, but if you like Bonobo you might like Pepe Deluxe's new album Queen of the Wave.
It's my favourite new album of 2012. Retro psych rock and as mad as cheese.
If you're that price sensitive/need big RAM and can use RIs you should really be looking at dedicated servers, too. (for which Rackspace is expensive as well, but actually provides a service level to justify it in many cases)
I disagree. The markup does not simply justify cheery service folks. I want knowledgeable. When they don't have support for Nginx, to me, that's not knowledgeable.
Rackspace prices are insanely high and I can't wait to move off of them.
I'm ok with a dedicated server provider being good at the physical and network without much focus on applications on the host. You can use a third party for host level sysadmin, but you don't have a third party choice for the infrastructure.
That's a crazy amount to be paying a month for under $500 worth of RAM (2 x 16GB DDR3-1600 registered ECC for example). As the other reply mentioned, you should look into dedicated servers or colocation if you don't mind dealing with hardware yourself.
Interesting, are you using any cloud management platform (e.g. RightScale)? They'll make it simpler to migrate.
If you're scenario is more complicated than a single server, you might find our tool useful to forecast your costs: https://my.shopforcloud.com/?guest=true (this link will create a guest account for you so you can play with it quickly)
If latency to europe is acceptable for you then you might want to look into Hetzner.
$657/mo will rent you eight 32G/i7 servers there.
If europe is not acceptable then you should still look into american colos (e.g. LeaseWeb recently opened a US DC) which are not as cheap as Hetzner but will still give you 3-4 servers for the money.
Doesn't Germany have crazy data retention laws and privacy regulations? I'd be more concerned about having to follow two sets of laws than trans-atlantic latency.
I'm not sure if others have noticed, but Rackspace London is probably the worst host I've ever used in my entire life. I'm not sure if this is because they were acquired or what, but it's genuinely frightening how awful the service is and how rude, irresponsible, and incompetent the staff are.
That couldn't be any further from my experience in dealing with Rackspace London. Have always found them very helpful and their technical knowledge is the best I've seen.
I moved because Rackspace was unable to resolve performance problems and outages with the interaction of the LBs in front of Cloud Sites and the Cloud Sites PHP nodes in DFW.
They don't (RH has its own datacenters) but I think it's still in some kind of beta. Great thing is that you can rip your app out of there and maintain the "service" yourself. Also, alternative providers for OpenShift will probably pop up over time.
It looks like this is affecting iTunes Match, possibly. I have two tracks just sitting there, waiting to upload and running lsof -i shows iTunes with a connection to an AWS machine.
UTC TIME STEP
on the 1st of July 2012
A positive leap second will be introduced at the end of June 2012.
The sequence of dates of the UTC second markers will be:
2012 June 30, 23h 59m 59s
2012 June 30, 23h 59m 60s
2012 July 1, 0h 0m 0s
8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. Amazon Elastic Compute Cloud (N. Virginia)
This is the kind of weather conditions that spawn very electrically active storms. I don't doubt they could cause the issues. Last night was probably the most electrically active storm I've ever seen up here -- virtually non-stop lightning strikes for an hour or two, and there's another just like it over Virginia right now.
If you have a load balancer you may have balanced across availability zones (Not regions) you'd still be up. So US-EAST didn't all go down, just one AZ.
But many people are saying that despite paying for multi-AZ for RDS, they were still down. Do you think they didn't also load-balance across AZs for their webservers?
AWS is not down. Only US-East. If your app is down, it's only because you don't care about the availability of your service.
It's pointless to complain. We've all seen before that Amazon can't keep whole regions up. If you rely on a region being up, you will have downtime and it's your fault.
According to the AWS status page, only one availability zone within US-East is down, not the whole US-East region. Running a highly-available service exclusively from US-East is a reasonable strategy as long as you're spread across multiple availability zones.
I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.
That's how I feel about most hackathons. There isn't enough work done on existing projects, but quick hacks the demonstrate concepts are the norm. It would be great to see a hackathon where the primary purpose was to contribute to an existing open source project or something similar.
If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems.
We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
Thanks for the most constructive post in this whole discussion. Hacker News is a much better place to talk about solutions than just whining.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.
DNS based failover in practice will not give you 6 second failover, unlike BGP. Too much of DNS is broken and out of your control.
(The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)
Exactly. We announce different subnets for each continent or region, and all the sites within that region announce the same subnet. When one site ceases announcing the shared subnet, BGP usually converges in just more than 5 seconds. I was astonished the first time I saw it. It really makes you appreciate the solid engineering behind the core routers.
It's nontrivial to determine exactly when to drop the announcement. And be careful, because if you are too eager to drop the announcement, you may do it in more than one site at a time.
At first we used DNS with short timeouts, but those timeouts are only advisory and are ignored by some implementations. We would see most traffic tail off within 10 minutes for a one minute timeout, but it took many hours for all the traffic to migrate over to the new DNS. The folklore on using less than one minute for a DNS timeout is that a huge percentage of implementations ignore sub-minute timeout. Funny how much of the Internet's operation is passed along as folklore and not really known for sure.
Thanks for asking. Hacker News should be about sharing best practices and making the Internet a more reliable place.
^ This. You'll do it if you care enough or if it's absolutely required. Everyone knows in advance that this kind of shit can and will happen. Also, if people don't plan on hosting in multiple areas/AZs, I wonder why the hell anyone would even consider overpriced cloud technology while you can get practically 5 powerful dedicated servers for the same price, except that it's more performant than a shitty VM on EC2. That said, if you have 5 dedicated servers, why even bother with EC2? It's more expensive in every way. "Pay more as you grow" is actually extremely expensive for what you actually get. Infinite scalability? Please. When people think about scalability, they think about adding a few gigs of ram to their VM with a little more I/O throughput (e.g. migrating from VM1 to VM2) for hundreds of dollars, and it's still shitty VM performance compared to raw metal. Instead, why not spend that money on a few good dedicated boxes with 96-128gb+ ram and a bunch of true (not virtual) CPU cores, then you're done for a while, and for the same price. Hardware is dirt cheap these days.
The only useful/sane use case I can see in Amazon EC2 would be for services like Heroku where they need to automatically be able to manage a truckload of VM's as their rapidly growing infrastructure, unless you want to do it yourself which I imagine is quite a headache unless you work closely with someone like Amazon or Rackspace.
The "scalability" thing isnt about adding some ram or getting a larger proc. It's about adding a few dozen (or hundred) instances in minutes. Or getting hosts turned up in 7 different regions. Anyone can do that right now with AWS, let me know how your Equinox negotiations go for the next month.
Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time. Personally I run "a bunch" of physical sites across the globe. It would be waaaay easier to be able to turn up rackspace/aws/google instances as needed.
> The "scalability" thing isnt about adding some ram or getting a larger proc.
You'd be surprised how many people that actually use EC2 think it is.
> Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time.
It's called planning ahead of time. If not, then here's a suggestion: Use EC2 until you set it up and migrate, if you cannot wait that is.
All in all I don't mind whether people use EC2 for whatever reason. Just stating my opinion. I agree of course that in terms of "convenience" is has the upper hand. Not having to wait for boxes to be added to data centers, being able to spin up boxes in multiple regions through a single company/console. Maybe your use case does justify using EC2. Many other people clearly do not (hence all the whining because of all the downtime, which they wouldn't have had if they deployed to multiple AZs/Regions).
I know outages happen all the time at hosts, and maybe as a result of either a) news is more accessible now, or b) Amazon is bigger than most other hosts....I feel like Amazon & Heroku are going down WAYYYYY too much.
I am starting to wonder if this "tell all" policy is really best.
Simple solution to this is to have a backup or failover to a non-AWS Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS Azure/Google/Rackspace
This not only spreads your risks but keeps your customers happy.
173 comments
[ 10.2 ms ] story [ 253 ms ] threadSeriously though, as horrible as downtime is, I think most internet users aren't terribly surprised when they can't go to a specific website for a short period of time.
Anyone else care to speculate how this could be good press?
http://www.codinghorror.com/blog/2011/04/working-with-the-ch...
vs
"This site is so popular it barely works"
If you want to be extra paranoid, you could always pick a secondary data center location where it would be safer in the summer, since Denver would be safe from tornadoes in the winter. [2] Or you could just put one on the West Coast and hope that there's not a 1-in-a-thousand tornado within a week of a big earthquake.
[1] https://en.wikipedia.org/wiki/File:Tornado_Alley_Diagram.svg
[2] http://www.weather.com/outlook/weather-news/severe-weather/a...
Yup, everywhere has got its natural disasters.
At the time It was like voodoo, and you had to triple-check your datastore actions, because they could fail for no reason at the backend.
When I got the frantic texts when EC2 first dropped offline, sure enough, the AWS status page was all green, but twitter was alight with people talking about it.
I suspect a service disruption would have to be Godzilla.
http://www.youtube.com/watch?v=tVEPvXBEOSE
Is this not a 'service disruption' situation? At the bottom of the page, the yellow icon is associated with 'performance issues'.
If there's one thing that's shocked me about AWS, it's the total failure to acknowledge the severity of service disruptions. Like the above case, or the fact that a 3-hour loss of connectivity is displayed on the service history as a green tick with a small 'i' box: http://oi46.tinypic.com/x5qtch.jpg
AWS needs to blantantly copy Heroku's status system, which is worlds better for people needing fast updates on their infrastructure.
https://status.heroku.com/ vs http://status.aws.amazon.com/
You can at least mitigate it by keeping utilization of your instances below what would be needed to handle load of one zone failing. I suspect to save money, many people are pushing utilization as high as possible.
Apart from Apple's legendary secrecy, Amazon's EC2 is a solid #2 in terms of impenetrability.
Linode, in comparison, has always been straight-forward, personal, and as honest as a company in that space can be.
What's perhaps odd is Amazon's customer service for their material goods is usually superb.
I can't speak to why Heroku was impacted as much as it was. It could be that they have single points of failure, or run at a utilization level that makes losing a single zone difficult.
Good luck, friends.
This isn't the first time this has happened to AWS - we moved our app to linode last year after this happened to us and it seems to affect AWS more than any other hosting i've ever used, i'd be interested to know how their infrastructure is set up because it doesn't seem particularly robust.
I've been thinking about building a site with a Parse backend, and they're up, which is good to discover.
Wait for the dust to settle. We're all just going to be a bunch of Fonzies here.
EDIT: Looks like API access has been restored, so I'm cautiously optimistic about things working. Note though that some instances may have rebooted or be otherwise impacted so check your error logs.
EDIT2: Nope, ELB is still hosed. Continue to be skeptical.
It's a wonderful idea!
I even have a list of possible locations they should look into. Beyond the Virginia site, they should be looking at DCs in Oregon, California, Ireland, Singapore, Tokyo and even Sao Paulo. What do you think?
US East or US West plus Direct Connect to your own colo space, with AWS for the burst capacity, and your own redundancy for the database servers, might be the best plan if you can't do wide area "over the Internet" database replication. (I might get an extra 10G DC (since I need 1G DC myself) and then have some colo for sale with it in US East/US West later this year.)
It's somewhat important to the original spirit of the comment since Acts of God might indeed happen anywhere (1) whereas acts of man might not. For example, disruption caused by on going war in Syria wouldn't be covered by an Act of God clause.
(1) I think that's BS though...there are definitely some places where nature is considerably more stable than others.
http://www.youtube.com/watch?v=Ua2loiGHZ38
.
...what do you mean lowbrow humor isn't allowed on hacker news?
Rackspace's prices are insane. $1,314/mo for a cloud server with 30gb of ram, compared to $657/mo for 34gb on AWS.
Plus with AWS you can use reserved instances to get that cost down to $286/mo. Rackspace has no way to get the cost down.
That makes Rackspace cost over 4.5x more when comparing based on ram.
Rackspace prices are insanely high and I can't wait to move off of them.
If you're scenario is more complicated than a single server, you might find our tool useful to forecast your costs: https://my.shopforcloud.com/?guest=true (this link will create a guest account for you so you can play with it quickly)
disclaimer: I'm co-founder of ShopForCloud.com
$657/mo will rent you eight 32G/i7 servers there.
If europe is not acceptable then you should still look into american colos (e.g. LeaseWeb recently opened a US DC) which are not as cheap as Hetzner but will still give you 3-4 servers for the money.
Or they're just like Heroku and sit on top of AWS?
http://i.imgur.com/d5pEP.png
It's pointless to complain. We've all seen before that Amazon can't keep whole regions up. If you rely on a region being up, you will have downtime and it's your fault.
I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.
That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?
We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
(The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)
It's nontrivial to determine exactly when to drop the announcement. And be careful, because if you are too eager to drop the announcement, you may do it in more than one site at a time.
At first we used DNS with short timeouts, but those timeouts are only advisory and are ignored by some implementations. We would see most traffic tail off within 10 minutes for a one minute timeout, but it took many hours for all the traffic to migrate over to the new DNS. The folklore on using less than one minute for a DNS timeout is that a huge percentage of implementations ignore sub-minute timeout. Funny how much of the Internet's operation is passed along as folklore and not really known for sure.
Thanks for asking. Hacker News should be about sharing best practices and making the Internet a more reliable place.
The only useful/sane use case I can see in Amazon EC2 would be for services like Heroku where they need to automatically be able to manage a truckload of VM's as their rapidly growing infrastructure, unless you want to do it yourself which I imagine is quite a headache unless you work closely with someone like Amazon or Rackspace.
Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time. Personally I run "a bunch" of physical sites across the globe. It would be waaaay easier to be able to turn up rackspace/aws/google instances as needed.
How do cloud services compare to a gym membership? Are you implying you can't get out of your AWS contract?
You'd be surprised how many people that actually use EC2 think it is.
> Yes white boxes are cheap. Site negotiations, design, procurement, networking, operations, and maintenance are expensive in dollars and time.
It's called planning ahead of time. If not, then here's a suggestion: Use EC2 until you set it up and migrate, if you cannot wait that is.
All in all I don't mind whether people use EC2 for whatever reason. Just stating my opinion. I agree of course that in terms of "convenience" is has the upper hand. Not having to wait for boxes to be added to data centers, being able to spin up boxes in multiple regions through a single company/console. Maybe your use case does justify using EC2. Many other people clearly do not (hence all the whining because of all the downtime, which they wouldn't have had if they deployed to multiple AZs/Regions).
I know outages happen all the time at hosts, and maybe as a result of either a) news is more accessible now, or b) Amazon is bigger than most other hosts....I feel like Amazon & Heroku are going down WAYYYYY too much.
I am starting to wonder if this "tell all" policy is really best.
Second or I believe third power outage/loss of service for AWS in the past 10-days if I'm not mistaken.
This is wild. I wonder what's going on at Amazon and if they're capable of handling this much usage in addition to having power issues, etc.
Instagram and Netflix servers are down from what I hear and have been down for a few hours. Now it makes sense that they're being hosted on AWS.