3 out of 4 us-east-1 AZs are unavailable from the outside. The AZ that is working is the one that has chronic availability issues, so there is no way to recover service in that zone.
I should have been more specific, I was in a rush =)
The original us-east-1b has chronic capacity issues, meaning that it is always at or near its capacity. AWS refuses to sell instance reservations for this AZ, and it's often difficult/impossible to launch new instances in this zone.
Keep in mind that AZs are remapped for all accounts newer than a certain point in time (say, 1 yr ago). Your 1b may not be my 1b.
Imagine launching your startup on the day Amazon AWS goes down? On another note.. why don't these power outages ever affect Amazon.com from going down? Come'n, eat your own dog food!!!
What? You mean without a backup plan to substitute in for the platform with known reliably issues? Yeah I guess that would suck.
Edit re your edit on dogfooding: supposedly they use the same tech, just not the same servers, so their AZs are probably separate form ec2s. Also, they're in all of their geographical locations, not just us east.
Wow - looks like you're right! I dug into the presentation (link below, important slide is #33) and Amazon are now claiming that all their web servers run on AWS. It's only their webservers (not their DBs, load-balancers or "services" which I presume means anything stateful). Of course, the co-location option for 'difficult' services is only available to Amazon, and before the switch-over in November they presumably weren't using AWS nearly so heavily, but this is at least a step in the right direction.
Amazon specifically tells you to set up multiple servers in multiple availability zones. They probably follow their own advice, and, as a result, don't go down.
I've talked to some people about AWS about this, and the reason why they have availability zones is because they don't want to charge you the speed cost of syncing data between zones if your app doesn't need 100% uptime. Generalized replication slows down your app. AWS gives you the option of not having replication or bringing your own.
The last two outages have affected multiple availability zones in the US East region. To really account for it, you'd need instances in different regions.
Amazon.com doesn't run on the same, precisely speaking, infrastructure. My understanding from other Amazon employees was that it (EC2) started as an internal tool.
...and only Amazon can afford to follow their own advice. Multiple AZ hosting ain't cheap. Most CEO/CIO/CTO types spit out their coffee when they see the costs of fully redundant hosting in the cloud at which point they decide "For that price we can afford to be down for 24 hours."
Very few businesses need 100% uptime. As long as you have good recovery strategies, and exercise them routinely, you should be set. When was the last time you ran a failover simulation? do your ops guys know what to do? are there clear lines of communication as to the status of the event?
Outages are hard to avoid, but the pain can be lessened if your customers are aware of the recovery progress and you can deliver on your recovery time goals. Nothing is worse than being down, and leaving customers in the dark to start rumors that your guys are not even aware of the problem.
Huh. My piddly little website hosted on a micro instance is down. This kind of sucks, not that I get a lot of hits, but I have a few things hosted there that local stuff depends on.
damn, http://status.twilio.com/ - calls are still going through, just a few extra rings before my service receives the http request from twilio... gotta love twilio they rock - looks like no call recordings...
Using dropbox is a bad idea overall in my opinion. They lied to their customers about what they can and cannot view in your account. They lied to you. They intentionally told you that they could not view your files when in fact they can. If that's not enough, they had a critical security vulnerability (log in with no password) for four hours, proving that their systems fail open. Finally, if all this wasn't bad enough yet, they do not encrypt your files when they store them[1].
[1] Technically they do encrypt the files, but the keys are right next to them on the same infrastructure. Doesn't do any more good than not encrypting them.
From a security conscious perspective it is a bad idea, but too many people undervalue noncritical data and don't secure it like they should (That's part of why infosec is such big buisness). A prevelant lack of security sense is basically the only reason dropbox is still in business.
But from a HN perspective they've received YC cash so your post is currently voted <= 0.
For the bits that need to be protected there is Tarsnap (http://www.tarsnap.com/). The client is open source, so you can check for yourself that things are encrypted before going on the wire.
as of 9:59 PM CST it appears we are back up. Additionally, all our instances are running normal. It appears to be a network connectivity issue for AWS.
I didn't get any SMSes or calls from PagerDuty even though I see 4 incidents triggered in the PD web UI. Although that could be due to Twilio going down.
There's a good reason for that: when AWS has problems, we are under very heavy load because many of our customers are on AWS. The last thing we want to do at that point is an emergency flip to a secondary provider.
I experienced intermittent issues. It tried retrieving the content to stream and failed. This lasted for about 20-30 mins. Was using wii streaming during that time.
Two large, lengthy outages within a few months of each other? I hate to piggy-back on Amazon's follies, but it's times like these that make me love my Linode boxes. Total downtime over the last 5 years? 5 hours. I really couldn't ask for a better hosting company.
Edit: The 5 hours are what I've noticed from either personal servers or those of friends/acquaintances. YMMV.
I'm guessing you haven't been affected by any of the Fremont(Hurricane Electric DC) outages? I have a lot of Linode boxes and love their service but I'm still splitting load with different providers as well.
Yep, the Fremont DC has been down at least 3 times in the past year. I still love Linode, but i'll probably move my (personal) VPS to another of their facilities.
Linode is pretty good about accurate updates for Requests For Outages(RFO): http://status.linode.com/
I don't keep track of the totals but I think it you search their forums, I seem to recall someone who had monitored them with Pingdom or Wormly(or equivalent) for a fair period of time.
I remember the outages at the Freemont location. Every outage this year has been related to power issues. The company running the Fremont datacenter is Hurricane Electric. I hope Hurricane Electric gets things sorted soon at the Fremont datacenter. Every time I think about the company name "Hurricane Electric", I think "now there's a company name that inspires confidence." The name "Disaster Service" must have already been taken.
It's worth pointing out that I am in the US-East with 8 beefy instances and my site stayed up during the entirety of both outages, so don't assume you're getting better uptime than all AWS customers.
Yep, I'm on Linode, too. Love it. Linode guarantees three nines but in practice it gets close to four nines. EC2 looks like it's falling below three nines for some regions this year, even though it guarantees 99.95%.
For the vast majority of projects I've worked on, I just can't seem to justify the cost of Amazon or similar cloud environments, especially given the relatively poor performance and increasing risk. A simple shared host works for most of what I'm doing, and when it doesn't, a virtual dedicated or something like Linode works well. While these environments don't protect from single points of failure, it seems that neither does Amazon. A smart clustering / distributed approach would work equally well in the Amazon cloud as it would for shared hosting / VPS type services.
I'm not sure I understand the draw of Amazon for startups. Even from an economic perspective, it seems to me to be more expensive than I had hoped, and the promise that the Cloud allows for highly flexible scale at low risk doesn't seem to match to reality. Then again, there have been precious few outages, so maybe the reliability is there? In any case, it was the economics and poor performance, rather than these outages, that leave me wondering if Amazon is really worth it.
Overall they have great service. Only their Fremont location has been a bit touchy this year. I've been with them since the days they used UML before Xen and if it's any consolation, I have experienced at least one downtime in each DC(but this was over the course of many years now). Unless you plan for distributed systems, expect some failures from time to time no matter who is the provider.
Just a fluke. I've been a Linode customer since 2008 and have managed multiple Linodes (on the East coast) and as far as I can tell, Linode has given me more than four nines.
From what I've encountered, and what I've read, Fremont has been the most patchy Linode DC - they've had two outages there recently. One just a day or two ago, and another in early May. Apparently there was a significant outage late last year too.
p.s. The last two weren't network outages - they were total power failures. Your 'node would have been hard-booted as a result.
I run a handful of virutal machines on Rackspace's private cloud. And while their customer service tends to be dire the VPS's themselves are pretty solid. Desperately needs more features though - with a private back-end network between the servers (rather than shared!?) it'll be appropriate for many more applications.
You don't get what you don't pay for. "Linode works for me" is just anecdotal. I suspect Linode has smaller margins than Amazon, though, and tries to pass savings onto their customers despite not having Amazon's economies of scale.
An Amazon small instance has 1.7 GB of memory, and 160 GB of "local instance storage" (which Amazon will wipe when your instance goes down), and "1 EC2 unit" (a single core on a 1.2 GHz 2007 Xeon processor). It will cost you about $60 a month.
A similar plan on Linode will give you 1.5 GB RAM, and 60 GB storage. They will also throw in bandwidth and persistent storage, though.
Amazon tends to give you more flexibility (on demand prices, spot prices, dedicated prices, bigger instances, smaller instances, persistent storage, cheap temporary storage, and so on). But it's not as easy to use as Linode.
Linode targets people who want a cloud server. Amazon has a number of uses.
An Amazon "small" instance in no way compares to a Linode 1.5GB server. Linode beats the pants off of Amazon at every price point so badly that it's hard to even compare the two. A 4GB Linode performs faster than an 8GB EC2 "large" instance. Only the Amazon "High-CPU" instances come close, so you'd need to compare an EC2 "medium" with a Linode 2GB to make it even close to fair. The Linode also comes with 8 virtual cores, the EC2 unit with 2.
The real trouble with Amazon is the erratic performance of the EBS system which can kill an otherwise fast system. Linode's I/O does seem to be much more consistent and predictable.
Amazon's storage capacity and the cost of storage is unmatched, though. If you have a mountain of data, Linode will be prohibitively expensive.
151 comments
[ 3.5 ms ] story [ 216 ms ] threadDUH - doesn't seem to be - that was west coast, IIRC, and the only issue I see now is in Virginia.
The original us-east-1b has chronic capacity issues, meaning that it is always at or near its capacity. AWS refuses to sell instance reservations for this AZ, and it's often difficult/impossible to launch new instances in this zone.
Keep in mind that AZs are remapped for all accounts newer than a certain point in time (say, 1 yr ago). Your 1b may not be my 1b.
Edit re your edit on dogfooding: supposedly they use the same tech, just not the same servers, so their AZs are probably separate form ec2s. Also, they're in all of their geographical locations, not just us east.
http://www.slideshare.net/AmazonWebServices/2011-aws-tour-au...
I've talked to some people about AWS about this, and the reason why they have availability zones is because they don't want to charge you the speed cost of syncing data between zones if your app doesn't need 100% uptime. Generalized replication slows down your app. AWS gives you the option of not having replication or bringing your own.
Outages are hard to avoid, but the pain can be lessened if your customers are aware of the recovery progress and you can deliver on your recovery time goals. Nothing is worse than being down, and leaving customers in the dark to start rumors that your guys are not even aware of the problem.
Must be this.
[1] Technically they do encrypt the files, but the keys are right next to them on the same infrastructure. Doesn't do any more good than not encrypting them.
But from a HN perspective they've received YC cash so your post is currently voted <= 0.
http://reports.panopta.com/cloudharmony-borderless/server/57...
various ec2 clients affected:
- dotcloud
- dropbox
- engine yard
- foursquare
- heroku
- hootsuite
- instagram
- kicksend
- netflix
- pagerduty
- reddit
- twilio
Perhaps they should do the same for the website just to instill extra confidence, though.
http://www.codinghorror.com/blog/2011/04/working-with-the-ch...
Edit: The 5 hours are what I've noticed from either personal servers or those of friends/acquaintances. YMMV.
I don't keep track of the totals but I think it you search their forums, I seem to recall someone who had monitored them with Pingdom or Wormly(or equivalent) for a fair period of time.
http://en.wikipedia.org/wiki/Nines_(engineering)
I'm not sure I understand the draw of Amazon for startups. Even from an economic perspective, it seems to me to be more expensive than I had hoped, and the promise that the Cloud allows for highly flexible scale at low risk doesn't seem to match to reality. Then again, there have been precious few outages, so maybe the reliability is there? In any case, it was the economics and poor performance, rather than these outages, that leave me wondering if Amazon is really worth it.
p.s. The last two weren't network outages - they were total power failures. Your 'node would have been hard-booted as a result.
An Amazon small instance has 1.7 GB of memory, and 160 GB of "local instance storage" (which Amazon will wipe when your instance goes down), and "1 EC2 unit" (a single core on a 1.2 GHz 2007 Xeon processor). It will cost you about $60 a month.
A similar plan on Linode will give you 1.5 GB RAM, and 60 GB storage. They will also throw in bandwidth and persistent storage, though.
Amazon tends to give you more flexibility (on demand prices, spot prices, dedicated prices, bigger instances, smaller instances, persistent storage, cheap temporary storage, and so on). But it's not as easy to use as Linode.
Linode targets people who want a cloud server. Amazon has a number of uses.
The real trouble with Amazon is the erratic performance of the EBS system which can kill an otherwise fast system. Linode's I/O does seem to be much more consistent and predictable.
Amazon's storage capacity and the cost of storage is unmatched, though. If you have a mountain of data, Linode will be prohibitively expensive.