Half the startup world is offline right now, yet the AWS status page is all greens with one "info" notice about EBS (which I doubt is the issue here). I'm glad to have moved off a fully AWS stack back to my own servers over a year ago. My uptime, bills, and stress level are all much improved.
That's hardly surprising. The groupthink on cloud services is that they magically make everything scalable, cheap and easy. Making redundant software is hard work.
They posted an update on amazon: We continue to investigate this issue. We can confirm that there is both impact to volumes and instances in a single AZ in US-EAST-1 Region. We are also experiencing increased error rates and latencies on the EC2 APIs in the US-EAST-1 Region.
Also Amazon Relational Database Service (N. Virginia) is unavailible.
I just got up (it's morning in Israel), and a client of mine in the US with a major, mission-critical application was screaming (rightly so) that things are down.
We're already looking into alternatives -- perhaps not leaving Heroku altogether, but certainly not depending on them 100 percent. There's no way that we can entrust the business to something that can just catastrophically fail at any moment. I've been running my own servers for years, and they've never had such unpredictable issues.
I increasingly have to think that a few servers, on different providers, with the application deployed via Capistrano, will be more fault-tolerant than Heroku. At least, it seems that way right now.
I know AWS EC2 does support multiple global regions and within each region multiple availability zones. You're supposed to host your application across more than one AZ if you want good fault tolerance. Not sure if Heroku uses this though.
> There's no way that we can entrust the business to something that can just catastrophically fail at any moment.
Anything, including service providers, can catastrophically fail at any moment. Fault-tolerant architectures are based on redundancy (including infrastructure provider redundancy, as you mention), not on "guaranteed" SLAs.
Provider redundancy goes against the concept of PaaS IMO (ignoring the sci-fi future where there are multiple 100% compatible providers). Heroku needs to become internally redundant to really live up to its promise.
They could have done it to the extreme -- show the numbers for the past hour. Then they could almost always report 100% uptime. And if they ever went down, wait an hour, then go back to reporting 100% uptime again.
It doesn't matter whether gauged by the month or year. It's a percentage. If they have 99.97% uptime every month for a year, they'll still have 99.97% uptime for the year.
That was their uptime for May, but looking at June it's going to be a worse picture. They're already down to 99.63% if you only include "red" incidents and down to 99.25% if you include "amber" incidents as well (as of 39m of downtime for this latest incident and assuming they don't have any more downtime for the month).
Various software used to hardcode 1a, so 1a received disproportionate load. Now, everyone's a-e is randomized among the "true" a-e, meaning that even if everyone hardcodes 1a, the load will still be evenly distributed.
Seems like someone posted a link to a project that made self-hosting a previously Heroku hosted site simple, but I can't find it now...
...would be cool if there was a Linux package (or distro) that you could boot-up and then just change your git remote to and have your app up-and-running on your own hardware.
BTW, while the point is to enable private paas, you won't get around the issues that hit sites like this without heeding all the warnings and recommendations about building in redundancy for high availability.
"""It is a lot cheaper to add 1% uptime to a 95% SLA than it is to add 0.09% to a 99.9% SLA. Cloud application vendors (SaaS) need to pay very close attention to the additional resources that are invested in order to support a 99.9XX…% uptime SLA, and perhaps build it into their pricing plans."""
Yeah I'm just evaluating options, if nothing else it would be excellent to have the equivalent of a "donut spare" on private hardware that we could throw on when events like this occur (although having done plenty of "self-hosted" sites during the first boom, I'm always running the numbers for either option).
I made Dokuen a few weeks ago, maybe that's what you're thinking of? If you're willing to live with a few warts, it's working pretty well for my personal use. My blog is still on Heroku so you can't really read about it now, but you can check out the code.
I saw this a while ago and will actually be using it very shortly to deploy 3-4 internal apps on our own mini cloud. I love Heroku and can't stand all of the open source "alternatives" like Cloud Foundry. Yours is exactly what I wanted and I can't wait to really start using it and contributing via github! Thanks again!
During the previous outage (which wasn't AWS related), Heroku's status page was down entirely (among other things, it relied on static assets from heroku.com), so I can't say I agree with that.
Other datacenters can go down, too. In many cases, the complexity of running an application across different platforms (say, AWS + Rackspace Cloud) might not be worth it.
That's odd, because I thought the error occurred in my application. No? You sure? Just look at this error message, "An error occurred in the application and your page could not be served."
Oh, so you mean the error message aren't accurate? That even though the error is actually above Heroku, a much greater failure, that my application will still be blamed, and that's what my users and bosses are going to see?
And you're saying this is going to happen multiple times per month?
Well shit, and here I thought Heroku was an improvement. I guess rapid scalability and beautifully integrated add-ons doesn't make up for compromised stability. I mean, if it's not going to work, who cares?
Honestly, I haven't the resources to guarantee site uptime, and have accepted this will happen as a result; my complaint was more targeted at the default error messages.
It seems this can be changed, though, fortunately for Heroku.
You can serve your own error pages instead of Heroku's and show your boss whatever you want. They're specified as URLs, so they can be hosted on some other platform. I don't know if that feature works when AWS is failing Heroku, but if Heroku's up enough to serve the error page, maybe it's up enough to serve the error page you configured instead of its own.
115 comments
[ 3.1 ms ] story [ 167 ms ] threadEDIT: Various non-Heroku EC2-East-based sites (e.g. Quora) seem to be down as well, lending more evidence to this being an EC2/EBS outage.
1: http://status.aws.amazon.com/
That IP isn't in the EC2 IP ranges (https://forums.aws.amazon.com/ann.jspa?annID=1528), so at the very least it's not hosted on EC2.
Also Amazon Relational Database Service (N. Virginia) is unavailible.
Seems like its snowballing.
edit: I meant literally the EngineYard website at that address. Some EngineYard websites were up and some were down, no doubt based on region.
We're already looking into alternatives -- perhaps not leaving Heroku altogether, but certainly not depending on them 100 percent. There's no way that we can entrust the business to something that can just catastrophically fail at any moment. I've been running my own servers for years, and they've never had such unpredictable issues.
I increasingly have to think that a few servers, on different providers, with the application deployed via Capistrano, will be more fault-tolerant than Heroku. At least, it seems that way right now.
Anything, including service providers, can catastrophically fail at any moment. Fault-tolerant architectures are based on redundancy (including infrastructure provider redundancy, as you mention), not on "guaranteed" SLAs.
Which is ridiculous, 9s for most services use years as standard. Of course if heroku did that they wouldn't look so good.
[edit: i realise they could be hiding worse months, but i don't think that's what the post i am replying to meant. perhaps i am reading it wrong.]
31 versus 28 days.
The measurement in question is supposed to be about consistency.
*edit: Came up 12:42 AM ET.
That said, an outage is still an outage.
Various software used to hardcode 1a, so 1a received disproportionate load. Now, everyone's a-e is randomized among the "true" a-e, meaning that even if everyone hardcodes 1a, the load will still be evenly distributed.
https://twitter.com/pocket/status/213481664670732288
...would be cool if there was a Linux package (or distro) that you could boot-up and then just change your git remote to and have your app up-and-running on your own hardware.
http://cloudfoundry.org/
BTW, while the point is to enable private paas, you won't get around the issues that hit sites like this without heeding all the warnings and recommendations about building in redundancy for high availability.
This was noted well in this post: http://www.newvem.com/blog/main/2012/06/aws-cloud-best-pract...
"""It is a lot cheaper to add 1% uptime to a 95% SLA than it is to add 0.09% to a 99.9% SLA. Cloud application vendors (SaaS) need to pay very close attention to the additional resources that are invested in order to support a 99.9XX…% uptime SLA, and perhaps build it into their pricing plans."""
https://github.com/peterkeen/dokuen
My love hate relationship with Heroku continues...
You know there are OTHER data centers, right?
Oh, so you mean the error message aren't accurate? That even though the error is actually above Heroku, a much greater failure, that my application will still be blamed, and that's what my users and bosses are going to see?
And you're saying this is going to happen multiple times per month?
Well shit, and here I thought Heroku was an improvement. I guess rapid scalability and beautifully integrated add-ons doesn't make up for compromised stability. I mean, if it's not going to work, who cares?
It seems this can be changed, though, fortunately for Heroku.
https://devcenter.heroku.com/articles/error-pages
Tonight's events would've resulted in the misleading default error message, regardless of customer error pages.
With AWS failing, that's a big issue and platform failure should be vocalized correctly.