I meant "the irony of running a status page that's reporting when a platform is down on the platform itself", not the irony that heroku isn't running its status page on itself.
I was wondering why http://intercom.io was down and now I come here and see this at the top. Must be it. Update, they seem back now, but their site was throwing errors a minute ago.
Interesting. I was just considering them over AWS for hosting. Current users, how reliable is Heroku? While I appreciate openess, looking over that status page does not inspire a lot of confidence.
At the size of Heroku, the probability of a server outage per day is actually quite high. As long as the errors are just "one of our machines has gone down", its an interesting information for everyone on the machine, but it doesn't say a lot about the reliability at all.
For ref, the heroku app is rails, 2x dynos and a shared DB with minimal traffic. The others are PHP/apache apps, one using RDS, on plain EC2.
Heroku is at least 5 layers extra to go wrong on top of EC2. On the one hand I feel that each layer adds to the probability of failure. But I'd hope that one of the benefits of Heroku would be extra redundancy and failure handling, though my limited personal experience hasn't been that way so far.
EDIT: Regarding openness, I certainly feel that every single outage is admitted by Heroku, whereas I feel Amazon's status page has not listed a few outages I've suffered. Again personal feeling, not evidence.
When Heroku has issues during working hours, their support is great - you can pretty easily be put in contact with an engineer or a product manager. The catch here is that there doesn't appear to be any guaranteed response time to any tickets (my urgent ticket (everything is on fire) from Sunday was accepted Monday morning).
Amazon on the other hand is quite opaque - SQS/SimpleDB, for example, went down today for a few minutes (as in the service endpoints told us as such), but there wasn't a peep on the AWS status dashboard.
May not be related, but if you're on Cedar you'll want to make sure that Pingdom is sending a GET request. I've seen a few uptime monitoring services that send HEAD requests, these worked on Aspen/Bamboo as nginx would translate them on the way through.
I've no idea why these services didn't report that my apps were permanently down :(
Lets hope the simplicity of 'git push heroku' hasn't distracted most developers (and customers of Heroku) from Werner Vogels' core tenet of administration:
Personally I don't think it makes sense to build in redundancy if you're using Heroku, maybe a fallback to a simple offsite "we're down page" but that's it.
My reasoning is that if you are on Heroku you should be small and leveraging Heroku's management to the fullest. If you are big you are paying an incredible premium to be on Heroku (thought experiment: when does the premium start costing more than a full-time systems engineer). There is no Heroku equivalent from another company, at least nothing that's really turnkey in the same way, so if you build redundancy you're throwing away the benefit of Heroku's PaaS.
So in my mind if you need redundancy you need to be building your own system stack. Whether it's co-located, dedicated, or cloud-based is a separate issue because it's more or less commoditized. You build your system on a portable platform and then redundancy is just a proportional amount of extra work.
Thanks for the thoughtful reply. I think that you raise some valid points here.
Regarding "If you are big you are paying an incredible premium to be on Heroku": Sure, Heroku is great (or cheap) for development and small-time deployment, however it is likely true that people get locked into it and (hopefully) grow quickly before they can scale away from a PaaS solution. In these cases, it is imperative to consider the marginal cost of alternative/off-site failover vs. the cost of downtime. This is what I meant by "separate the men from the boys". Those guys who are paying the incredible premium to continue to host on Heroku likely -know- that they are doing so, and have made that decision consciously.
While the situation is different, I am reminded of when AWS went down awhile back and the sites that had automated failover to alternative clouds really stood out.
The bottom line is that no one solution is ever enough. Tiny apps can likely get away with unexpected//unplanned downtime, however it ultimately delegitimizes them in the eyes of their users (read: customers), and that is universally bad.
Sure, but my point is that if you determine you need 100% uptime and you are currently on Heroku, you are going to need to build a complete non-Heroku system to fail over to. At that point should you A) pick another high level PaaS provider and maintain two very different stacks, each of which is commanding a high premium, or B) build on a standard Linux stack that can be deployed to multiple completely separate clouds?
Or C) Heroku should develop and deploy fully independent availability zones and regions. Amazon hasn't yet done a great job of keeping AZ and even regions independent but better connected than separate providers, but maybe Heroku could do better (even if built on EC2).
That would be a valuable service for Heroku to provide, but being better connected also implies more likely for joint failure, even if Heroku does everything right, there is stampede failure risk (which is what happened in the last major EC2 cross-zone outage). In any case, I think if you are seriously pursuing 100% uptime, you can't outsource your redundancy.
"better connected" could just be a billing thing; they make sure there is great transport (their own, transit providers, whatever) between AZs running in separate ASes, and don't bill for inter-AZ traffic.
10-20% of apps developed by people on Hacker News probably use Heroku. It's not just a random service people use; if there is an outage, service operators will need to field end user inquiries, restore service, etc.
hn is probably not a replacement for a good offsite monitoring service, but for me, I was browsing "why does my friend's new app not work; is he updating it live?", switched tabs to hn, saw "heroku down", and all was clear.
I was going by other private surveys of specific subsets of the hn population. Number of companies, it's 10-20%. Lots of smaller companies or earlier stage; basically everyone who does Rails deployments (since that was originally Heroku's strength).
GAE is the one very few people are using or want to continue using, especially after the Google price hike.
I still prefer colo/managed hosting at scale, with EC2 for trials or surge capacity, myself.
No need to make it harder for them by voting it to the top of a major tech news site...
Curious what you mean by this? Could you clarify? My initial impression of this made me think that you're saying that we shouldn't draw attention to a service provider's downtime.
If a Heroku application's DNS is setup correctly then you can usually avoid problems like this on the new Heroku stack (CEDAR). It's important to reread the documentation on using custom domains but also be VERY aware of using naked domains (mysite.com) in your application. This isn't just a Heroku problem, it's a DNS problem. Heroku have recently had a spate of routing issues (usually DDoS attacks) which would have been largely negated by correct DNS setup and either avoiding naked domains or using a DNS host that lets you cname a naked domain to a host.
Could you please elaborate on why exactly using (non-CNAMEd) "naked domains" is a problem? What do "naked domains" and DDoS attacks have to do with each other?
DNS only allows you to put an IP addresses on the apex (read: bare/naked/etc) domain A record. That means in the case of a DDoS or other problem affecting one of those IPs you run the risk of having degraded redundancy. Some people unfortunately only put one IP in, which means you're only one machine away from going offline. A long way from what you'd want from the cloud.
DNSimple have created an ALIAS record type which gets around this problem nicely, and Route 53 from Amazon takes a similar approach.
48 comments
[ 1.9 ms ] story [ 103 ms ] threaddoesn't sound good
Major outages are rare and I am quite pleased.
For ref, the heroku app is rails, 2x dynos and a shared DB with minimal traffic. The others are PHP/apache apps, one using RDS, on plain EC2.
Heroku is at least 5 layers extra to go wrong on top of EC2. On the one hand I feel that each layer adds to the probability of failure. But I'd hope that one of the benefits of Heroku would be extra redundancy and failure handling, though my limited personal experience hasn't been that way so far.
EDIT: Regarding openness, I certainly feel that every single outage is admitted by Heroku, whereas I feel Amazon's status page has not listed a few outages I've suffered. Again personal feeling, not evidence.
When Heroku has issues during working hours, their support is great - you can pretty easily be put in contact with an engineer or a product manager. The catch here is that there doesn't appear to be any guaranteed response time to any tickets (my urgent ticket (everything is on fire) from Sunday was accepted Monday morning).
Amazon on the other hand is quite opaque - SQS/SimpleDB, for example, went down today for a few minutes (as in the service endpoints told us as such), but there wasn't a peep on the AWS status dashboard.
I've no idea why these services didn't report that my apps were permanently down :(
Everything fails all the time. (http://technocation.org/files/videos/original/mysqlconf2008/...)
This might separate the men from the boys (excuse the idiom) to see who has adequately prepared a DR/redundancy plan for when the cloud fails.
Edit: Improved link to Vogels' MySQL keynote from which the quote originated.
My reasoning is that if you are on Heroku you should be small and leveraging Heroku's management to the fullest. If you are big you are paying an incredible premium to be on Heroku (thought experiment: when does the premium start costing more than a full-time systems engineer). There is no Heroku equivalent from another company, at least nothing that's really turnkey in the same way, so if you build redundancy you're throwing away the benefit of Heroku's PaaS.
So in my mind if you need redundancy you need to be building your own system stack. Whether it's co-located, dedicated, or cloud-based is a separate issue because it's more or less commoditized. You build your system on a portable platform and then redundancy is just a proportional amount of extra work.
Regarding "If you are big you are paying an incredible premium to be on Heroku": Sure, Heroku is great (or cheap) for development and small-time deployment, however it is likely true that people get locked into it and (hopefully) grow quickly before they can scale away from a PaaS solution. In these cases, it is imperative to consider the marginal cost of alternative/off-site failover vs. the cost of downtime. This is what I meant by "separate the men from the boys". Those guys who are paying the incredible premium to continue to host on Heroku likely -know- that they are doing so, and have made that decision consciously.
While the situation is different, I am reminded of when AWS went down awhile back and the sites that had automated failover to alternative clouds really stood out.
The bottom line is that no one solution is ever enough. Tiny apps can likely get away with unexpected//unplanned downtime, however it ultimately delegitimizes them in the eyes of their users (read: customers), and that is universally bad.
Services go down sometimes, especially cloud ones. No need to make it harder for them by voting it to the top of a major tech news site...
edit - spelling
hn is probably not a replacement for a good offsite monitoring service, but for me, I was browsing "why does my friend's new app not work; is he updating it live?", switched tabs to hn, saw "heroku down", and all was clear.
https://news.ycombinator.com/item?id=3466168
GAE is the one very few people are using or want to continue using, especially after the Google price hike.
I still prefer colo/managed hosting at scale, with EC2 for trials or surge capacity, myself.
Rubyists.
Curious what you mean by this? Could you clarify? My initial impression of this made me think that you're saying that we shouldn't draw attention to a service provider's downtime.
I don't have a site on Heroku, but I clicked through anyway - it's on the top of Hacker News, so must be worth looking at.
DNSimple have created an ALIAS record type which gets around this problem nicely, and Route 53 from Amazon takes a similar approach.