Linode has a 99.99% uptime guarantee, which amounts to 43 minutes and 12 seconds offline in a 30 day month. While I don't have a Linode in Freemont, their status page appears to indicate, at minimum, 1 hour of downtime. As repayment for the inconvenience, users get the lost time added to their balance.
Linode.com provides a 99.9% uptime guarantee on all Linode hardware,
and on network connectivity. In any given month, if your Linode is
down for more than 0.1%, you may request a pro-rated credit for the
down-time.
Wait so the only backlash they get should they break their guarantee is that they credit me for the down time? This doesn't seem like a motivating factor to keep uptime. Is this standard practice in the industry? I know prgmr has refunded me the entire month when they broke their SLA.
All that said, I won't be getting rid of my Linode box in Fremont simply because it's one of the best vps's I've had performance wise. It just seems like an odd SLA.
What is EC2's SLA, and what cases of downtime have they experienced over the past couple of years? I'm assuming you wouldn't change providers without first finding this out?
From status page: Initial indications are pointing towards another power outage. We are now working on bringing Linodes up as soon as possible. We will continue to post updates as we have more information.
And as luck would have it this happened right after one of my biggest clients sent out a bunch of holiday coupon codes to all of their customers. Do. Not. Want.
Are there clever solutions for routing traffic to other linode datacenters during one of these outages? I'm moving to AWS if I can't do something akin to elastic ip across datacenters. Linode currently only allows you to swap ip addresses within a dc.
use a DNS roundrobin with low TTLs and use monitoring to remove IPs when they go down. if you have 3 hosts in your production app's roundrobin, that makes it much less likely that they'll hit the dead server and if the TTL is 300, then it's only a few minute window anyway.
Aside from choosing a terrible datacenter to colocate in, linode hasn't really done anything wrong here, and isn't responsible for this outage. The outage is actually the responsibility of Hurricaine Electric, the datacenter's actual owners.
Remember guys & gals: this can happen to you. I am a fanatical Linode customer, and they are, in general, an extremely rock solid service, but that doesn't alleviate my responsibility to ensure my application is available regardless of datacenter outage. Power outages happen to EVERY datacenter. There are some very well documented and understood ways to alleviate this problem.
More than one server across multiple datacenters is the only way. The use either a load balancer (which brings its own problems) or some kind of status aware DNS scheme to route traffic to live servers.
I wish linode would offer some kind of high availability load balancing like amazon ec2 does, that really is a killer feature.
I wouldn't be so sure that ELB's are actually failing whole IP addresses across availability zones. Given the current ELB architecture is heavily dependent on dynamic DNS resolution, I am pretty suspicious that they are using DNS for failover. The separate set of DNS servers specifically for elb.amazonaws.com and the 60 second TTLs on the CNAMEs they generate seem to indicate this. Of course, there is no documentation to either confirm, or deny this. Amazon simply states in vague terms that it's highly available, but doesn't state how it's designed, or even possible failure outcomes.
Simple: distribute your application across two isolated datacenters. I'm not trying to downplay the complexity of doing this, as I fully understand the amount of effort required. If the cost of being down for an hour or two isn't significant enough to warrant this level of effort, then a power outage caused by Linode's datacenter vendor probably isn't going to really cost you much either.
Could you provide an always-reliable example? Even anycast has it's downfalls (rogue networks advertising blocks & rerouting traffic onto their networks, security disaster). Most clustered setups rely on a central storage, and if your storage box loses connectivity, your site will be down. Additionally, if you're going to recommend mirroring san content to an identical box at a third party datacenter, the cost of transferring that is going to cost you a boatload of money in bandwidth bills, every single month.
Please change the title to indicate this is just the Fremont datacenter. The current title needlessly scares those of us with linodes in other datacenters.
This is why Linode is better than many other services I've used. Every data center has outages, but at least Linode makes a point to update the server status promptly and (it seems) implement the fixes as quickly as possible.
rbranson is right: the responsibility for redundancy lies with you.
Can anyone recommend good documentation on load balancing a LAMP app across multiple independent VPS providers?
We've had a high profile site affected by these Fremont outages so I'm thinking the only real solution is to have VPSs at Linode, Slicehost and EC2 then load balance among them. Of course, then the question is, what do you do when the load balancer goes down?
38 comments
[ 3.2 ms ] story [ 105 ms ] thread.0001 * 30 * 24 * 60 = 4.32 minutes
With 99.9% uptime:
.001 * 30 * 24 * 60 = 43.2 minutes
All that said, I won't be getting rid of my Linode box in Fremont simply because it's one of the best vps's I've had performance wise. It just seems like an odd SLA.
Most of us don't pay Linode anything important enough to warrant an SLA.
You don't get very strict 99.9% SLAs for $20 and $80 a month anyways...
http://status.linode.com
Even Amazon and Rackspace go down...
Even Google has had outages...
I wish linode would offer some kind of high availability load balancing like amazon ec2 does, that really is a killer feature.
Anyway, its a nice feature that it just works without me needing to set it up.
rbranson is right: the responsibility for redundancy lies with you.
We've had a high profile site affected by these Fremont outages so I'm thinking the only real solution is to have VPSs at Linode, Slicehost and EC2 then load balance among them. Of course, then the question is, what do you do when the load balancer goes down?