When will the rest of the industry start to bundle a battery on every server, a la laptops and Google servers, so that PDU glitches like this would be no big deal?
Honest question, would a normal battery be able to keep a decent server running through a 5 minute outage? I was under the impression that google adds something like 9v's to last a few seconds through a flicker, not any sustained outage.
(And all of that's assuming your network stack stays up, and things like routers/firewalls survive...)
The idea is not to be reachable during the outage, it's to avoid filesystem corruption, not lose data in memcached, etc. These things are recoverable, of course, but if you can avoid them for the cost of a $2 battery... then why not?
Never, I hope. As someone who actually touches the hardware in datacenters, ever, I'd like not to have the added fire and hazmat risk of that much lithium and non-EPSable power everywhere.
Five nines is impossible Really. It’s just not going to happen.
Actually, it is going to happen. One individual part may fail more than 99.999% of the time, but overall system integrity can certainly be designed with greater than 99.999% uptime.
Just get a second datacenter, get a second transit provider. Just as folks scale horizontally, you can built out reliability into the many many 9's such that when one component fails, the overall system integrity isn't impacted.
It's not easy, and not always cheap, but it's quite doable.
Perhaps "impossible" is overly dramatic, but "economically impractical" may, in this context (general purpose networked computing), be the same thing.
It seems to me that "five nines" is achieved more often with careful use of definitions, rather than careful engineering. Low-hanging fruit are scheduled maintenance and degraded performance.
Otherwise, wouldn't Tandem have been wildly successful?
Complexity costs. It costs money and it costs downtime. Look at Google: they have redundant everything, super-smart people, super good operations and still have some downtime.
According to Amazon, EC2 can remove a server from an Elastic Load Balancer rotation at a polling interval of 5 seconds. You can pool servers in different availability zones into the same ELB. I'm not sure the question even makes sense in that context, which I sort of view as the logical extreme of what davidu wrote.
I've worked places that had that setup with BGP. Of course, you need to announce the same IP block out of both data centers, your application needs to be able to deal with that, and that sort of thing gets pricy, but it's doable.
5 nines is completely unrealistic for the internet. The internet backbone doesn't even have 5 nines. Nobody has 5 nines, not even google.
5 nines is 5 minutes of downtime per year, or an hour per decade. If you have an hour of downtime, you need to have 100% uptime for a decade to achieve 5 nines. The only thing that has 5 nines uptime is the land based telephony network in the US, and only after many decades of effort struggling to get there. Think about the massive amount of redundancy, extra capacity, and backup systems the phone network has, much of it mandated by law. Not merely battery backups everywhere, but backup generators too.
Think about the implications of building a system for actual 5 nines reliability. With today's technology it is very, very costly. If google built their systems the same way the telephone network does they would be operating at a huge loss instead of a hefty profit. Over-engineering is a waste of money. For the vast majority of cases, 5 nines reliability on the web is excessive and unrealistic.
I was actually a bit dissatisfied with how slicehost (part of rackspace, or at least using their DC) handled this last night.
If something like this goes down, I want to know what is up within minutes of it happening. At least a "Yeah, things are going wrong. It's our fault, not yours.". I checked slicehost's website but...nothing. The only way I even knew that others were having a problem was by checking twitter.
Seriously, guys...one of the things behind the pane of glass that says "break in case of emergency" needs to a sheet of paper with big red bold letters on it that says "TELL THE CUSTOMERS WHAT IS HAPPENING RIGHT NOW! DO NOT WAIT!"
/yeah, yeah, I'm over-reacting. It's just really frustrating to be completely in the dark when something goes wrong, even if it is only for a few minutes.
However, in a couple of my career cycles I've managed data centers and/or server farms upon which lots and lots of companies relied.
Many times these unexpected outages are not always easy to diagnose even a rough root cause quickly. Sometimes you don't know if it's your issue, or an upstream issue, and so on.
So, you put out a statement 5 seconds after and outage, and, oops, you misdiagnosed it. Now you get to deal with the shitty customers who want to play 1000 questions about why you told them it was a transfer switch when it turns out it was really a main breaker. Not that it matters much in the end.
You're damned if you do, damned if you don't, but most of the time it's better overall to release an accurate statement later than a wrong statement early.
There is a significant distinction between what you would expect of a free service (Twitter) and what you would expect of a service/site you are paying money for and building your corporation upon.
In both cases, I had clients that I knew were rational and logical. Those are the ones that we would issue an early release to, along the lines of "Something broke and we're on it, more info to follow".
However the irrational types seemed to far outweigh the logical types. Those are the ones where I can tell you from my own first hand experience that overall you are better off giving accurate information late than wrong or incomplete information early.
Today, I run a small bit of a data center as one of my personal side-project companies. I only accept low-overhead experienced clients. They don't give me a lot of headaches, and the 1 single time we had an issue (power related, natch) I told them about it as soon as I found out and thing was a completely forgettable experience for everyone.
Those are the ones where I can tell you from my own first hand experience that overall you are better off giving accurate information late than wrong or incomplete information early
My expirience is the opposite. All customers I have dealt with were very happy to receive an acknowledgement early and a post-mortem analysis later.
Most customers don't care one bit about what is wrong, they care about when their service will be restored. Knowing "they are working on it" is already one step up from "Does this affect only me? Have they even noticed, yet?".
to be clear, the above is meant as the lesson to the customer, not rackspace. It would be very nearly impossible for rackspace to make their virtual servers stay up if a whole data center went down. It's quite difficult to keep a VPS up when even just the hardware it is on goes down.
You need to put a switchover device somewhere. And that device (or NOC, or power, or router feeding it), can fail.
Unless you want to give you dns addresses a 10 second ttl, and update them as needed.
Really, how? I would love to do this, but I can't think of any method that does not have a single point of failure somewhere. (And I'm not as big as google with my own network lines, something that a regular budget can do.)
BGP. the idea is that you could have a data center in San Jose, connected via he.net, and another in, say, phoenix, connected via Cogent. you announce the same /24 out each data center... if one goes down, it stops announcing and all traffic goes to the other data center.
The problem is that it's not a cheap answer, and that your application needs to be able to deal with queries going to either data center at any time.
another option I've seen people use which is not nearly as robust but still 'automatic' is to have two authoritative nameservers; one at each location. But the nameserver at location 1 resolves all A records to IPs at location 1. At location 2, the nameserver is configured to resolve all A records to IP addresses at location 2. (of course, this has all the dns caching issues as setting the TTL to 10 minutes)
A third option would be to have some sort of redirector on a network that is made redundant via BGP, then have some sort of hook so you can change the target of the redirector. But I don't know of anyone offering that service for cheap.
No, the right lesson is to take a deep breath and ask yourself what three hours of downtime is worth to your business and compare that to the cost of full failover.
The cost of full failover is first off a doubling of your hardware and datacenter costs. Then add the cost of developing a software system that can handle seamless failover, as well as testing it. By this I also mean the cost of significantly slower release-cycles, because you're not going to release a new feature without testing if it breaks the fail-over, are you?
If this comes out in favor of doing full failover, you're a bank or an airline or similar. You're probably not even Google or Facebook. Chances are that you're not taking infrastructure advice from an online discussion thread.
Although I think your overall point is valid I wonder whether there is a middle ground on the hardware. For many purposes offering a degraded service would be better than nothing and could perhaps be handled with 1.5x your needed resources rather than 2x. It may also be possible to make use of this fabled cloud to take the 0.75x resources you have left after an outage back up to 1x.
All this does add development time though and, as you say, for most sites it isn't going to be worth it.
Infrastructure, for most businesses, is a vanishingly small percentage of the total costs of doing business. Hell, I sell infrastructure, and at 750 customers, my infrastructure costs are still less than I'd pay for my time if I was paying myself market rate.
But you do have a good point, that you do need to weigh the cost of a few hours downtime every now and again with the cost of putting yourself in a position where you can avoid such downtime, because yes, sometimes avoiding the downtime is more costly than just taking the hit.
I think at a minimum, though, you need to have off-site backups, and the ability to restore to a new provider if your first provider has a HyperVM level disaster that kills all your data and all the backups at that provider.
I received this email an hour back from Rackspace:
At approximately 12:29am CST this morning, our Dallas - Fort Worth (DFW) data center experienced a power disruption, and consequently an interruption of our services. The power disruption was the result of issues during a maintenance effort that was scheduled and expected to be non-impacting.
This summer our DFW facility had power issues, and as a result, we invested significant resources to improve all aspects of our power systems. Last night, during one of these steps, we encountered issues and had a brief loss in power. The power disruption was approximately 5 minutes in duration. Despite this short power disruption, many customers experienced downtime that was significantly longer. Since the power disruption hit the core of many of our cloud services, recovery of full operations required more effort than simple recovery of power. The experience you had last night is not acceptable to us.
Here is what we know about the events:
· The scheduled maintenance was planned to occur from 12:05am - 6:05am CST in our DFW data center. This maintenance is part of a preventative maintenance schedule for several PDUs in UPS Cluster G at the DFW datacenter. The PDUs were down for a total of 5 minutes before power was restored. At approximately 12:29am CST, all PDUs behind UPS Cluster G lost power.
· Although the power outage was very brief (5 minutes), it forced a hard re-boot to occur on a portion of our cloud infrastructure. As our engineers worked to bring hardware back online, we experienced several unforeseen hardware failures. Further complicating our recovery effort, the incident also created internal DNS issues, which caused additional delays. With that said, the vast majority of cloud customers affected by this outage had service restored within one hour's time (many in as little as five minutes); however, depending upon the service, a few customers experienced service interruptions for up to few hours.
Here is how we plan to deal with it:
· We have invested massively in the DFW facility to ensure it delivers at a level you expect from Rackspace - despite last night, we feel very good about our plan and have high confidence in the DFW facility - clearly we have to prove it.
· We are reviewing our maintenance notifications - we typically do not share information on expected non-impacting events, but clearly we need to ensure we calibrate these events and are fully transparent.
· We are reviewing our procedures and systems for quickly resuming cloud operations when an unexpected event like this occurs - unexpected events will happen, our job is to minimize their impacts.
We live by high standards and clearly have not lived up to them. We welcome any feedback. If you would like a call from me, or anyone on our senior team to discuss these issues personally, please reply with a phone number.
We have work to do to earn back your trust. We will not rest until we have.
31 comments
[ 4.6 ms ] story [ 158 ms ] thread(And all of that's assuming your network stack stays up, and things like routers/firewalls survive...)
Actually, it is going to happen. One individual part may fail more than 99.999% of the time, but overall system integrity can certainly be designed with greater than 99.999% uptime.
Just get a second datacenter, get a second transit provider. Just as folks scale horizontally, you can built out reliability into the many many 9's such that when one component fails, the overall system integrity isn't impacted.
It's not easy, and not always cheap, but it's quite doable.
It seems to me that "five nines" is achieved more often with careful use of definitions, rather than careful engineering. Low-hanging fruit are scheduled maintenance and degraded performance.
Otherwise, wouldn't Tandem have been wildly successful?
5 nines is 5 minutes of downtime per year, or an hour per decade. If you have an hour of downtime, you need to have 100% uptime for a decade to achieve 5 nines. The only thing that has 5 nines uptime is the land based telephony network in the US, and only after many decades of effort struggling to get there. Think about the massive amount of redundancy, extra capacity, and backup systems the phone network has, much of it mandated by law. Not merely battery backups everywhere, but backup generators too.
Think about the implications of building a system for actual 5 nines reliability. With today's technology it is very, very costly. If google built their systems the same way the telephone network does they would be operating at a huge loss instead of a hefty profit. Over-engineering is a waste of money. For the vast majority of cases, 5 nines reliability on the web is excessive and unrealistic.
If something like this goes down, I want to know what is up within minutes of it happening. At least a "Yeah, things are going wrong. It's our fault, not yours.". I checked slicehost's website but...nothing. The only way I even knew that others were having a problem was by checking twitter.
Seriously, guys...one of the things behind the pane of glass that says "break in case of emergency" needs to a sheet of paper with big red bold letters on it that says "TELL THE CUSTOMERS WHAT IS HAPPENING RIGHT NOW! DO NOT WAIT!"
/yeah, yeah, I'm over-reacting. It's just really frustrating to be completely in the dark when something goes wrong, even if it is only for a few minutes.
However, in a couple of my career cycles I've managed data centers and/or server farms upon which lots and lots of companies relied.
Many times these unexpected outages are not always easy to diagnose even a rough root cause quickly. Sometimes you don't know if it's your issue, or an upstream issue, and so on.
So, you put out a statement 5 seconds after and outage, and, oops, you misdiagnosed it. Now you get to deal with the shitty customers who want to play 1000 questions about why you told them it was a transfer switch when it turns out it was really a main breaker. Not that it matters much in the end.
You're damned if you do, damned if you don't, but most of the time it's better overall to release an accurate statement later than a wrong statement early.
In both cases, I had clients that I knew were rational and logical. Those are the ones that we would issue an early release to, along the lines of "Something broke and we're on it, more info to follow".
However the irrational types seemed to far outweigh the logical types. Those are the ones where I can tell you from my own first hand experience that overall you are better off giving accurate information late than wrong or incomplete information early.
Today, I run a small bit of a data center as one of my personal side-project companies. I only accept low-overhead experienced clients. They don't give me a lot of headaches, and the 1 single time we had an issue (power related, natch) I told them about it as soon as I found out and thing was a completely forgettable experience for everyone.
My expirience is the opposite. All customers I have dealt with were very happy to receive an acknowledgement early and a post-mortem analysis later.
Most customers don't care one bit about what is wrong, they care about when their service will be restored. Knowing "they are working on it" is already one step up from "Does this affect only me? Have they even noticed, yet?".
The right lesson is "have more than one data center, use automatic failover"
You need to put a switchover device somewhere. And that device (or NOC, or power, or router feeding it), can fail.
Unless you want to give you dns addresses a 10 second ttl, and update them as needed.
Really, how? I would love to do this, but I can't think of any method that does not have a single point of failure somewhere. (And I'm not as big as google with my own network lines, something that a regular budget can do.)
The problem is that it's not a cheap answer, and that your application needs to be able to deal with queries going to either data center at any time.
another option I've seen people use which is not nearly as robust but still 'automatic' is to have two authoritative nameservers; one at each location. But the nameserver at location 1 resolves all A records to IPs at location 1. At location 2, the nameserver is configured to resolve all A records to IP addresses at location 2. (of course, this has all the dns caching issues as setting the TTL to 10 minutes)
A third option would be to have some sort of redirector on a network that is made redundant via BGP, then have some sort of hook so you can change the target of the redirector. But I don't know of anyone offering that service for cheap.
The cost of full failover is first off a doubling of your hardware and datacenter costs. Then add the cost of developing a software system that can handle seamless failover, as well as testing it. By this I also mean the cost of significantly slower release-cycles, because you're not going to release a new feature without testing if it breaks the fail-over, are you?
If this comes out in favor of doing full failover, you're a bank or an airline or similar. You're probably not even Google or Facebook. Chances are that you're not taking infrastructure advice from an online discussion thread.
All this does add development time though and, as you say, for most sites it isn't going to be worth it.
But you do have a good point, that you do need to weigh the cost of a few hours downtime every now and again with the cost of putting yourself in a position where you can avoid such downtime, because yes, sometimes avoiding the downtime is more costly than just taking the hit.
I think at a minimum, though, you need to have off-site backups, and the ability to restore to a new provider if your first provider has a HyperVM level disaster that kills all your data and all the backups at that provider.
At approximately 12:29am CST this morning, our Dallas - Fort Worth (DFW) data center experienced a power disruption, and consequently an interruption of our services. The power disruption was the result of issues during a maintenance effort that was scheduled and expected to be non-impacting.
This summer our DFW facility had power issues, and as a result, we invested significant resources to improve all aspects of our power systems. Last night, during one of these steps, we encountered issues and had a brief loss in power. The power disruption was approximately 5 minutes in duration. Despite this short power disruption, many customers experienced downtime that was significantly longer. Since the power disruption hit the core of many of our cloud services, recovery of full operations required more effort than simple recovery of power. The experience you had last night is not acceptable to us.
Here is what we know about the events:
· The scheduled maintenance was planned to occur from 12:05am - 6:05am CST in our DFW data center. This maintenance is part of a preventative maintenance schedule for several PDUs in UPS Cluster G at the DFW datacenter. The PDUs were down for a total of 5 minutes before power was restored. At approximately 12:29am CST, all PDUs behind UPS Cluster G lost power.
· Although the power outage was very brief (5 minutes), it forced a hard re-boot to occur on a portion of our cloud infrastructure. As our engineers worked to bring hardware back online, we experienced several unforeseen hardware failures. Further complicating our recovery effort, the incident also created internal DNS issues, which caused additional delays. With that said, the vast majority of cloud customers affected by this outage had service restored within one hour's time (many in as little as five minutes); however, depending upon the service, a few customers experienced service interruptions for up to few hours.
Here is how we plan to deal with it:
· We have invested massively in the DFW facility to ensure it delivers at a level you expect from Rackspace - despite last night, we feel very good about our plan and have high confidence in the DFW facility - clearly we have to prove it.
· We are reviewing our maintenance notifications - we typically do not share information on expected non-impacting events, but clearly we need to ensure we calibrate these events and are fully transparent.
· We are reviewing our procedures and systems for quickly resuming cloud operations when an unexpected event like this occurs - unexpected events will happen, our job is to minimize their impacts.
We live by high standards and clearly have not lived up to them. We welcome any feedback. If you would like a call from me, or anyone on our senior team to discuss these issues personally, please reply with a phone number.
We have work to do to earn back your trust. We will not rest until we have.
Apparently the whole thing is is dependent on one guy who is nowhere to be found. Whats up with that?