These kinds of domino effects are one reason why scalability is so hard to get right. It reminds me of precipitation in supersaturated solutions. Everything seems normal until you reach some unforeseen tipping point, and then all hell breaks loose.
I like his little veiled pitch for Google's services when he talks about how easy it was to bring more request routers online given their elastic architecture. It makes me wonder why that elasticity isn't automated -- more routers should automatically be brought online if any routers hit their maximum load.
It's interesting to see this sort of thing happen again and again; I remember the first time I read the explanation behind AT&T's infamous nationwide crash in 1990, which was another domino-effect (a crashed switching station would reboot, broadcast that it was back, and the broadcast would crash other switching stations, which would reboot, broadcast that they were back...), and wondering how the hell you'd plan for or even test for that sort of problem.
I could imagine that this could mask bugs actually. Recently a project I work on started deadlocking. This meant randomly a single request thread would spin endlessly.
Now imagine we had a magic "start more servers" mode. We push new builds once a month, say. We'd just keep spinning up servers automatically without anyone noticing probably. Soon we'd hit some other system bottleneck (I'd bet something like too much DB traffic), and then we'd be off on a wild goose chase wondering what was wrong with the database or something.
(These are semi-hypothetical. As a team we discussed whether we wanted to have our monitoring processes spin up new frontends and decided that we should probably keep humans in the loop ;))
We saw the same thing recently, we became cpu bound before we expected to and as the load balancers compensated it increased the load on the others. All we could do was to start turning off expensive features until it evened out again. And add more capacity as fast as we could of course.
Its nice to see them being so transparent about what happened and how they plan on fixing it in the future. They're obviously working on anticipating problems in the future, but what I wonder about is things like this, where they thought they were covered. How does one go about finding these failure points on systems that span multiple locations? I hope they followup with lessons learned on their quest to improve reliability.
Hours are longer than minutes. Fifty is larger than Forty, which is why $49.99 seems so much cheaper than $50.
But that's the spin, and I suspect you're querying the preference. And you're right - whether something was down for 100 minutes or 1 Hour and 40 minutes wouldn't make much difference to my life.
I admire the transparency but I don't pretend for a second it's the whole story. This happened during work hours and if they indeed did get notified so fast, I'm wondering why it took over 90 minutes to recover.
Also, the outage, for me anyway, seemed to last much longer then the stated 100 minutes. I seem to remember being unable to access GMail for a span of about 3 hours today.
I'm super impressed that they responded so quickly. When a first notification comes in, it is rarely with a fully fleshed out diagnosis. It is usually some runner/monitoring tool sending you an alert saying something is busted. If you get lucky, you had enough monitoring tools across the system to atleast get close to the source. If not, you have some debugging ahead of you. And then, you need to figure out what a good fix is and given that the entire system is in a wobbly state, a bad fix could make the situation worse. And then, you figure out how to rollout the fix and actually make the change. If you're smart, you'll do it in a staged manner and be able to roll back the moment something goes wrong.
In short, these things take time. Going from notification to working fix in 90 minutes for what sounds like a nasty network hardware issue is very good.
Agreed. 100 minutes from first detection through diagnosis to solution is an excellent turnaround time, and although others have mocked them for plugging their architecture in the post, it's a credit to their architecture that the simplest solution -- flood the system with additional capacity -- was available and so quick to implement.
[um, I work for Yahoo, but do you really need a disclaimer on a compliment?]
I try to mention that whenever a major company which competes with Microsoft in some way is involved. Even the most innocent of statements have a way of getting badly misinterpreted, taken out of context, etc.
I just re-read the article and I was mistaken. I thought they determined it was a capacity problem immediately, but it sounds like that's actually what took so long to figure out.
I work for a web company, and a 100 minute outage is considered absolutely outrageous, but of course it happens.
This is probably the most glaring flaw in SaaS and cloud computing. Even the giants go down eventually. Couple that with your own ISP's issues and your potential downtime is doubled.
Seriously. I've worked at a large bank and a small software company and both of their in-house email systems would go down at least a few times a year for all kinds of weird reasons. There are lots of ways for things to go wrong in general.
How does hosting it yourself protect you from these issues (downtime)? The only thing you gain is control over the situation, which may be a worse prospect if you don't have the same level of resources that Google/whoever has to fix it.
I keep my cell phone next to my ear when I go to sleep. It is set to send out a piercing whistle I get an email from a particular address, which indicates that my website has been unavailable for 15 minutes. That whistle has gone off exactly once. (Credit more to Slicehost and my software stack than to me.)
You know who got blasted out of sleep at 3 AM because my email was down? I don't know, but it sure as heck wasn't me.
Luckly half our ops team is in a timezone 9hrs offset from mine, it is extremely rare for either team to have someone woken up in the middle of the night. Sleep is good.
Furthermore, it's so much cheaper/less labor intensive to let Google host your mail. How many important things is your business not doing because your technical folk are futzing with an email server? Errors of commission are easier to identify than errors of omission. Did your business fail to identify a way to attract new customers? Nobody's tracking that, but they love to count up the minutes that email's down.
I seem to be getting buried for a bit of a misunderstanding. If a company is hosting it internally, THEY cannot use that program in the downtime. If the entire cloud is down, NONE of the companies can do any work for a day.
"Well, if my email is down, then my competitor's is too!"
I don't really know how much your argument applies to companies looking for hosted email; if they can't send, they can't send, so it doesn't matter if someone else can't receive. In fact, probably makes it better b/c they look less individually incompetent.
GE's central email farm went down when I interned there a couple summers ago. All 300k+ employees' emails went out.
The business is still affected by the outage if they host locally, there will probably just be more of them. And there's the illusion of control: GE had noone to blame but themselves (and the incompetent contractor who hit the big red button), whereas they could've pinned this on Google and felt worried about external risks and dependencies.
On the corollary, as SaaS becomes more prevalant and popular, these outages and problems will help out and help solve these types of problems so that most of them don't have to happen again in the future.
In other words, this type of outage is pretty unlikely to ever plague Google again.
35 comments
[ 8.3 ms ] story [ 87.4 ms ] threadI like his little veiled pitch for Google's services when he talks about how easy it was to bring more request routers online given their elastic architecture. It makes me wonder why that elasticity isn't automated -- more routers should automatically be brought online if any routers hit their maximum load.
The correct solution, as outlined in their post, is for the servers to slow down when overloaded instead of trying to push load onto another server.
Now imagine we had a magic "start more servers" mode. We push new builds once a month, say. We'd just keep spinning up servers automatically without anyone noticing probably. Soon we'd hit some other system bottleneck (I'd bet something like too much DB traffic), and then we'd be off on a wild goose chase wondering what was wrong with the database or something.
(These are semi-hypothetical. As a team we discussed whether we wanted to have our monitoring processes spin up new frontends and decided that we should probably keep humans in the loop ;))
Compare to:
http://developer.amazonwebservices.com/connect/message.jspa?...
http://en.wikipedia.org/wiki/Northeast_Blackout_of_1965
"it's all just a little bit of history repeating"
But that's the spin, and I suspect you're querying the preference. And you're right - whether something was down for 100 minutes or 1 Hour and 40 minutes wouldn't make much difference to my life.
Also, the outage, for me anyway, seemed to last much longer then the stated 100 minutes. I seem to remember being unable to access GMail for a span of about 3 hours today.
I'm super impressed that they responded so quickly. When a first notification comes in, it is rarely with a fully fleshed out diagnosis. It is usually some runner/monitoring tool sending you an alert saying something is busted. If you get lucky, you had enough monitoring tools across the system to atleast get close to the source. If not, you have some debugging ahead of you. And then, you need to figure out what a good fix is and given that the entire system is in a wobbly state, a bad fix could make the situation worse. And then, you figure out how to rollout the fix and actually make the change. If you're smart, you'll do it in a staged manner and be able to roll back the moment something goes wrong.
In short, these things take time. Going from notification to working fix in 90 minutes for what sounds like a nasty network hardware issue is very good.
[um, I work for Yahoo, but do you really need a disclaimer on a compliment?]
I work for a web company, and a 100 minute outage is considered absolutely outrageous, but of course it happens.
You know who got blasted out of sleep at 3 AM because my email was down? I don't know, but it sure as heck wasn't me.
I did too, and I didn't get much sleep in those days.
Only people who were unfortunate enough to be working on the GMail team from UTC+7, perhaps UTC+6, instead of in Mountain View.
"Well, if my email is down, then my competitor's is too!"
I don't really know how much your argument applies to companies looking for hosted email; if they can't send, they can't send, so it doesn't matter if someone else can't receive. In fact, probably makes it better b/c they look less individually incompetent.
GE's central email farm went down when I interned there a couple summers ago. All 300k+ employees' emails went out.
The business is still affected by the outage if they host locally, there will probably just be more of them. And there's the illusion of control: GE had noone to blame but themselves (and the incompetent contractor who hit the big red button), whereas they could've pinned this on Google and felt worried about external risks and dependencies.
In other words, this type of outage is pretty unlikely to ever plague Google again.