I knew that this was a big event when it happened last Sunday, because the AWS service status page had a yellow triangle rather than a green tick. Usually when they have an outage, they just put a tiny blue 'i' on the green tick...
Heh - I love the image in my head of the flywheel providing a few extra seconds of power to the coffee urn in the Blackwoods warehouse out the back and to all the fan heaters and big screen TVs in Toongabbie - just as Foxtel, Dominos, and Channel 9's Nagios dashboards all start turning red and their ops staff phones start beeping.
I love reading about problems like these, it's great that Amazon is forthcoming about them. There's always some new wrinkle.
E.g. in this case, in normal operation, power from the utility power grid spins a flywheel. When the grid fails, the flywheel provides a holdover until Amazon's diesel generators can start.
But in this failure the voltage from the grid sagged, rather than going away completely. The breaker isolating the flywheel from the grid didn't open quickly enough. So power from the flywheel was sent out to the grid. It didn't succeed in powering the grid for very long. Oops.
Something else must have failed or was not properly configured, because backup Diesel generators should kick-in after 2-15 seconds of voltage drop, regardless of the flywheel. The flywheel is used in critical systems to cover only that <1m gap.
Amazon addresses that in their report. Each UPS is fed by generator power and grid power. Because the UPSes had been forced to try to supply the grid, and because they are giant spinning weights that you really don't want to go wrong and kill someone or destroy property, they did a safety inspection before powering them back up, which meant a delay before the facility could be supplied by generator power.
This also baffles me. Here are the times converted to AEST:
Sun July 5th
3:25 PM -- Initial Power outage
4:42 PM -- Instance launching in unaffected AZ's restored
4:46 PM -- Power Restored
6:00 PM -- 80% of instances recovered
7:49 PM -- DNS recovered
Mon July 6th
1:00 AM -- Almost all instances recovered
I'm making a throwaway so as not to be associated with "conspiracy theories", which is a ridiculous charge given that I'm part of half a dozen "conspiracies" and so is any reader who's ever been part of any organization, of any size, including a simple startup with six customers. As we know tens of thousands of people belong to organizations whose missions would be described in the most neutral possible, and only possible, terms as conspiracies, by any definition.
So anyway this really pisses me off. Because AWS is a service, it's the backbone of the next net. Where you summarize:
>Or, in summary, "Uninterruptable power supply is actually interrupted."
I summarize it as "article is a series of over the top stories that are to a cover story as santa claus is to dad."
Come on. If someone's UPS failed, they would be PISSED, the UPS literally has ONE JOB. They would fire the UPS company and bitch and moan about it and completely decry fly wheels etc. They wouldn't be like, "meh, it sagged."
This is like "here's our fully redundant, great architecture. Explain how it all went down." I mean you can explain fucking anything that way. Here's why a nuclear missile disappeared. First of all, the guy who was in charge of it had a really bad headache, so he stayed home. Normally it would have meant the place would be left empty. But the cleaning company had just gone bankrupt, and the place hadn't been cleaned for weeks. Now normally it's no problem if there's no cleaning for a few weeks in a nuclear silo, but in this case there was a fear that a previous worker had some esoteric disease so it was important to resterilize the place every day. so the decision was made to let a replacement company in. Now, normally a replacement cleaning company has to go through a background check, but in this case there was a national outage over the infrastructure that the background check runs under, so rather than perform the background check, more crapity crap crap, then a decision was made to allow a trusted employee to just go and perform basic cleaning. The trusted employee had a previous heart condition that led to a sudden heart attack causing him to fall right in the middle of the automatic doors, which opened, and closed, and opened, and closed, in a cycle that would have been comic if not for hte dead body. Now normally there's more than one automatic door between a nuclear silo and the local 7-11 but in this case the inner doors had just been removed for replacement by an even better and more secure version so it's just bad timing. anyhoo luckily some local teenagers were walking by and noticed the automatic doors pummelling this poor dead body so they did what could be reasonably expected and went inside to tell everyone, hey, there's this dead guy being pumelled by doors opening and closing. They didn't find anyone inside, because as we said the whole place was empty, so they looked for a number. There's no cell phone reception in the area because we disable it whenever we shutter up shop at our nuclear silos, so they grabbed the first thing they saw that had a telephone number on it, which happened to be an 1200 pound tactical nuclear warhead with a prominent emergency number, and unfortunatey they didn't have anything to write with (they were skaters making a skating video and so only one of them had their phone on them, so they wouldnt' damage it, and that one phone was full from all the videos they took.) They didn't take the time to trim the video since htey thought it would just be faster to take the warhead and hen return it later, so they just put it onto their truck with a dolly and started going home so they could skype for some help. They didn't attach the warhead very securely and didn't notice but it did drop at some point on their way home, however by the time they noticed they saw that one of them had actua...
I'm a bit dubious about their "if you used multi-AZ you'll be fine" when I had multiple outages in a multi-AZ Elastic Beanstalk application of over an hour. Methinks the load balancers aren't as magical as they'd like to make out.
> Methinks the load balancers aren't as magical as they'd like to make out.
Agreed. I'm still put off by the fact that ELBs specifically can not handle a sudden spike in traffic orders of magnitude higher than the previous rate. They fall on their face, bad. If you expect a spike like that to happen, you literally have to submit a ticket and ask them to pre-warm your ELB...
Our instances with ap-southeast-2 were out for around 12 hours. We used multiple availability zones and it didn't prevent downtime at all. It's very interesting the difference between AWS and Google outage responses. AWS is down for 12+ hours for some customers, force each customer to chase service level credits and sign off the postmortem with a nameless & faceless "-The AWS Team". Not one person at AWS was willing to take responsibility for this failure.
Whereas Google was recently down for less than 18 minutes. A VP at Google sent an email advising all affected customers, posted continuous updates to their status page, sent a further apology email at the conclusion, posted a service credit exceeding the SLA to all customers in the zone (without forcing customers to chase this themselves with billing) and lastly wrote one of the most well written post mortems I've ever seen. AWS has much to learn from Google about how to handle outages properly.
I wonder if due to the scale of AWS and certain AWS customers, if AWS signs a post mortem with -Fred would a very large AWS customer have the pull to say "Amazon, fire Fred"? Just curious if anonymous postmortems are company policy at certain places and why that might be.
Agreed. I have a couple of clients that have a pretty substantial AWS spend, but the cost of switching to Azure is too high compared to the difference in offerings. You don't want to spend tens of thousands of dollars of developer time and risk switching datacenters for a small improvement.
It's difficult to change when you have reserved instances. A business decision yesterday might not be the best decision today. That being said, I think overall AWS is very good but Google is definitely starting to create some real competition which is positive for everyone.
> We used multiple availability zones and it didn't prevent downtime at all.
Can you explain this a little more? Amazon says this only affected one AZ, and they specifically note:
For this event, customers that were running their applications across multiple
Availability Zones in the Region were able to maintain availability throughout
the event.
Apart from one internal project which mistakenly had all it's app server instances in -2b (ooops!) - all my production mobile app backends are spread across the 3 Sydney AZs. That's a few dozen EC2 app servers across about 15 projects.
My monitoring reported a worst case of 57 seconds of degraded connectivity - which was an instance in -2b going offline and the ELB not taking it out of the rotation very quickly, the app running on that had interruption, but only while waiting for the timeouts. Crashlytics and GA crash reporting didn't bat an eyelid... I had under 70 users active at the time, 1/3rd of them may have seen a minute or less of loading spinner if they'd fired of a UI blocking api call during those 57 seconds. I'm not looking _super_ closely, but nothing I'm monitoring apart from EC2 - like RDS, S3, ELB, SNS - showed _any_ glitches (I'd _probably_ have caught even single digit second problems for _some_ of that...)
I'm actually quite happy with how everything went - we don't go to any particular heroic lengths to ensure HA or uptime, we just follow recommended best practice, and at least in this outage, that worked out fine for us (except for that project where all the app servers were in -2b, and I'm happy to wear that as our fuckup)
I independently monitor availability of 150 public cloud services and only observed 1.73 hours downtime for this event. This is the first EC2 outage I've observed in any region for over 6 months. According to my stats, in 2015 EC2 was highly available with 6 of 9 regions (including every US and EU region) having no outages, and total average service availability of 99.998% (78% of downtime in sa-east-1). I haven't observed a single outage in us-west-2 or eu-west-1 in nearly 3 years. This compared to 16-33 minutes of downtime in every region for GCE in 2015, and total average availability of 99.995%. Additionally, since 2013 I've never observed a global EC2 outage like the 4/11/2016 GCE event.
Most of the criticism seemed to be centered around communication/PR so statistics don't really address that. Definitely important when considering a provider though.
While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hours that's a really significant difference.
I guess it really depends on how you monitor and how comprehensive it is. Do you monitor from multiple ISPs on different network paths in multiple regions/countries? Do you monitor each of the services under different load conditions and monitor multiple accounts? Sometimes "up" only tells part of the story.
2 hours aligns with the postmortem, 12 hours does not. The monitoring is based on a sampling of instances running in each service and region. Outages are verified from multiple network paths. While not comprehensive, over the years this has been generally accurate because most outage events have been network or power related, thus impacting most or all instances in the affected data center. There have been some isolated hardware failures, but they are rare.
I may be mistaken, but 2 hours only aligns with their post mortem for 80% of the impacted instances. By their own account there were instances impacted until 8AM and a small number after that even.
Your monitoring sounds really comprehensive. That's a very cool way to advertise the service it's built on. Do you monitor service providers for outages that reduce capacity or increase latencies but otherwise the service is "up"?
The stats are simply to provide some context. I agree criticism is centered around communication/PR, but also seems to imply the outage was of the same magnitude or worse than GCE's global outage, which is inaccurate. Per HA best practices, multi-region load balancing or failover would have adverted downtime during this event, whereas it would not have during the global GCE outage. Had this been a global outage, I think the postmortem and customer outreach would have been much better.
Do you have proof? Your claims directly contradict what was written in their post mortem both in terms of time and scope. AWS might be using nice wording but I can't see them lying that much about the scope of the incident.
>> The specific signature of this weekend’s utility power failure resulted in an unusually long voltage sag (rather than a complete outage)
It is false to assume that the state of the electrical supply is either on or off. This may come as a surprise, but not to me. In 2008, Eskom (South Africa's electricity suppliers) experienced similar faults. The mains supply voltage is 220v here. At one point, some devices started to fail in my house, and others, such as lights, continued to work, but significantly dimmer. We measured 180v at the plugs. There were similar outages in my area last year, where an outright cut-off was preceded by voltage drops. This outage is interesting because it is an example of a bug owing to false assumptions!
There have also been incidences where certain cables have been stolen [1] and that has caused the opposite: voltage spikes.
[1] I couldn't tell you which, or what kind, but I remember it has something to do with "the neutral"
It doesn't sound like they were treating it as binary, they have breakers in place for brownout detection too -- those breakers just weren't triggered fast/early enough in the brown out.
39 comments
[ 3.3 ms ] story [ 92.9 ms ] threadE.g. in this case, in normal operation, power from the utility power grid spins a flywheel. When the grid fails, the flywheel provides a holdover until Amazon's diesel generators can start.
But in this failure the voltage from the grid sagged, rather than going away completely. The breaker isolating the flywheel from the grid didn't open quickly enough. So power from the flywheel was sent out to the grid. It didn't succeed in powering the grid for very long. Oops.
(Mutter, mutter, … something about Americans and their timezones … and northern hemispherians and their seasons …)
I am reminded of 'The Event' from That Mitchell and Webb Look [0]. We don't talk about The Event.
https://www.youtube.com/watch?v=wnd1jKcfBRE
So anyway this really pisses me off. Because AWS is a service, it's the backbone of the next net. Where you summarize:
>Or, in summary, "Uninterruptable power supply is actually interrupted."
I summarize it as "article is a series of over the top stories that are to a cover story as santa claus is to dad."
Come on. If someone's UPS failed, they would be PISSED, the UPS literally has ONE JOB. They would fire the UPS company and bitch and moan about it and completely decry fly wheels etc. They wouldn't be like, "meh, it sagged."
This is like "here's our fully redundant, great architecture. Explain how it all went down." I mean you can explain fucking anything that way. Here's why a nuclear missile disappeared. First of all, the guy who was in charge of it had a really bad headache, so he stayed home. Normally it would have meant the place would be left empty. But the cleaning company had just gone bankrupt, and the place hadn't been cleaned for weeks. Now normally it's no problem if there's no cleaning for a few weeks in a nuclear silo, but in this case there was a fear that a previous worker had some esoteric disease so it was important to resterilize the place every day. so the decision was made to let a replacement company in. Now, normally a replacement cleaning company has to go through a background check, but in this case there was a national outage over the infrastructure that the background check runs under, so rather than perform the background check, more crapity crap crap, then a decision was made to allow a trusted employee to just go and perform basic cleaning. The trusted employee had a previous heart condition that led to a sudden heart attack causing him to fall right in the middle of the automatic doors, which opened, and closed, and opened, and closed, in a cycle that would have been comic if not for hte dead body. Now normally there's more than one automatic door between a nuclear silo and the local 7-11 but in this case the inner doors had just been removed for replacement by an even better and more secure version so it's just bad timing. anyhoo luckily some local teenagers were walking by and noticed the automatic doors pummelling this poor dead body so they did what could be reasonably expected and went inside to tell everyone, hey, there's this dead guy being pumelled by doors opening and closing. They didn't find anyone inside, because as we said the whole place was empty, so they looked for a number. There's no cell phone reception in the area because we disable it whenever we shutter up shop at our nuclear silos, so they grabbed the first thing they saw that had a telephone number on it, which happened to be an 1200 pound tactical nuclear warhead with a prominent emergency number, and unfortunatey they didn't have anything to write with (they were skaters making a skating video and so only one of them had their phone on them, so they wouldnt' damage it, and that one phone was full from all the videos they took.) They didn't take the time to trim the video since htey thought it would just be faster to take the warhead and hen return it later, so they just put it onto their truck with a dolly and started going home so they could skype for some help. They didn't attach the warhead very securely and didn't notice but it did drop at some point on their way home, however by the time they noticed they saw that one of them had actua...
* http://www.smh.com.au/national/australias-wild-weather-sydne...
* http://www.abc.net.au/news/2016-06-07/sydney-weather-storm-d...
* http://www.sbs.com.au/news/gallery/pictures-wild-weather-sav...
Agreed. I'm still put off by the fact that ELBs specifically can not handle a sudden spike in traffic orders of magnitude higher than the previous rate. They fall on their face, bad. If you expect a spike like that to happen, you literally have to submit a ticket and ask them to pre-warm your ELB...
Whereas Google was recently down for less than 18 minutes. A VP at Google sent an email advising all affected customers, posted continuous updates to their status page, sent a further apology email at the conclusion, posted a service credit exceeding the SLA to all customers in the zone (without forcing customers to chase this themselves with billing) and lastly wrote one of the most well written post mortems I've ever seen. AWS has much to learn from Google about how to handle outages properly.
Compared to the sole region ap-southeast-2 (for you) in AWS's case. (though it was a longer outage).
That being said, I 1000% agree that Google cloud platform's response to fix issues, postmortems and actions to make things right are top notch.
Can you explain this a little more? Amazon says this only affected one AZ, and they specifically note:
Apart from one internal project which mistakenly had all it's app server instances in -2b (ooops!) - all my production mobile app backends are spread across the 3 Sydney AZs. That's a few dozen EC2 app servers across about 15 projects.
My monitoring reported a worst case of 57 seconds of degraded connectivity - which was an instance in -2b going offline and the ELB not taking it out of the rotation very quickly, the app running on that had interruption, but only while waiting for the timeouts. Crashlytics and GA crash reporting didn't bat an eyelid... I had under 70 users active at the time, 1/3rd of them may have seen a minute or less of loading spinner if they'd fired of a UI blocking api call during those 57 seconds. I'm not looking _super_ closely, but nothing I'm monitoring apart from EC2 - like RDS, S3, ELB, SNS - showed _any_ glitches (I'd _probably_ have caught even single digit second problems for _some_ of that...)
I'm actually quite happy with how everything went - we don't go to any particular heroic lengths to ensure HA or uptime, we just follow recommended best practice, and at least in this outage, that worked out fine for us (except for that project where all the app servers were in -2b, and I'm happy to wear that as our fuckup)
https://cloudharmony.com/status-for-aws
While the numbers are nice, if some outages only impact a subset of customers and your monitoring accounts aren't one of them it's hard to determine how good your monitoring data really is. If he was impacted by a 12 hour outage and you only show ~2 hours that's a really significant difference.
I guess it really depends on how you monitor and how comprehensive it is. Do you monitor from multiple ISPs on different network paths in multiple regions/countries? Do you monitor each of the services under different load conditions and monitor multiple accounts? Sometimes "up" only tells part of the story.
Your monitoring sounds really comprehensive. That's a very cool way to advertise the service it's built on. Do you monitor service providers for outages that reduce capacity or increase latencies but otherwise the service is "up"?
It is false to assume that the state of the electrical supply is either on or off. This may come as a surprise, but not to me. In 2008, Eskom (South Africa's electricity suppliers) experienced similar faults. The mains supply voltage is 220v here. At one point, some devices started to fail in my house, and others, such as lights, continued to work, but significantly dimmer. We measured 180v at the plugs. There were similar outages in my area last year, where an outright cut-off was preceded by voltage drops. This outage is interesting because it is an example of a bug owing to false assumptions!
There have also been incidences where certain cables have been stolen [1] and that has caused the opposite: voltage spikes.
[1] I couldn't tell you which, or what kind, but I remember it has something to do with "the neutral"