"We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it."
Interesting bit on recovery w.r.t. the electrical grid
> flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ...
I wish there was a bit more detail in here. What's the worst case there? Brownouts, exploding transformers? Or less catastrophic?
I don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they were idle because they had no traffic.
Do large providers like Facebook really provision less power than their servers would require at 100% utilization? Seems like they could just use fewer servers with power sized at 100% if their power system going to constrain utilization anyway?
I don't know the answer. But it's not too uncommon, in general, to provision for reasonable use cases plus a margin, rather than provision for worst case scenario.
All of the components in the supply chain will be rated for greater than max load, however power generation at grid scale is a delicate balancing act.
I’m not an electrical engineer, so the details here may be fuzzy, however in broad strokes:
Grid operators constantly monitor power consumption across the grid. If more power is being drawn than generated, line frequency drops across the whole grid. This leads to brownouts and can cause widespread damage to grid equipment and end-user devices.
The main way to manage this is to bring more capacity online to bring the grid frequency back up. This is slow, since spinning up even “fast” generators like natural gas can take on the order of several minutes.
Notably, this kind of scenario is the whole reason the Tesla battery in South Australia exists. It can respond to spikes in demand (and consume surplus supply!) much faster than generator capacity can respond.
The other option is load shedding, where you just disconnect parts of your grid to reduce demand.
Any large consumers (like data center operators) likely work closely with their electricity suppliers to be good citizens and ramp up and down their consumption in a controlled manner to give the supply side (the power generators) time to adjust their supply as the demand changes.
Note that changes to power draw as machines handle different load will also result in changes to consumption in the cooling systems etc. making the total consumption profile substantially different coming from a cold start.
You're talking about the grid, the OP was talking about datacenter infrastructure -- which one is the weak link?
If a datacenter can't go from idle (but powered on) servers to fully utilized servers without taking down the power grid, then it seems that they'd have software controls in place to prevent this, since there are other failure modes that could cause this behavior other than a global Facebook outage.
Unfortunately the article doesn’t provide enough explicit detail to be 100% sure one way or the other, however my read is that it’s probably the grid.
> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk.
“Electrical systems” is vague and could refer to either internal systems, external systems or both.
That said, if the DC is capable of running under sustained load at peak (which we have to assume it is, since that’s its normal state when FB is operational) it seems to me like the externality of the grid is the more likely candidate.
In terms of software controls preventing this kind of failure mode, they do have it - load shedding. They’ll cut your supply until capacity is made available.
In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand.
Now consider that, on modern servers, most of the power consumption is from the CPU, and also there's a significant difference on the amount of power consumed between 100% CPU and idle. Imagine for instance 1000 servers (a single rack can hold 40 servers or more), each consuming 2kW of power at full load, and suppose they need only half that at idle (it's probably even less than half). Suddenly switching from idle to full load would mean 1MW of extra power has to be generated; while the generators are catching up to that, the voltage drops, which means the current increases to compensate (unlike incandescent lamps, switching power supplies try to maintain the same output no matter the input voltage), and breakers (which usually are configured to trip on excess current) can trip (without breakers, the wiring would overheat and burn up or start a fire).
If the load changes slowly, on the other hand, there's enough time for the governor on the generators to adjust their power source (opening valves to admit more water or steam or fuel), and overcome the inertia of their large spinning mass, before the voltage drops too much.
If it's all at once at the end of one leg and unplanned? Yes.
The question is somewhat similar to a thought experiment. If a ship is docked and loading cargo, is it a good idea to use all the cranes to suddenly fill up one outer side of the ship?
No. All balancing authorities are required to keep a certain amount of "spinning reserve" available for fast adjustments like this. But if I do it and the next guy does it and a transmission like is down and...etc
>generated; while the generators are catching up to that, the voltage drops, which means the current increases to compensate...
Close- you won't see an increase in load of a synchronous machine operating at constant throttle manifest as a voltage sag, you'll see it manifest as a decrease in frequency (this generators literally slow down, like a guy on a bike going uphill). Voltage sags are more related to transmission line phenomenon.
Disk arrays have been staggering drive startup for a long time for this reason. Sinking current into hundreds of little starting motors simultaneously is a bad idea.
One case is automated protection systems in the grid detecting a sudden hop of current and assuming an isolation failure along the path - basically, not enough current to trip the short-circuit breakers, but enough to raise an alarm.
This isn't really a thing. Transmission and distribution protection doesn't operate on any kind of di\dt basis, other than those defined by overcurrent, in which case the line trips. A sudden increase in load will just manifest in ACE (area control error) as a load imbalance and be dealt with by increasing generation from the spinning reserve that the balancing authority is required to have on hand.
Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker.
A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal to about 20% of the load behind the distribution transformer. That quantity is about as much as an aluminum smelter switching on or off.
But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly.
Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?
I assume they do have backup generators, though I don’t know.
However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.
Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.
My girlfriend works in a large grid operator (in Europe). According to her there are lots of regulations and contracts on the grid operators about how they must handle reliability. So it's unlikely that Facebook would be liable if this took down half the country, because then it was the grid operator not living up to their agreements on reliability.
There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...
For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer.
Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.
I guess a DC would have UPS/battery power on hand to cover an instantaneous brown out. Then the generators could be on and restoring battery power while running the DC.
For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly
If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec.
You'd typically have some short-term power protection to keep your datacenter running until the generators start.
I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.
> That quantity is about as much as an aluminum smelter switching on or off.
Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use...
That's pretty interesting, I'm sure those aluminium foundries would need to be careful about turning the power on as well.
Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness.
I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.
If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low.
I used to do electrical compliance testing in a previous life, with brown out testing being one of our safety tests. You would drape a piece of cheese cloth over the power supply and slowly ramp the line voltage down. At the time, the power supplies didn't have good line side voltage monitoring. There was almost always smoke, and sometimes cheese cloth fires. Since this was safety testing, pass/fail was mostly based on if the cheese cloth caught fire, not if the power supply was damaged.
All standard computer components require a regulated voltage, then they consume power as a consequence of their operation. The steady voltage is required because the transistors in ICs will break down if voltages go too high, or stop operating if they go too low. Forcing something like an IC to always use the same amount of power, even if it were idle, isn't really possible, because nobody would build it that way.
I’m very close with someone who works at a FB data center and was discussing this exact issue.
I can only speak to one problem I know of (and am rather sure I can share): a spike might trip a bunch of breakers at the data center.
BUT, unlike me at home, FBs policy is to never flip a circuit back on until you’re positive of the root cause of said trip.
By itself that could compound issues and delay ramp up time as they’d work to be sure no electrical components actually sorted/blew/etc. A potentially time sucking task given these buildings could be measured in whole units of football fields.
> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally
of course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.
Not really, there’s essentially always a button that blows everything up. Catastrophic failures usually end up being a large set of safety systems malfunctioning which would otherwise prevent the issue when that button is pressed.
But yes, for these types of problem, the ultimate fault is never “that guy Larry is an idiot”, it takes a large team of cooperating mistakes.
I doubt Facebook engineers are free-typing commands on Bash, so it’s probably not an individual error. More likely to be a race condition or other edge case that wasn’t considered during a review. This might be a script that’s run 1000s of times before with no problems.
Back in Ye Old Dark Ages, I caused a BIG Google outage by running a routine maintenance script that had been run dozens if not hundreds of times before.
Turns out the underlying network software had a race condition that would ONLY be hit if the script ran at the exact same time as some automated monitoring tools polled the box.
At FAANG scale, "one in a million" happens a lot more often than you'd think.
> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses.
What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel like I'm missing something.
With an outage this big, even a post for a technical audience will get read by non-technical people (including journalists), so I'm sure it helps to include details like this.
Both those groups of people. I imagine, they would either be accused of it being either too complicated or dumbed down, so they do both in the same article.
Huh? I would hardly describe this as technical. Someone with a high school education can read it and get the gist. It's actually somewhat impressive how it toes the line between accessibility and 'just detailed enough'.
I'm guessing it has multiple target audiences. Those that won't understand some of the technical jargon (e.g., "IP addresses") will still be able to follow the general flow of the article.
Those of us who are familiar with the domain of knowledge, on the other hand, get a decent summary of events.
It's a balancing act. I think the article does a good enough job of explaining things.
I'm reading your comment as a form of "feigning surprise", in other words a statement along the lines of "I can't believe target audience doesn't know about x concept".
There are plenty of technical people (or people employed in technical roles) who don't understand how DNS works. For example, I field questions on why "hostname X only works when I'm on VPN" at work.
FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no requirement) to provide the general public with more info than they want subject to any legal requirement. If the government wants more (and are entitled to more info) they can ask for it and FB can decide if they are required to comply.
FB is a business. Their customers are not the tech community looking to get educated and avoid issues themselves at their companies or (as mentioned) be entertained. And their customers (advertisers or users) can decide if they want to continue to patronize FB.
I always love on HN seeing 'hey where is the post mortem' as if it's some type of defacto requirement to air dirty laundry to others.
If I go to the store and there is not paper towels there I don't need to know why there are no towels and what the company will do going forward to prevent any errors that caused the lack of that product. I can decide to buy another brand or simply take steps to not have it be an issue.
> If I go to the store and there is not paper towels there I don't need to know why there are no towels
You don't _need_ to know, but it's human to want to know, and it's also human to want to satisfy other human's curiosity, especially if it doesn't bring any harm to you.
Also, your post is not really answering any of GP's questions. I presume you wanted to say that FB doesn't _owe_ any explanation to us, but the GP asked, as they already provided one, to whom is it addressed.
That the main form of personal communication for hundreds of millions of users is down and there is no mandatory reporting is irresponsible. That Facebook is a business does not mean that they do not have responsibilities towards society.
Facebook is not your local supermarket, it has global impact.
One would imagine a large local supermarket going down would owe the people it serves some explanation. That's where their food comes from.
At this point, I am completely sick of the pro-corporate rhetoric to let businesses do whatever they want. They exist to serve the public and they should be treated as such.
> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally
From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun.
At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.
Seems like it. It's kinda like typing hostname and accidentally poking your yubikey (Not that I've done that...) or the date command that both let's you set the date and format the date
For context, parent comment is trying to decipher this heavily-PR-reviewed paragraph:
> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust).
I'd be curious to see what systemic changes happen at FB as a result, if any.
To expand on why this made me think of the Google outage:
It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change.
"But wait, a global backbone isolation? Google wasn't totally down," you might say. That's because Google has two (primary) backbones (B2 and B4), and only B4 was isolated, so traffic spilled over onto B2 (which has much less capacity), causing heavy congestion.
> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication.
...
Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end."
Ultimately, that faulty command changed router configuration globally.
The Google outage was triggered by a configuration change due to an automation system gone rogue. But hey, it too was triggered by a human issuing a command at some point.
I'm inclined to believe the later post as they've had more time to assess the details. I think the point of the earlier post is really to say "we weren't hacked!" but they didn't want to use exactly that language.
This is kind of like Chernobyl where they were testing to see how hot they could run the reactor to see how much power it could generate. Then things went sideways.
The Chernobyl test was not a test to drive the reactor to the limits, but actually a test to verify that the inertia of the main turbines is big enough to drive the coolant pumps for X amount of time in the case of grid failure.
As already said the test was about something entirely different. And the dangerous part was not the test itself, but the way they delayed the test and then continued to perform it despite the reactor being in a problematic state and the night shift being on duty, who were not trained on this test. The main problem was that they ran the reactor at reduced power long enough to have significant xenon poisoning, and then put the reactor at the brink when they tried to actually run the test under these unsafe conditions.
I'd say the failure at Chernobyl was that anyone who asked questions got sent to a labor camp and the people making the decisions really had no clue about the work being done. Everything else just stems from that. The safest reactor in the world would blow up under the same leadership.
At first i thought it was inappropriate hyperbole to compare Facebook to Chernobyl, but then i realized that i think Facebook (along with twitter and other "web 2.0" graduates) has spread toxic waste across far larger of an area than Chernobyl. But I would still say that it's not the _outage_ which is comparable to Chernobyl, but the steady-state operations.
Google also had a runaway automation outage where a process went around the world "selling" all the frontend machines back to the global resource pool. Nobody was alerted until something like 95% of global frontends had disappeared.
This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expanding it. I.e. if your command was supposed to be `drain -cell xx` to locally turn-down a small resource pool but `drain` without any arguments drains the whole universe, you have developed a tool which is too dangerous to exist.
As FB opines at the end, at some point, it's a trade-off between power (being able access / do everything quickly) and safety (having speed bumps that slow larger operations down).
The pure takeaway is probably that it's important to design systems where "large" operations are rarely required, and frequent ops actions are all "small."
Because otherwise, you're asking for an impossible process (quick and protected).
SREs live in a dangerous world, unfortunately. It's entirely possible the "tool" in question is a shell script that gets fed a list of bad cells but some bug causes it to get a list of all the cells instead.
Some tools are well engineered, capable of the Sisyphean task of globally deploying updates but others are rapid prototypes that, sure, are too dangerous to exist, but the whole point of SREs being capable programmers is that the work has problems that are most efficiently solved with one-off code that just isn't (because it can't be) rigorously tested before being used. You can bet there was some of that used in recovering from this incident. (I'm sure there were many eyes reviewing the code before being run, but that only goes so far when you're trying to do something that you never expected, like having to revive Facebook.)
The other problem is scale: the standard "save me" for tools like this is a --doit and --no-really-i-mean-it and defaulting to a "this is what I would've done" mode. That falls apart the moment the list of actions is longer then the screen but you're expecting that: after all how can you really tell the difference unless the console scrolls for a really long time?
There's solutions to that, but of course these sorts of tools all come into existence well before the system reaches a size where how they work becomes dangerous.
I feel like this explains so much about why the gcloud command works the way it does. Sometimes feels overly complicated for minor things, but given this logic, I get it.
> I'd be curious to see what systemic changes happen at FB as a result, if any.
If history is any guide, Facebook will decide some division charged with preventing problems was an ineffective waste of money, shut it down, and fire a bunch of people.
> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust).
Bullshit.
I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of personal and professional observations of people getting fucked over by google and having absolutely nobody they could contact to try and resolve the situation.
You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you.
The fundamental problem googlers have is that they all think they're so smart/good at what they do, it just doesn't seem to occur that they could have possibly screwed something up, or something could go wrong or break, or someone might need help in a way your help page authors didn't anticipate...and people might need to get ahold of an actual human to say "shit's broke, yo." Or worse, none of you give a shit. The company certainly doesn't. When you've got near monopoly and have your fingers in every single aspect of the internet, you don't need to care about fucking your customers over.
I cannot legitimately name a single google product that I, or anyone I know, likes or wants to use. We just don't have a choice because of your market dominance.
Maps, Mail, Drive, Scholar, and Search are all the best or near the best available. That doesn’t mean I like every one of them or I wouldn’t prefer others, but as far as I can tell the competition doesn’t exist that works better.
GCP and Pixel phones are a toss-up between them and competitors.
It isn’t market dominance, nobody has made anything better.
Search is famously kind of bad the last few years, but even Maps isn’t that great.
(Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)
Hi there. I'm a Googler and I've directly interfaced with a nontrivial number of customers such that I alone have interfaced with more than 0.000001% of the entire world population.
All you need to do is browse any online forum, bug tracker, subreddit dedicated to a consumer-facing Google product to know that Google does not give a rat's ass about customer service. We know the customer is ultimately not the consumer.
> leadership read out emails from customers telling us how we let them down and lost their trust).
That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.
> a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust
Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1].
"Blameless Postmortem" does not mean "No Consequences", even if people often want to interpret it that way. If an organization determines that a disconnect between ground work and a customer's experience is a contributing factor to poor decision making then they might conclude that making engineers more emotionally invested in their customers could be a viable path forward.
Relentless customer service is never going to screw you over in my experience... It pains me that we have to constantly play these games of abstraction between engineer and customer. You are presumably working a job which involves some business and some customer. It is not a fucking daycare. If any of my customers are pissed about their experience, I want to be on the phone with them as soon as humanly possible and I want to hear it myself. Yes, it is a dreadful experience to get bitched at, but it also sharpens your focus like you wouldn't believe when you can't just throw a problem to the guy behind you.
By all means, put the support/enhancement requests through a separate channel+buffer so everyone can actually get work done during the day. But, at no point should an engineer ever be allowed to feel like they don't have to answer to some customer. If you are terrified a junior dev is going to say a naughty phrase to a VIP, then invent an internal customer for them to answer to, and diligently proxy the end customer's sentiment for the engineer's benefit.
I think of this is terms of empathy: every engineer should be able to provide a quick and accurate answer to "What do our customers want? And how do they use our product?"
I'm not talking esoterica, but at least a first approximation.
Because we as engineers create software for our customers, and if you don't understand who your customers are how can you create software that actually suits their needs?
From the SRE book: "For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the 'wrong' thing prevails, people will not bring issues to light for fear of punishment."
If it's really the case that engineers are lacking information about the impact that outages have on customers (which seems rather unlikely), then leadership needs to find a way to provide them with that information without reading customer emails about how the engineers "let them down", which is blameful.
Furthermore, making engineers "emotionally invested" doesn't provide concrete guidance on how to make better decisions in the future. A blameless portmortem does, but you're less likely to get good postmortems if engineers fear shaming and punishment, which reading those customer emails is a minor form of.
I work at Google and have written more than a few blameless postmortems. You don't need to quote things to me.
Is what was described above "finger pointing or shaming"? I don't work in TI so I didn't experience this meeting but it doesn't seem like it is. It also doesn't sound to me like this was the only outcome, where the execs just wagged their fingers at engineers and called it a day. Of course there'd be all sorts of process improvements derived from an understanding of the various system causes that led to an outage.
Yes, if I were made to attend a mandatory training in which my leaders read customer emails saying that the outage caused them to lose trust in the company, I would feel ashamed. That was surely the goal of that exercise. The fact that there were also process improvements doesn't make it any less wrong.
Thankfully, other comments in this thread suggest that this is not how Google normally does things.
I don't think Google really cares about listening to their users. I have spent more than 6 hours trying to get simple warranty issues resolved. I wish they had to feel the pain of their actions and decisions.
Not the original googler responding, but I have never experienced what they describe.
Postmortems are always blameless in the sense that "Somebody fat fingered it" is not an acceptable explanation for the causes of an incident - the possibility to fat finger it in the first place must be identified and eliminated.
> Not the original googler responding, but I have never experienced what they describe.
I have also never experienced this outside of this single instance. It was bizarre, but tried to reinforce the point that something needed to change -- it was the latest in a string of major customer-facing outages across various parts of TI, potentially pointing to cultural issues with how we build things.
(And that's not wrong, there are plenty of internal memes about the focus on building new systems and rewarding complexity, while not emphasizing maintainability.)
Usually mandatory trainings are things like "how to avoid being sued" or "how to avoid leaking confidential information". Not "you need to follow these rules or else all of Cloud burns down; look, we're already hemorrhaging customer goodwill."
As I said, there was significant scar tissue associated with this event, probably caused in large part by the initial reaction by leadership.
The most remarkable thing about this is learning that anyone at Google read an email from a customer. Given the automated responses to complaints of account shutdowns, or complaints about app store rejections, etc, this is pretty surprising.
I'd love to get a read receipt each time someone at Google has actually read my feedback. Then it might be possible to determine whether I'm just shaking my fists at the heavens or not.
Was this a way to delete a lot of evidence before shit really hit the fan?'
After reading this, I can't help but feel this was a calculated move.
It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'.
I'm not usually that cynical, but the timing of it combined with facebook's lengthy abusive relationship with customers' privacy (and what kind of company morals that implies) makes me think that it's definitely a possibility.
the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.
this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.
> The backbone is the network Facebook has built to connect all our computing facilities together, which consists of tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers.
This makes it sound like Facebook has physically laid "tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers". Is this in fact true?
Although the difference here is that loosing connection and out-of-band for a single data center shouldn't be as catastrophic for Facebook, so your examples would be tolerable?
That's the trick, though: if you don't do that level of planning for all of your datacenters and POPs (and fiber huts out in the middle of nowhere), it's inevitable that the one you most need to access during an outage will be the one where your OOB got backhoe'd.
heh i toured a large data center here in dallas and listened to them brag about all the redundant connectivity they had while standing next to the conduit where they all entered the building. One person, a pair of wire cutters, and 5 seconds and that whole datacenter is dark.
Two is One, One is None. There are absolutely ways around this, it's called redundancy. The marginal cost of laying an extra pair during physical plant installation is basically $0, which is why you'd never go "well we need a backup for the backup, so there's no point in having two pairs). Similarly, the marginal cost for having a second UPS and PDU in a rack is effectively $0 at scale, so nobody would argue this is unnecessary to deal with possible UPS failure or accidentally unplugging a cable.
In this case, there are likely several things that can be changes systemically to mitigate or prevent similar failures in the future, and I have every faith that Facebook's SRE team is capable of identifying and implementing those changes. There is no such thing as "no way around it", unless you're dealing with a law of physics.
By "no way around it" I mean you're going to need to create a circular dependency at some point, whether it's a maintenance network that's used to manage itself, or the prod network for managing the maintenance network.
I absolutely agree that installing a maintenance network is a good idea. One of the big challenges, though, is making sure that all your tooling can and will run exclusively on the maintenance network if needed.
(Also, while the marginal cost of laying an extra pair of fiber during physical installation may be low, making sure that you have fully independent failure domains is much higher, whether that's leased fiber, power, etc.)
How do you go from "have a separate access method that doesn't depend on your main system" to "turtles all the way down"? The secondary access is allowed to have dependencies, just not on your network.
I worked with a guy who was an amateur pilot, and he had an opinion about dual engine planes. He said the purpose of the second engine was to get you to the scene of the crash.
DNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?
But DNS didn't actually fail. Their design says DNS must go offline if the rest of the network is offline. That's exactly what DNS did.
Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given.
> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.
I'm not sure the design was even wrong, since the DNS servers being down didn't meaningfully contribute to the outage. The entire Facebook backbone was gone, so even if the DNS servers continued giving out cached responses clients wouldn't be able to connect anyway.
DNS being down instead of returning an unreachable destination did increase load for other DNS resolvers though since empty results cannot be cached and clients continued to retry. This made the outage affect others.
Note that FB.com currently expires its records in 300 seconds, which is 5 minutes.
PowerDNS (used by ordns.he.net) caches servfail for 60s by default — packetcache-servfail-ttl — which isn't very far from the 5min that you get when things aren't failing.
Personally, I do agree with DJB — I think it's a better user experience to get a DNS resolution error right away, than having to wait many minutes for the TCP timeout to occur when the host is down anyways.
An average webpage today is several megabytes in size.
How would a few bytes over a couple of UDP packets for DNS have any meaningful impact on anyone's network? If anything, things fail faster, so, there's less data to transmit.
For example, I often use ordns.he.net as an open recursive resolver. They use PowerDNS as their software. PowerDNS has the default of packetcache-servfail-ttl of 60s. OTOH, fb.com A response currently has a TTL of 300s — 5 minutes. So, basically, FB's DNS is cached for roughly the same time whether or not they're actually online.
The rest of the internet sucked yesterday, and my understanding was it was due to a thundering herd of recursive DNS requests. Slowing down clients seems like a good thing.
You cannot blame other operators if your own operator has broken software.
If your network cannot accommodate another network's DNS servers being unreachable, the problem is your network, not the fact that the other network is unreachable.
A network being unreachable is a normal thing. It has been widely advocated by DJB (http://cr.yp.to/djbdns/third-party.html) and others, since decades ago, that it's pointless and counterproductive for single-site operators to have redundant DNS, so, it's time to fix your software if decades later somehow it still makes the assumption that all DNS is redundant and always available.
I didn't notice any slowdowns on Monday, BTW. I don't quite understand why a well written DNS recursive cache software would even have any, when it's literally just a couple of domains and a few FQDNs that were at stake for this outage. How will such software handle a real outage of a whole backbone with thousands of disjoint nameservers, all with different names and IP addresses?
DNS was very much a proximate cause. In most cases you want your anycast dns servers to shoot themselves in the head if they detect their connection to origin to be interrupted. This would have been an big outage anyways just at a different layer.
Oddly enough, one could consider that behavior something that was put in place to "mitigate DNS misconfiguration"
> DNS seems to be a massive point of failure everywhere
Emphasis on the "seems". DNS gets blamed a lot because it's the very first step in the process of connecting. When everything is down, you will see DNS errors.
And since you can't get past the DNS step, you never see the other errors that you would get if you could try later steps. If you knew the web server's IP address to try to make a TCP connection to it, you'd get connection timed out errors. But you don't see those errors because you didn't get to the point where you got an IP address to connect to.
It's like if you go to a friend's house but their electricity is out. You ring the doorbell and nothing happens. Your first thought is that the doorbell is messed up. And you're not wrong: it is, but so is everything else. If you could ring it and get their attention to let you inside in their house, you'd see that their lights don't turn on, their TV doesn't turn on, their refrigerator isn't running, etc. But those things are hidden to you because you're stuck on the front porch.
Actually he was promoted to C-Level, CII, Chief Imperial Intern ‘For Life’.
It’s a great accomplishment to be be fair, comes with a lifetime weekly stipend and access to whatever Frontend books/courses you need to be a great web developer.
The blog post is putting the blame on a bug in the tooling which should have made the command impossible to issue, which is exactly where the blame ought to go.
tldr; a maintenance query was issued that inexplicably severed FB's data
centers from the internet, which unnecessarily caused their DNS servers to
mark themselves defunct, which made it all but impossible for their guys to
repair the problem from HQ, which compelled them to physically dispatch field
units whose progress was stymied by recent increased physical security
measures.
Imagine you have some DNS servers at a POP. They're connected to a peering router there which is connected to a bunch of ISPs. The POP is connected via a couple independent fiber links to the rest of your network. What happens if both of those links fail?
Ideally the rest of your service can detect that this POP is disconnected, and adjust DNS configuration to point users toward POPs which are not disconnected. But you still have that DNS server which can't see that config change (since it's disconnected from the rest of your network) but still reachable from a bunch of local ISPs. That DNS server will continue to direct traffic to the POP which can't handle it.
What if that DNS server were to mark itself unavailable? In that case, DNS traffic from ISPs near that POP would instead find another DNS server from a different POP, and get a response which pointed toward some working POP instead. How would the DNS server mark itself unavailable? One way is to see if it stopped being able to communicate with the source of truth.
Yesterday all of the DNS servers stopped being able to communicate with the source of truth, so marked themselves offline. This code assumes a network partition, so can't really rely on consensus to decide what to do.
Most of the large DNS services are anycasted via BGP. (All POPs announce the same IP prefix)
It makes sense to stop the BGP routing if the POP is unhealthy. Traffic will flow to the next healthy POP.
In this case if the DNS sevice in the POP is unhealthy and IP address belonging to the DNS service are removed from the POP.
Note those are anycast addresses, my guess is the DNS server gives out addresses for FB names pointing your traffic to the POP the DNS server is part of.
If the POP is not able to connect to the rest of Facebook's network, the POP stops announcing itself as available and that DNS and part of the network goes away so your traffic can go somewhere else.
> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job is to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible.
I found this to be an extremely deceptive conclusion. This makes it sound like the issue was that Facebook's physical security is just too gosh darn good. But the issue was not Facebook's data center physical security protocols. The issue was glossed over in the middle of the blogpost:
> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
The issue was faulty audit code. It is disingenuous to then attempt to spin this like the downtime was due to Facebook's amazing physec protocols.
Going on at lengths about how the trade off between prolonged downtime and strict security protocols is a worthy trade off is erecting a nonsensical strawman, the literal definition of spinning a story. The key issue had nothing to do with Facebook's data center security protocols.
And as they say, it lead to slower recovery from the event, which was caused, as they also clearly say, by something else. Given that "why is it taking so long to revert a config change?!!!" was a common comment, relevant to the discussion.
It's disingenuous to point to a paragraph in the article and complain that it doesn't mention the root cause when they already said before that, in the same article "This was the source of yesterday’s outage" about something else.
The original error was the network command, but the slower response and lengthy outage was partially due to the physical security they put in place to prevent malicious activity. Any event like this has multiple root causes.
Yes, but the fact that the blogpost concludes on this relatively tangential note (which notably also conveniently allows Facebook to brag about their security measures) and not on the note that their audit code was apparently itself not sufficiently audited, is what makes this deceptive spin.
I agree that there's an awkward emphasis on how FB prioritizes security and privacy but nothing is deceptive here. Had the audit bug not subsequently cut off access to internal tools and remote regions it would be easy to revert. Had there not been a global outage nobody would have known that the process for getting access in an emergency was too slow.
Huge events like this always have many factors that have to line up just right. To insist that the one and only true cause was a bug in the auditing system is reductive.
Our postmortems have three sections. Prevention, detection, and mitigation. They all matter.
Shit happens. People ship bugs. People fat-finger commands. An engineering team’s responsibility doesn’t stop there. It also needs to quickly activate responders who know what to do and have the tools & access to fix it. Sometimes the conditions that created the issue are within acceptable bounds; the real need for reform is in why it took so long to fix.
Seems like appropriate emphasis given how many people yesterday were asking why aren't they back online yet. For every person asking why they deleted their routes there were two people asking why they didn't put them back.
No, they just wanted to cover both "what caused it?" and "why did it take too long to fix it?" since both are topics people were obviously extremely interested in.
It would have been surprising and disappointing if they didn't cover both of them.
Simple Testing Can Prevent Most Critical Failures[1], "We found the majority of catastrophic failures could
easily have been prevented by performing simple testing
on error handling code – the last line of defense – even
without an understanding of the software design."
Having a separate testing instance of the internet might not be practical. How exactly would you test such a change? Simulating the effect of router commands is a very daunting challenge.
302 comments
[ 4.7 ms ] story [ 83.4 ms ] threadhttps://engineering.fb.com/2021/10/04/networking-traffic/out...
"We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it."
- Your engineering team.
[0] https://engineering.fb.com/2021/10/04/networking-traffic/out...
Should that be repeated in a somewhat more technical discussion of why it happened?
> flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ...
I wish there was a bit more detail in here. What's the worst case there? Brownouts, exploding transformers? Or less catastrophic?
Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons.
I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.
Do large providers like Facebook really provision less power than their servers would require at 100% utilization? Seems like they could just use fewer servers with power sized at 100% if their power system going to constrain utilization anyway?
I’m not an electrical engineer, so the details here may be fuzzy, however in broad strokes:
Grid operators constantly monitor power consumption across the grid. If more power is being drawn than generated, line frequency drops across the whole grid. This leads to brownouts and can cause widespread damage to grid equipment and end-user devices.
The main way to manage this is to bring more capacity online to bring the grid frequency back up. This is slow, since spinning up even “fast” generators like natural gas can take on the order of several minutes.
Notably, this kind of scenario is the whole reason the Tesla battery in South Australia exists. It can respond to spikes in demand (and consume surplus supply!) much faster than generator capacity can respond.
The other option is load shedding, where you just disconnect parts of your grid to reduce demand.
Any large consumers (like data center operators) likely work closely with their electricity suppliers to be good citizens and ramp up and down their consumption in a controlled manner to give the supply side (the power generators) time to adjust their supply as the demand changes.
Note that changes to power draw as machines handle different load will also result in changes to consumption in the cooling systems etc. making the total consumption profile substantially different coming from a cold start.
If a datacenter can't go from idle (but powered on) servers to fully utilized servers without taking down the power grid, then it seems that they'd have software controls in place to prevent this, since there are other failure modes that could cause this behavior other than a global Facebook outage.
> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk.
“Electrical systems” is vague and could refer to either internal systems, external systems or both.
That said, if the DC is capable of running under sustained load at peak (which we have to assume it is, since that’s its normal state when FB is operational) it seems to me like the externality of the grid is the more likely candidate.
In terms of software controls preventing this kind of failure mode, they do have it - load shedding. They’ll cut your supply until capacity is made available.
In the electricity grid, demand and generation must always be precisely matched (otherwise, things burn up). This is done by generators automatically ramping up or down whenever the load changes. But most generators cannot change their output instantly; depending on the type of generator, it can take several minutes or even hours to respond to a large change in the demand.
Now consider that, on modern servers, most of the power consumption is from the CPU, and also there's a significant difference on the amount of power consumed between 100% CPU and idle. Imagine for instance 1000 servers (a single rack can hold 40 servers or more), each consuming 2kW of power at full load, and suppose they need only half that at idle (it's probably even less than half). Suddenly switching from idle to full load would mean 1MW of extra power has to be generated; while the generators are catching up to that, the voltage drops, which means the current increases to compensate (unlike incandescent lamps, switching power supplies try to maintain the same output no matter the input voltage), and breakers (which usually are configured to trip on excess current) can trip (without breakers, the wiring would overheat and burn up or start a fire).
If the load changes slowly, on the other hand, there's enough time for the governor on the generators to adjust their power source (opening valves to admit more water or steam or fuel), and overcome the inertia of their large spinning mass, before the voltage drops too much.
The question is somewhat similar to a thought experiment. If a ship is docked and loading cargo, is it a good idea to use all the cranes to suddenly fill up one outer side of the ship?
A lot of horror stories start that way.
Close- you won't see an increase in load of a synchronous machine operating at constant throttle manifest as a voltage sag, you'll see it manifest as a decrease in frequency (this generators literally slow down, like a guy on a bike going uphill). Voltage sags are more related to transmission line phenomenon.
A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal to about 20% of the load behind the distribution transformer. That quantity is about as much as an aluminum smelter switching on or off.
Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?
I assume they do have backup generators, though I don’t know.
However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.
There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.
If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec.
You'd typically have some short-term power protection to keep your datacenter running until the generators start.
I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.
Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use...
https://www.oregonlive.com/silicon-forest/2015/10/small-town...
And Washington state too:
https://www.bizjournals.com/seattle/blog/techflash/2015/11/p...
Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness.
I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.
I used to do electrical compliance testing in a previous life, with brown out testing being one of our safety tests. You would drape a piece of cheese cloth over the power supply and slowly ramp the line voltage down. At the time, the power supplies didn't have good line side voltage monitoring. There was almost always smoke, and sometimes cheese cloth fires. Since this was safety testing, pass/fail was mostly based on if the cheese cloth caught fire, not if the power supply was damaged.
I can only speak to one problem I know of (and am rather sure I can share): a spike might trip a bunch of breakers at the data center.
BUT, unlike me at home, FBs policy is to never flip a circuit back on until you’re positive of the root cause of said trip.
By itself that could compound issues and delay ramp up time as they’d work to be sure no electrical components actually sorted/blew/etc. A potentially time sucking task given these buildings could be measured in whole units of football fields.
Imagine being this person.
Tomorrow on /r/tifu.
> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
But yes, for these types of problem, the ultimate fault is never “that guy Larry is an idiot”, it takes a large team of cooperating mistakes.
Turns out the underlying network software had a race condition that would ONLY be hit if the script ran at the exact same time as some automated monitoring tools polled the box.
At FAANG scale, "one in a million" happens a lot more often than you'd think.
And it happens less than you think too, sometimes, which I think is closer to the original point.
What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel like I'm missing something.
It's like the birds and the bees.
Those of us who are familiar with the domain of knowledge, on the other hand, get a decent summary of events.
It's a balancing act. I think the article does a good enough job of explaining things.
more on the concept: https://noidea.dog/blog/admitting-ignorance
Separate point to your question.
FB is under no obligation to provide more details than they need to because a small segment of the population (certainly relative to their 'customers') might find it interesting or helpful or entertaining. FB is a business. They can essentially do (and should be able to) do whatever they want. There is no requirement (and there should be no requirement) to provide the general public with more info than they want subject to any legal requirement. If the government wants more (and are entitled to more info) they can ask for it and FB can decide if they are required to comply.
FB is a business. Their customers are not the tech community looking to get educated and avoid issues themselves at their companies or (as mentioned) be entertained. And their customers (advertisers or users) can decide if they want to continue to patronize FB.
I always love on HN seeing 'hey where is the post mortem' as if it's some type of defacto requirement to air dirty laundry to others.
If I go to the store and there is not paper towels there I don't need to know why there are no towels and what the company will do going forward to prevent any errors that caused the lack of that product. I can decide to buy another brand or simply take steps to not have it be an issue.
You don't _need_ to know, but it's human to want to know, and it's also human to want to satisfy other human's curiosity, especially if it doesn't bring any harm to you.
Also, your post is not really answering any of GP's questions. I presume you wanted to say that FB doesn't _owe_ any explanation to us, but the GP asked, as they already provided one, to whom is it addressed.
That the main form of personal communication for hundreds of millions of users is down and there is no mandatory reporting is irresponsible. That Facebook is a business does not mean that they do not have responsibilities towards society.
Facebook is not your local supermarket, it has global impact.
At this point, I am completely sick of the pro-corporate rhetoric to let businesses do whatever they want. They exist to serve the public and they should be treated as such.
From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun.
At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.
> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
Would fit muscle memory, but if that wasn't caught by the automated tool they have some work to do.
https://status.cloud.google.com/incident/cloud-networking/19...
This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust).
I'd be curious to see what systemic changes happen at FB as a result, if any.
It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change.
"But wait, a global backbone isolation? Google wasn't totally down," you might say. That's because Google has two (primary) backbones (B2 and B4), and only B4 was isolated, so traffic spilled over onto B2 (which has much less capacity), causing heavy congestion.
> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication.
...
Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end."
Ultimately, that faulty command changed router configuration globally.
The Google outage was triggered by a configuration change due to an automation system gone rogue. But hey, it too was triggered by a human issuing a command at some point.
https://www.youtube.com/watch?v=Ijst4g5KFN0
This is a presentation to students by an MIT professor that goes over exactly what happened, the sequence of events, mistakes made, and so on.
This was an important lesson for SREs inside and outside Google because it shows the dangers of the antipattern of command line flags that narrow the scope of an operation instead of expanding it. I.e. if your command was supposed to be `drain -cell xx` to locally turn-down a small resource pool but `drain` without any arguments drains the whole universe, you have developed a tool which is too dangerous to exist.
If your tool is capable of draining the whole universe, period, it is too dangerous to exist.
That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)
Why did I think of humans, when I read this. :P
The pure takeaway is probably that it's important to design systems where "large" operations are rarely required, and frequent ops actions are all "small."
Because otherwise, you're asking for an impossible process (quick and protected).
Some tools are well engineered, capable of the Sisyphean task of globally deploying updates but others are rapid prototypes that, sure, are too dangerous to exist, but the whole point of SREs being capable programmers is that the work has problems that are most efficiently solved with one-off code that just isn't (because it can't be) rigorously tested before being used. You can bet there was some of that used in recovering from this incident. (I'm sure there were many eyes reviewing the code before being run, but that only goes so far when you're trying to do something that you never expected, like having to revive Facebook.)
There's solutions to that, but of course these sorts of tools all come into existence well before the system reaches a size where how they work becomes dangerous.
That's something that should never happen.
If history is any guide, Facebook will decide some division charged with preventing problems was an ineffective waste of money, shut it down, and fire a bunch of people.
Bullshit.
I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of personal and professional observations of people getting fucked over by google and having absolutely nobody they could contact to try and resolve the situation.
You googlers can't even disdain yourselves to talk to other workers at the company who are in a caste lower than you.
The fundamental problem googlers have is that they all think they're so smart/good at what they do, it just doesn't seem to occur that they could have possibly screwed something up, or something could go wrong or break, or someone might need help in a way your help page authors didn't anticipate...and people might need to get ahold of an actual human to say "shit's broke, yo." Or worse, none of you give a shit. The company certainly doesn't. When you've got near monopoly and have your fingers in every single aspect of the internet, you don't need to care about fucking your customers over.
I cannot legitimately name a single google product that I, or anyone I know, likes or wants to use. We just don't have a choice because of your market dominance.
GCP and Pixel phones are a toss-up between them and competitors.
It isn’t market dominance, nobody has made anything better.
(Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)
We must know different googlers then. It's good to avoid painting a group with the same brush
That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.
Is that normal at Google? Making people feel bad for an outage doesn't seem consistent with the "blameless postmortem" culture promoted in the SRE book[1].
[1] https://sre.google/sre-book/postmortem-culture/
By all means, put the support/enhancement requests through a separate channel+buffer so everyone can actually get work done during the day. But, at no point should an engineer ever be allowed to feel like they don't have to answer to some customer. If you are terrified a junior dev is going to say a naughty phrase to a VIP, then invent an internal customer for them to answer to, and diligently proxy the end customer's sentiment for the engineer's benefit.
I'm not talking esoterica, but at least a first approximation.
Very rarely are we our own customers
I'd say from a technical perspective SREs are, but there's a potential (depends on product) gap between their technical goals and user goals.
e.g. What does "p95 latency is spiking" actually mean to the end user?
If it's really the case that engineers are lacking information about the impact that outages have on customers (which seems rather unlikely), then leadership needs to find a way to provide them with that information without reading customer emails about how the engineers "let them down", which is blameful.
Furthermore, making engineers "emotionally invested" doesn't provide concrete guidance on how to make better decisions in the future. A blameless portmortem does, but you're less likely to get good postmortems if engineers fear shaming and punishment, which reading those customer emails is a minor form of.
Is what was described above "finger pointing or shaming"? I don't work in TI so I didn't experience this meeting but it doesn't seem like it is. It also doesn't sound to me like this was the only outcome, where the execs just wagged their fingers at engineers and called it a day. Of course there'd be all sorts of process improvements derived from an understanding of the various system causes that led to an outage.
Thankfully, other comments in this thread suggest that this is not how Google normally does things.
Postmortems are always blameless in the sense that "Somebody fat fingered it" is not an acceptable explanation for the causes of an incident - the possibility to fat finger it in the first place must be identified and eliminated.
Opinions are my own, as always
I have also never experienced this outside of this single instance. It was bizarre, but tried to reinforce the point that something needed to change -- it was the latest in a string of major customer-facing outages across various parts of TI, potentially pointing to cultural issues with how we build things.
(And that's not wrong, there are plenty of internal memes about the focus on building new systems and rewarding complexity, while not emphasizing maintainability.)
Usually mandatory trainings are things like "how to avoid being sued" or "how to avoid leaking confidential information". Not "you need to follow these rules or else all of Cloud burns down; look, we're already hemorrhaging customer goodwill."
As I said, there was significant scar tissue associated with this event, probably caused in large part by the initial reaction by leadership.
The same leadership that demanded tighter and tighter deadlines and discouraged thinking things through?
After reading this, I can't help but feel this was a calculated move.
It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'.
The timing is too perfect if you ask me.
https://twitter.com/cullend/status/1445156376934862848?t=P5u...
> the team dispatched to the Facebook site had issues getting in because of physical security but did not need to use a saw/ grinder.
https://twitter.com/MikeIsaac/status/1445196576956162050
https://twitter.com/MikeIsaac/status/1445196576956162050
https://twitter.com/cullend/status/1445212476652535815
This makes it sound like Facebook has physically laid "tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers". Is this in fact true?
https://datacenterfrontier.com/facebook-will-begin-selling-w...
Don't create circular dependencies.
- Okay, we'll set up a separate maintenance network in case we can't get to the regular network.
- Wait, but we need a maintenance network for the maintenance network...
"Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network)
True diversity is HARD (not unsolvable, just hard. especially at scale)!
Murphy is a jerk.
In this case, there are likely several things that can be changes systemically to mitigate or prevent similar failures in the future, and I have every faith that Facebook's SRE team is capable of identifying and implementing those changes. There is no such thing as "no way around it", unless you're dealing with a law of physics.
I absolutely agree that installing a maintenance network is a good idea. One of the big challenges, though, is making sure that all your tooling can and will run exclusively on the maintenance network if needed.
(Also, while the marginal cost of laying an extra pair of fiber during physical installation may be low, making sure that you have fully independent failure domains is much higher, whether that's leased fiber, power, etc.)
"Don't put two engines on the plane because both of them might fail" is not how fault tolerance works.
Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given.
> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.
DNS errors are actually still cached; it's something that has been debunked by DJB like a couple of decades ago, give or take:
http://cr.yp.to/djbdns/third-party.html
> RFC 2182 claims that DNS failures are not cached; that claim is false.
Here are some more recent details and the fuller explanation:
https://serverfault.com/a/824873
Note that FB.com currently expires its records in 300 seconds, which is 5 minutes.
PowerDNS (used by ordns.he.net) caches servfail for 60s by default — packetcache-servfail-ttl — which isn't very far from the 5min that you get when things aren't failing.
Personally, I do agree with DJB — I think it's a better user experience to get a DNS resolution error right away, than having to wait many minutes for the TCP timeout to occur when the host is down anyways.
I don't know BGP well, but it seems easier for peers to just drop FB's packets on the floor than deal with a DNS stampede.
How would a few bytes over a couple of UDP packets for DNS have any meaningful impact on anyone's network? If anything, things fail faster, so, there's less data to transmit.
For example, I often use ordns.he.net as an open recursive resolver. They use PowerDNS as their software. PowerDNS has the default of packetcache-servfail-ttl of 60s. OTOH, fb.com A response currently has a TTL of 300s — 5 minutes. So, basically, FB's DNS is cached for roughly the same time whether or not they're actually online.
If your network cannot accommodate another network's DNS servers being unreachable, the problem is your network, not the fact that the other network is unreachable.
A network being unreachable is a normal thing. It has been widely advocated by DJB (http://cr.yp.to/djbdns/third-party.html) and others, since decades ago, that it's pointless and counterproductive for single-site operators to have redundant DNS, so, it's time to fix your software if decades later somehow it still makes the assumption that all DNS is redundant and always available.
I didn't notice any slowdowns on Monday, BTW. I don't quite understand why a well written DNS recursive cache software would even have any, when it's literally just a couple of domains and a few FQDNs that were at stake for this outage. How will such software handle a real outage of a whole backbone with thousands of disjoint nameservers, all with different names and IP addresses?
Oddly enough, one could consider that behavior something that was put in place to "mitigate DNS misconfiguration"
Emphasis on the "seems". DNS gets blamed a lot because it's the very first step in the process of connecting. When everything is down, you will see DNS errors.
And since you can't get past the DNS step, you never see the other errors that you would get if you could try later steps. If you knew the web server's IP address to try to make a TCP connection to it, you'd get connection timed out errors. But you don't see those errors because you didn't get to the point where you got an IP address to connect to.
It's like if you go to a friend's house but their electricity is out. You ring the doorbell and nothing happens. Your first thought is that the doorbell is messed up. And you're not wrong: it is, but so is everything else. If you could ring it and get their attention to let you inside in their house, you'd see that their lights don't turn on, their TV doesn't turn on, their refrigerator isn't running, etc. But those things are hidden to you because you're stuck on the front porch.
It’s a great accomplishment to be be fair, comes with a lifetime weekly stipend and access to whatever Frontend books/courses you need to be a great web developer.
Will never touch ops again.
This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.
Imagine you have some DNS servers at a POP. They're connected to a peering router there which is connected to a bunch of ISPs. The POP is connected via a couple independent fiber links to the rest of your network. What happens if both of those links fail?
Ideally the rest of your service can detect that this POP is disconnected, and adjust DNS configuration to point users toward POPs which are not disconnected. But you still have that DNS server which can't see that config change (since it's disconnected from the rest of your network) but still reachable from a bunch of local ISPs. That DNS server will continue to direct traffic to the POP which can't handle it.
What if that DNS server were to mark itself unavailable? In that case, DNS traffic from ISPs near that POP would instead find another DNS server from a different POP, and get a response which pointed toward some working POP instead. How would the DNS server mark itself unavailable? One way is to see if it stopped being able to communicate with the source of truth.
Yesterday all of the DNS servers stopped being able to communicate with the source of truth, so marked themselves offline. This code assumes a network partition, so can't really rely on consensus to decide what to do.
In this case if the DNS sevice in the POP is unhealthy and IP address belonging to the DNS service are removed from the POP.
If the POP is not able to connect to the rest of Facebook's network, the POP stops announcing itself as available and that DNS and part of the network goes away so your traffic can go somewhere else.
I found this to be an extremely deceptive conclusion. This makes it sound like the issue was that Facebook's physical security is just too gosh darn good. But the issue was not Facebook's data center physical security protocols. The issue was glossed over in the middle of the blogpost:
> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
The issue was faulty audit code. It is disingenuous to then attempt to spin this like the downtime was due to Facebook's amazing physec protocols.
It's disingenuous to point to a paragraph in the article and complain that it doesn't mention the root cause when they already said before that, in the same article "This was the source of yesterday’s outage" about something else.
Huge events like this always have many factors that have to line up just right. To insist that the one and only true cause was a bug in the auditing system is reductive.
I guess deceptive was the wrong word, so whatever's the term for "awkward emphasis" :).
Shit happens. People ship bugs. People fat-finger commands. An engineering team’s responsibility doesn’t stop there. It also needs to quickly activate responders who know what to do and have the tools & access to fix it. Sometimes the conditions that created the issue are within acceptable bounds; the real need for reform is in why it took so long to fix.
It would have been surprising and disappointing if they didn't cover both of them.
1 https://www.eecg.utoronto.ca/~yuan/papers/failure_analysis_o...