Yes - under the root cause heading isn't the following raised as "the" root cause?
> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
Yes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
Right, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.
Right, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
That template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it doesnt' conflict epistemic idea that failure modes are caused by interacting events and processes. Graph vs tree etc.
Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been
due to human error.
tldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
I chuckled in agreement but then I started thinking if it could be actually a better term. Human error analysis might be a little blamey.. but availability design bug or something might be good.
> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
I wish this went into more detail about how an incomplete snapshot was created and how the incomplete snapshot was valid-enough to sort-of work.
I'm supposing that whatever interchange format was in use does not have any "END" delimiters (e.g. closing quotes/braces), nor any checksumming to ensure the entire message was delivered. I'm mildly surprised that there wasn't a failsafe to prevent automatically replacing a currently-in-use snapshot with one that lacks many of the services. (Adding a "type 'yes, I mean this'" user interaction widget is my preferred approach to avoid this class of problem in admin interfaces.)
>I'm supposing that whatever interchange format was in use does not have any "END" delimiters (e.g. closing quotes/braces), nor any checksumming to ensure the entire message was delivered.
Those only ensure you get the whole message, not that the message makes sense.
Quite true. I was assuming that the incomplete snapshot was a transmission error or storage error. It's quite possible that the bug was an error of omission inside the data itself (e.g. someone accidentally removed an important key-value mapping from the data generator).
One of the first things I did at Cloudflare was change the format of a file that contained vital information about the mapping between IPs and zones (no longer used these days) so that it had something like:
START <count> <sha1>
.
.
.
END
because it was just slurped up line by line assuming EOF was good.
Files can be immutable, cryptographically-verifiable, and distributed across servers without a blockchain.
Adding a blockchain to this would slow things down, add surface area for errors, and provide absolutely no value. This is true of every usage of blockchain other than creating tokens.
In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea.
In my experience, this has been called "ndjson", or newline-delimited JSON. It's a remarkably effective and useful pattern.
Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course.
Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser.
Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
> Is it really better than just using an array, though?
In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal.
Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.
I could see how a language might have a streaming json parser, but where a naïve "for each line in file" might read the file as a string - allocating all that memory anyway?
Used properly, "For each line in file" constructs in the major languages (Python, Go, Java, C#, Javascript, and so on) are well tested and really do read only one single line at a time. Just be sure to avoid the usual pitfalls, e.g. don't turn all the lines into a list before starting iteration.
I actually recently tried out a python based steaming JSON parser I found on SO and it was actually even more inefficient (for both CPU and memory) than the built-in JSON package (probably because Python and not a C-module). It was also quite a complex state machine, tricky to catch all corner cases. The SO module had some bugs that I fixed, but I ended up abandoning the entire approach because it wasn't saving any memory.
NDJSON certainly is a much simpler approach, which I like and appreciate.
End markers will help detect truncated files but not other kinds of brokenness, need checksums for those.
Also, I think the config files for certain off the shelf network devices don't come with end markers or checksums, and might not be very good at handling corrupted configs. The usual practice is to push updates to a small fraction of those, then wait and see what happens before proceeding further.
Unfortunately even checksums won't help you if you mistakenly dredge up old but valid config. That doesn't mean they're not worth doing, but we don't know, due to a lack of detail, if Google already has checksums or if they would even have helped in this situation.
Or it could have just been a recordio file that was being written by something that crashed in the middle of doing it, and the committed offset was at a valid end of record.
Really there's 1000 ways for this to happen all of which are going to sound obvious in retrospect but are easy to commit.
Even straight up Proto is vulnerable to this (either with an unlucky crash right between fields being actually written to a file, or when attempting to stream top-level fields).
It's more likely that the snapshot was incomplete in that it was based on incomplete precursors, rather than that the message itself was truncated. The details of something like that aren't always appropriate for this kind of report (i.e. they usually create more questions than they answer for a reader unfamiliar with the basics of the system).
No. That's not how it would be done at google. they invented a binary protocol to handle things like this reliably. And that protocol goes over an integrity-checking network transport. It's more likely an odd edge case occurred and somehow got past the normal QC checks- say, an RPC returned an error but the error handler didn't do a retry, it just fell through.
I ran into a problem like this with a service that used YAML for their config file. Basically when I edited it and saved the service would automatically pick up the change and load the config. However, the save hadn't completed so it only read a partial file which was still valid because, YAML.
I still like XML and XSD. People look at me these days like I'm insane. But in this case a partially loaded XML document would not parse let alone pass schema validation.
Again vindicated. YAML needs to go away. It is misery. I'd rather have XSLT than Helm templates.
Both you and I are very aware that the XSLT + XML config file would have a tool for translating/generating it from a YAML; and most users would use that tool instead of configuring in XML.
It's crazy how we still don't have the ability to atomically create/replace a file on the popular operating systems and filesystems , despite that being a super common use-case (without workarounds like renaming which have annoying limitations).
What would you want from an atomic replace of the contents of a file that rename() doesn't provide? Surely you've got to write the new contents somewhere before calling your atomic replace, and that's exactly what happens with rename/mv.
How do you atomically replace an existing file using rename? I believe rename fails if the target file already exists, instead of atomically replacing it. Plus you end up with orphaned temporary files if you crash at the wrong point.
There is another workaround using unnamed tempfiles on Linux, but I think even that approach doesn't work for replacing files.
And then there is the issue that this creates a new file with the same name, instead of replacing the contents of the existing file. I think that's often not the desired behaviour regarding symlinks or inode ids.
And finally all those workarounds are overly complicated, so almost no application uses them, while pretty much every application that saves some kind of document (image, textfile, etc.) would benefit from atomic saving.
mv uses rename(). No issue with the target file existing beforehand. No issue with symlinks going stale, because they link to the name not the inode. Hard links will preserve the old file and still refer to that though (which is sometimes what you want, sometimes not). But for most, symlinks are far more common.
I always think about how impossible it will be for GCP to compete with AWS. The work culture at AWS has been brutal for a decade. High standards for work, and insane amounts of oncall/ops reduction. A burn and churn machine that has created AWS. Google is a laid back company with great technology, but not the culture to grind out every detail of getting cloud to really work. Microsoft is another story altogether, as they already have a ton of corporate relationships to bring clients.
At googles scale and reliability target, I would hope they have multiple independent worldwide networks.
Each network would have its own config plane and data plane. Changes would only be made to one at once. Perhaps even different teams managing them so that one rogue or hacked employee can't take down the whole lot.
Then, if someone screws up and pushes a totally nuts config, it will only impact one network. User traffic would flow just fine over the other networks.
Obviously there would need to be thought on which network data will be routed over, failover logic between one and another, etc. And that failover logic would all be site specific, and rolled out site by site, so there is again so single point of global failure.
That isn't really how production networks work in my uneducated opinion. If they are connected to the production network then they are the production network, and the level of isolation required to make that not the case would be so extreme as to make things potentially more unreliable.
Others can correct me if I'm wrong about this. All I know is that the production network where I work is not air gapped in the way that would be required to truthfully consider testing networks a non production environment, so non prod changes typically wind up in front of the change review board anyway.
Ask your own sites network engineers and see if they have similar constraints because I would be interested to hear more perspectives on that.
One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems. That is mostly an abstraction created for applications people, those systems are not going to be totally blocked from interacting with eachother, they kind of have to. So if any of your "planes" are shared with production it is a production environment.
That would mean that all networks which peer with the Internet would necessarily be considered Production. This isn't that reasonable outside certain niches (i.e., national government networks).
Instead, what's commonly done is to provide a Controlled Interface (to borrow a term from those national government networks) that gates which things are at which level of trust. This is where security boundaries are enforced -- and if they are sound security boundaries things on either side can't reasonably damage the other side.
That's super interesting, and you're definitely right about the internet thing. I suppose our network guys must have some way to see if a change will propagate beyond a particular interface?
> One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems
If you use any sort of virtualization: the control plane (infra) vs data plane (apps) will naturally evolve from the architecture. The config plane and control plane can get squashed into the same thing though, but it can also be disparate for at both infra- and application level.
Data plane and control plane are definitely a thing in real physical systems- look at a classical router, where the packet processor works independently of, and is occasionally programmed by, or assisted by, a message passing from the data plane to the control plane. That control plane is typical elsewhere on the main board, talking to the data plane through a well-specific protocol.
Google's network is complicated, making many assumptions about "what is prod" etc hard to reason about.
I've seen a couple companies (credibly) claim multiple independent networks, but it seems to be pretty expensive to do in practice. There's a lot of cost to be saved by having a single network, and it's too tempting to share things making the independence illusory.
Probably you get better independence with smaller providers where the provider isn't yet big enough to actually run a global network. Then, each PoP is likely to run or fail independently. Google and peers have enough scope that they can run a single global network where one can experience the Haiku:
It's not BGP
There's no way it's BGP
It was BGP
(DNS also fits in there; any sort of automation to push DNS or BGP changes will probably fit, if the name was carefully chosen)
This "one global network" instead of, say, AWS's regions, was a selling point for Google against AWS. Big enterprises loved lovedLOVED not having to think in terms of regions.
Recently, AWS seemed to be finally capitulating to sales demand and globalizing things.
I wonder if that will slow now there's a competitor incident report with the answer being region sharding.
While correct that few services out of the total are cross-region I think only a few need to be to get the bulk of functionality.
If restoration(S3 & snapshots) and networking are both cross region it is possible to avoid outages without much effort.
S3 already supports global endpoint, cross regional replication and even manual failover.
RDS supports cross regional snapshots.
Networking has a bunch of options for crossing regional now.
If you're worried about AWS global endpoints and services going down the design is always around pointing to the global but being able to fall back to a specific region up.
At this point can just run EKS in N regions, have the clusters stateless and treat RDS+S3 for state.
Note that the region sharding is just about the config push, the network itself isn't getting shareded. The goal is to ensure a bad config push can't impact >1 region (in theory), not to segregate the actual network.
> Google's automation systems mitigated this failure by pushing a complete topology snapshot during the next programming cycle. The proper sites were restored and the network converged by 05:05 US/Pacific.
I think this is the most understated part of the whole report. The bad thing happened due to "automated clever thing" and then the system "automagically" mitigated it in ~7 minutes. Likely before a human had even figured out what had gone wrong.
How would you otherwise do it? Anything that automatically pushes updates should monitor for rapid increase in errors afterwards and roll back if so. You should do at least that if you are working on a critical system.
Sure, in an ideal world this is how nearly everything would work.
Getting a complex system to a level of maturity where this is feasible to do at scale in real life and actually work well is a respectable and non-trivial achievement.
I don't know if Amazon or Azure are able to confidently and effectively put in such automatic remediation measures globally. My sense is there are humans involved to triage and fix unusual types of outages at every other cloud provider, including the other bigs.
Leaving a comment on a message board saying how things ought to work is one thing (there's nothing wrong with your comment, I like it!); I only want to highlight, bold, and underscore how successfully achieving this level of automatic remediation atop a large and dynamic system is uncommon and noteworthy.
I wish literally everywhere had (mandated?) detailed public RFOs like this. My residential ISP down for 20 minutes? Tell me more about the cable headend or your bad firmware push, please!
"Programming" here seems to refer to configuring routing/topology stuff on the network. So the "programming cycle" is the loop of computing and deploying that configuration.
91 comments
[ 3.5 ms ] story [ 166 ms ] threadFor example why are there no processes to check for snapshot integrity or if they are, why were they not used?
They pointed out several issues that caused this and several mitigations to prevent it from happening again.
I have yet to see anyone else with public RCAs as good as Google Cloud
> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
What you’re saying can go into the lessons learned section.
Done.
I wish this went into more detail about how an incomplete snapshot was created and how the incomplete snapshot was valid-enough to sort-of work.
I'm supposing that whatever interchange format was in use does not have any "END" delimiters (e.g. closing quotes/braces), nor any checksumming to ensure the entire message was delivered. I'm mildly surprised that there wasn't a failsafe to prevent automatically replacing a currently-in-use snapshot with one that lacks many of the services. (Adding a "type 'yes, I mean this'" user interaction widget is my preferred approach to avoid this class of problem in admin interfaces.)
Those only ensure you get the whole message, not that the message makes sense.
Adding a blockchain to this would slow things down, add surface area for errors, and provide absolutely no value. This is true of every usage of blockchain other than creating tokens.
In all seriousness - just dropping 1gb json file with N million records in one end probably isn't great either. I suppose one could somehow marry js and subresource integrity protection[2] to get a json serialized structure with hash integrity check. It would probably be a terrible idea.
[1] https://jsonlines.org/
[2] https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
I wonder why not zero?
Apparently jsonlines is only a website, whereas ndjson is a website+spec. Given the conceptual simplicity, this claim seems a bit dubious. Having two of these identical things, each one clawing for mindshare is dumb and counter-productive. Why can't we all be friendly and get along for the greater good? Oh well, it's par for the course.
http://ndjson.org/
https://platform.openai.com/docs/guides/fine-tuning/prepare-...
And googled/ducked from there.
Is it really better than just using an array, though? Now you essentially first parse the data/newline files, splitting blindly on "\n" - hoping that a) you've got the data, and b) that each record can be parsed as json - by an entirely different parser.
Meanwhile - a valid json file (Array of records) - can be quite quickly parsed by an optimized json parser, will error on wrong or truncated input, will (have to) handle reasonable whitespace?
In many cases, absolutely yes it's better. Otherwise you can end up being forced to cram several GB or even 10's of GB of JSON into working memory. Not ideal.
Usually each ndjson file is generated by a single process, so there isn't much concern about invalid formatting making it's way in. It's one of those rare win-wins.
I actually recently tried out a python based steaming JSON parser I found on SO and it was actually even more inefficient (for both CPU and memory) than the built-in JSON package (probably because Python and not a C-module). It was also quite a complex state machine, tricky to catch all corner cases. The SO module had some bugs that I fixed, but I ended up abandoning the entire approach because it wasn't saving any memory.
NDJSON certainly is a much simpler approach, which I like and appreciate.
Also, I think the config files for certain off the shelf network devices don't come with end markers or checksums, and might not be very good at handling corrupted configs. The usual practice is to push updates to a small fraction of those, then wait and see what happens before proceeding further.
I suspect it was more a case of incorrect error handling in a loop... Eg.
Really there's 1000 ways for this to happen all of which are going to sound obvious in retrospect but are easy to commit.
Again vindicated. YAML needs to go away. It is misery. I'd rather have XSLT than Helm templates.
There is another workaround using unnamed tempfiles on Linux, but I think even that approach doesn't work for replacing files.
And then there is the issue that this creates a new file with the same name, instead of replacing the contents of the existing file. I think that's often not the desired behaviour regarding symlinks or inode ids.
And finally all those workarounds are overly complicated, so almost no application uses them, while pretty much every application that saves some kind of document (image, textfile, etc.) would benefit from atomic saving.
They were starting to bend to GCP sales pressure* to globalize.
Fingers crossed this slows that roll.
* actually enterprise I.T. purchasing demand, to be clear
Each network would have its own config plane and data plane. Changes would only be made to one at once. Perhaps even different teams managing them so that one rogue or hacked employee can't take down the whole lot.
Then, if someone screws up and pushes a totally nuts config, it will only impact one network. User traffic would flow just fine over the other networks.
Obviously there would need to be thought on which network data will be routed over, failover logic between one and another, etc. And that failover logic would all be site specific, and rolled out site by site, so there is again so single point of global failure.
Others can correct me if I'm wrong about this. All I know is that the production network where I work is not air gapped in the way that would be required to truthfully consider testing networks a non production environment, so non prod changes typically wind up in front of the change review board anyway.
Ask your own sites network engineers and see if they have similar constraints because I would be interested to hear more perspectives on that.
One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems. That is mostly an abstraction created for applications people, those systems are not going to be totally blocked from interacting with eachother, they kind of have to. So if any of your "planes" are shared with production it is a production environment.
Instead, what's commonly done is to provide a Controlled Interface (to borrow a term from those national government networks) that gates which things are at which level of trust. This is where security boundaries are enforced -- and if they are sound security boundaries things on either side can't reasonably damage the other side.
If you use any sort of virtualization: the control plane (infra) vs data plane (apps) will naturally evolve from the architecture. The config plane and control plane can get squashed into the same thing though, but it can also be disparate for at both infra- and application level.
Google's network is complicated, making many assumptions about "what is prod" etc hard to reason about.
Probably you get better independence with smaller providers where the provider isn't yet big enough to actually run a global network. Then, each PoP is likely to run or fail independently. Google and peers have enough scope that they can run a single global network where one can experience the Haiku:
(DNS also fits in there; any sort of automation to push DNS or BGP changes will probably fit, if the name was carefully chosen)Recently, AWS seemed to be finally capitulating to sales demand and globalizing things.
I wonder if that will slow now there's a competitor incident report with the answer being region sharding.
Short answer: no. cross-region services are few and far between.
Long answer: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...
If restoration(S3 & snapshots) and networking are both cross region it is possible to avoid outages without much effort.
S3 already supports global endpoint, cross regional replication and even manual failover.
RDS supports cross regional snapshots.
Networking has a bunch of options for crossing regional now.
If you're worried about AWS global endpoints and services going down the design is always around pointing to the global but being able to fall back to a specific region up.
At this point can just run EKS in N regions, have the clusters stateless and treat RDS+S3 for state.
I think this is the most understated part of the whole report. The bad thing happened due to "automated clever thing" and then the system "automagically" mitigated it in ~7 minutes. Likely before a human had even figured out what had gone wrong.
Getting a complex system to a level of maturity where this is feasible to do at scale in real life and actually work well is a respectable and non-trivial achievement.
I don't know if Amazon or Azure are able to confidently and effectively put in such automatic remediation measures globally. My sense is there are humans involved to triage and fix unusual types of outages at every other cloud provider, including the other bigs.
Leaving a comment on a message board saying how things ought to work is one thing (there's nothing wrong with your comment, I like it!); I only want to highlight, bold, and underscore how successfully achieving this level of automatic remediation atop a large and dynamic system is uncommon and noteworthy.