The issue started occurring intermittently at 08:26 US/Pacific. The issue is impacting Google’s Backbone network and may impact various services when accessing them from a different region or from the internet. Impacted services include Cloud Services (Workspace, Firebase, GCP) as well as other Google properties.
Connectivity within a zone should not be impacted.
Our engineering team has implemented a mitigation and is now monitoring the effectiveness of the change.
OVH last Wednesday, Oracle the very next day, Azure yesterday, GCP today... will a crime syndicate employing arsonists as well as hackers claim all these in a month as a demonstration of their capabilities?
These comments crop up regularly on outage posts. There's probably some psychological bias that explains it but I can't remember... I think the last time Google went down there were comments blaming it on Summer Interns :)
No, it was SolarWinds that blamed the intern for their security issues.
In all seriousness... if a single employee can accidentally single-handedly screw things up you've got a process problem. There should be checks and balances to make sure that what you're trying to do make sense. These might be in the form of a code-review, automated validation, a staging environments, etc. Yes, these might be things that don't exist at a scrappy startup and that is fine as they'll get there eventually but for any large company not having good processes is reckless.
These issues happen despite all those things. You can bet that Google has that and more in place. Just like in aerospace engineering, things go wrong when dozens of failures happen across the critical path, not any individual piece.
Yes, and issues will always occur. What I was trying to point out that it's never an individual's fault alone. Sure someone might have tripped the final domino but they might not have had anything to do with everything else that happened. That's why blameless postmortems are a thing. Blameless (to me) doesn't mean that you don't name any person but rather that there's no blame assigned to anyone. I want to know what Person A did to cause the incident but there shouldn't be any retribution against Person A.
Does Cloudflare use Google Cloud? I know they have a ton of their own hardware, but maybe they outsource some things?
I had my first ever Cloudflare outage earlier today. It's just a free site that has a single page rule that redirects to a different domain, but it started refusing connections for a little while today.
Why the most GCP outages are global? Even though it says issue started in US/Pacific, I'm not sure if it affects other regions and the status page of them does not make it clear.
Because Google engineers think that they are the smartest people in the world which means they don't actually engineer for minimization of a blast radius.
You should just see how Google builds the prefix lists that it allows customers to advertise to Google via PNIs. You would think they are building it off some registry lists because why else would Google insists that the routes are registered? Oh no, that's for small people. At Google they just bring up a session to you, let you advertise and take whatever you advertised to them during the setup as the allow list because everything is done via custom automated software!
50 comments
[ 5.8 ms ] story [ 98.0 ms ] threadConnectivity within a zone should not be impacted.
Our engineering team has implemented a mitigation and is now monitoring the effectiveness of the change.
It’s also embarrassing (and can cause stress for your customers) when your status page is down.
Perhaps alexdumitru was using 8.8.8.8?
Edit: things started working now.
In all seriousness... if a single employee can accidentally single-handedly screw things up you've got a process problem. There should be checks and balances to make sure that what you're trying to do make sense. These might be in the form of a code-review, automated validation, a staging environments, etc. Yes, these might be things that don't exist at a scrappy startup and that is fine as they'll get there eventually but for any large company not having good processes is reckless.
I had my first ever Cloudflare outage earlier today. It's just a free site that has a single page rule that redirects to a different domain, but it started refusing connections for a little while today.
You should just see how Google builds the prefix lists that it allows customers to advertise to Google via PNIs. You would think they are building it off some registry lists because why else would Google insists that the routes are registered? Oh no, that's for small people. At Google they just bring up a session to you, let you advertise and take whatever you advertised to them during the setup as the allow list because everything is done via custom automated software!
https://cloud.google.com/press-releases/2020/0124/abk-and-go...
I wish I could say that to my customers...