I wonder why they would choose not to automate updating the status page (because they 100% do automate having alarms for when things break) - maybe they think that, even with high error rates, it could be a false alarm or maybe a regional outage?
Because the more powerful parts of the business want it to look green all the time, regardless of the reality.
It would be easier to automate it. But they want to pretend that they have fewer outages than they actually do because if they were honest about how often things break, it would make them look bad in comparison with everyone else who is also lying on their status pages.
Although the initial public notice of an issue can take longer than I'd like, I don't think GCP hides or downplays their incidents the same way AWS does.
To double-down on this, I wish companies more broadly would stop lying about their uptime on their status pages. It just creates chaos and confusion.
It also forces otherwise-honest companies to also lie about their uptime, because they don't want to be the one status page that actually shows all the little blips that naturally occur with an actively maintained product.
>Really irritating that they never update the status pages automatically, or even in a timely manor
In my experience any real status page that matters is irrelevant as soon as the powers that be become aware of it and it becomes sort of non technical and politicized. To some extent that seems almost inevitable depending on the organization.
I worked at one place where "The guy who runs the twitter account." ran a status page. Not that he knew how to do anything (The one time I know he wanted to change the page he couldn't figure out how to do it.), but he was the only one allowed to approve / make updates. But it didn't matter as he wasn't on call, nobody was quite sure who that person was from day to day. He or she was in some marketing department and their idea of a sense of urgency was "sometime this week ... maybe".
What you need is an external status page run by someone else who is monitoring. Not sure who pays for that though, maybe they can make money through ads. What you want to know is "is it down or just me?"
Mostly what happened at that large company is customers would call their sales guy, their sales guy would call me or someone who knows and we'd tell them ;)
A few customers could call me directly too, I didn't mind that, they were good folks.
It's not a good process, but it is what naturally happens.
There is a broad Google Cloud outage since appx. 9am CST. Their status page is green but someone from Google acknowledged the issue on Twitter. The GCP slack is also abuzz with users reporting issues across many many services.
For ~40 minutes, it was also impossible to SSH into any Compute Engine instance that uses OS Login for authentication, but it seems to have been resolved in the last few minutes.
What is “Google Cloud Components”, sounds like something all the other products depend on. If that’s the case, a lot of other products should be marked as down in their status page, other users have confirmed a lot of products are down. This is really irritating.
As of 8:06:07 AM PDT, all of my Firebase projects in us-central1 are back up.
We saw a complete silencing of our highly used Cloud Functions between 7:15 AM PDT and 7:38 AM PDT, but had reports from our users as early as 7:10 AM PDT.
My favorite part about https://status.cloud.google.com/incident/zall/20005 is that 7:49 AM PDT's update acknowledges that customers can't create support requests. Not like they'd do anything. Every time we've gone to GCP support, we've been unhappy with excessive back-and-forth, non-responsiveness on the part of the responsible parties, and generally extremely long durations between "we are completely down" and "everything would be fine had you not hacked around our incompetence three days ago".
Maybe this is a "grass is greener" sort of thing but I can't imagine AWS is this unreliable. This is the third major GCP outage we've had this year, and last time (not even three weeks ago!) it was three hours long.
> Every time we've gone to GCP support, we've been unhappy with excessive back-and-forth
Exactly this! I love GCP (especially firebase functions), inspite of these service disruptions. But, I can't deal with GCP support. They are unbelievably incompetent and have no understanding of the problem or solution. I feel like paying for support only to waste my time - explaining my problem to them with no hope for a solution.
33 comments
[ 2.7 ms ] story [ 77.2 ms ] threadAll of our Firebase projects are missing from their console, deploys fail, auth fails, getting 404s for their libraries.
Seems intermittent at the moment. But spreading.
Really irritating that they never update the status pages automatically, or even in a timely manor. As if customers won't notice.
It would be easier to automate it. But they want to pretend that they have fewer outages than they actually do because if they were honest about how often things break, it would make them look bad in comparison with everyone else who is also lying on their status pages.
https://status.cloud.google.com/summary has 20 entries for March.
It also forces otherwise-honest companies to also lie about their uptime, because they don't want to be the one status page that actually shows all the little blips that naturally occur with an actively maintained product.
In my experience any real status page that matters is irrelevant as soon as the powers that be become aware of it and it becomes sort of non technical and politicized. To some extent that seems almost inevitable depending on the organization.
I worked at one place where "The guy who runs the twitter account." ran a status page. Not that he knew how to do anything (The one time I know he wanted to change the page he couldn't figure out how to do it.), but he was the only one allowed to approve / make updates. But it didn't matter as he wasn't on call, nobody was quite sure who that person was from day to day. He or she was in some marketing department and their idea of a sense of urgency was "sometime this week ... maybe".
A few customers could call me directly too, I didn't mind that, they were good folks.
It's not a good process, but it is what naturally happens.
The status page has been updated: https://status.cloud.google.com/
GKE is working as normal in west-3c
https://news.ycombinator.com/item?id=22707876
This confirms my suspicions and my concerns still persist when choosing GCP
We saw a complete silencing of our highly used Cloud Functions between 7:15 AM PDT and 7:38 AM PDT, but had reports from our users as early as 7:10 AM PDT.
My favorite part about https://status.cloud.google.com/incident/zall/20005 is that 7:49 AM PDT's update acknowledges that customers can't create support requests. Not like they'd do anything. Every time we've gone to GCP support, we've been unhappy with excessive back-and-forth, non-responsiveness on the part of the responsible parties, and generally extremely long durations between "we are completely down" and "everything would be fine had you not hacked around our incompetence three days ago".
Maybe this is a "grass is greener" sort of thing but I can't imagine AWS is this unreliable. This is the third major GCP outage we've had this year, and last time (not even three weeks ago!) it was three hours long.
Exactly this! I love GCP (especially firebase functions), inspite of these service disruptions. But, I can't deal with GCP support. They are unbelievably incompetent and have no understanding of the problem or solution. I feel like paying for support only to waste my time - explaining my problem to them with no hope for a solution.