Yeah that was what initially got me to check status. Checked twitter and saw others with the issue. Tried to do voice only call but Google Voice was down as well.
Not such good news if you were supposed to have a meeting in the morning with people in the UK. Due to timezones, it's had to be rescheduled to Tuesday morning.
So they're using history.push() rather than history.replace().
I love the history API when it's used approprately, but it is a double-edged sword that has to be used carefully. News sites injecting their homepage in history on an article sometimes make me wonder if it's worth it.
I think a better option would be an immutable history, but pages that are loaded for less than 10 seconds grayed out in the list and pages that are inserted given a light pink background.
I'm not super concerned about history injection other than when blindly clicking the back button. I've used history injection in a legitimate use case requested by users (think master/detail when linking directly to a detail view; back goes back to the master list). I'm not sure how to unobtrusively notify of malicious history injection (like news sites) without downgrading legitimate uses.
Hopefully the answer isn't another permissions request :)
Unfortunately GWT is deprecated so the chance of this getting fixed is slim. Instead bet on this status page getting deprecated and maybe rewritten by 2021.
"Please don't complain about website formatting, back-button breakage, and similar annoyances. They're too common to be interesting. Exception: when the author is present. Then friendly feedback might be helpful."
Of course. Green means the service is working for me. It is Red when the service is not working for me, and clearly neither for nobody else. Orange means it doesn't work for me, but you might have better luck.
I imagine its insanely difficult to write an automated status dashboard since a problem can cause the service to not work while not having everything entirely broken.
The apis tested by the automated system could all still be working. Probably the only way would be to have a constantly running vm testing everything via the web ui.
It's not that difficult, just a piece of work. Google has teams dedicated to just that kind of work and internally has the automated dashboards that update real-time. The confusion here is that the page linked is not it. This is some static HTML or such that is updated when a human has triaged the issue, determined and aggregated scope and got an OK from another human (or more, depending on the situation).
Trying to understand here, do you think companies make these as a minimum status directly into their monitoring, or as a communications dashboard that someone upates, in aggregate?
I am experiencing intermittent issues with GMail not loading new conversations in a timely fashion, and showing "something went wrong" notification messages.
There are definitely issues. GDocs functionality is intermittent. e.g. the ability to move a doc into a new folder or even see what folder it's in using the little folder icon at the top while editing has disappeared.
I've suggested to our team they copy/paste locally anything they're currently editing.
"Our business is not a coherent single entity and we don't have the best operational policies that try to remedy that so I won't make any promises about quality while still sounding like we're worth $1b"
"I apologize for having misspoke earlier. The correct quote is: '...while still sounding like we're maybe worth $1b.' We could be worth more than the disclosed amount; however, we aren't sure if or how often, if ever, someone may be counting that number. It's really up in the air on occasion at around this point." - Me, still CEO, now an award-winning CEO.
What does 'actual status' mean though? If I'm running a website from a single server, the power goes out and the server goes down then it's kinda obvious. I don't think it's so obvious when you have hundreds of data centers serving traffic to billions of people. If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service.
You need to draw a line in the sand somewhere, but whatever measure you choose is going to be somewhat arbitrary so I think it's reasonable to have a human make the final call (based on some known criteria).
> If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service.
Then put that on the dashboard! It certainly is an outage for the affected users (by your scenario).
- 999/1000 Data centers up
- 100,000 Users without service
- 6 paying customers without service (will get money back)
Just guessing here, but I imagine that updates for service-wide statuses which are used on the scale that Google operates require manual vetting.
It is frustrating that these pages are not updated in real-time, but I do understand wanting to be sure before publishing a message to your entire userbase that you are experiencing disruptions.
It's usually less to do with whether automation works and more to do with having an internal party to hold responsible, when it comes to these things.
Google doesn't need an internal party to hold responsible for the concerns of free mail users or their own contractors who are called "content creators." In part because Gmail users and YouTubers don't lead fortune 50 companies and play golf with Google Execs like Google Cloud customers do.
One of our GKE clusters has been stuck "upgrading" for 12 hours. Totally stalled, no way to cancel, and auto-scaling is non-functional. Support seems pretty stumped by the issue and they haven't updated us since about an hour after I reported, so I wonder if this is part of a bigger issue?
Doubt it, you're probably just unlucky and have a cursed instance that is misconfigured in a very subtle way. Anecdote: I had an AWS instance that was stuck 'terminating' for 10 days in 2017 and finally got killed - maybe the hypervisor finally timed out or a SRE saw the anomaly.
Why is the Google Status dashboard slower than HackerNews at reporting the downtime?
To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.
A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality.
Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now".
Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first.
Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.
They should have pro-active monitoring system and update status page automatically when it detected disruption. Especially when you account to it's extremely hard to get human supports from Google because they automated it.
Google has multiple internal alerting systems and people are aware of any disruption/delay/anomaly from the minute it happens.
The on-caller for the specific service decides on how to produce.
The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.
when soemthing this broadly painful happens at a company of Google's scale --typically you have something like a hundred people internally that all got paged.
Honestly, incident management is hard. It is a different skill set to straight up SRE management, and all the training in the world won't get you ready. Believe me it's true in AWS too, true in a lot of places.
It's complicated, and the human factor is massive in an unpredicted scenario.
I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it.
One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same
The status dashboard is something which will be updated by an operations employee manually (yes by hand), who is a couple of layers behind the actual SWE/SRE who gets the page.
Search is still up and snappy! I may be biased as an ex-search Xoogler, but Search's resilience is a testament to how battle-hardened a stack it is, considering GDocs and Gmail appear to be hard down.
Offtopic, but it blows my mind seeing <1 min old StackOverflow comments indexed by Google search, the sheer amount of work going on behind the scenes to make that happen.
I'm pretty sure this only happens when the site is actively pushing updates to google via their webmaster tools. I think your sites importance would also impact how fast google indexes it.
Big talk! Must be nice being able to serve 99% of the index and not have anyone notice. If the gmail threadlist comes up with 1 missing message everyone will lose their minds. And obviously if user authentication and homing is down then search can serve public docs but gmail etc just can't serve.
From what I understand, "mostly". Depends on what you mean by "platform". A couple key differences--search runs at a higher level of redundancy than most other services. And, importantly, it can run very well at reduced capacity.
For example, if you type a search in the homepage or address bar, you start getting suggestions instantaneously. Think about this... if Google decided to only give you suggestions every second keypress, or every third keypress, would you really notice?
Moreover, writes are hard, reads are easy, and everywhere that a write needs to happen when a user does a search can fail gracefully. If this search doesn’t end up in your search history, or doesn’t show an ad, or doesn’t contribute to your future personalization then oh well, you’ll never notice.
As a present search Googler, it's more of a testament to how easy it is to run a service that doesn't need to accept transactional writes from users, contains little or no private data, and where >1% taint is acceptable.
Search ads require transactional writes. But actually, as far as I remember big outages at Google in the past have been things like "someone pushed the wrong router config" and not "our database got overloaded". Maybe search is just more conservative?
There are two mechanisms helping keep search up when considering ads:
-> Search engine pages can be served w/ no ads, or even no personalization if components are down or lagging
-> Ad logs can be lost if necessary, since we'd under-bill in that case, which would be our own problem. That happened a small handful of times while I was there.
The billing system is indeed transactional, but has less need for 100% up-time and 0 latency in comparison to search. Not nearly as big of a deal if there's an hour delay on billing ads as compared to search being down for an hour.
Side note: Hacker News is an IT guys best friend, came here for confirmation after staff started complaining about gsuite being down and the disruption announcement is top spot. Much appreciated
247 comments
[ 4.7 ms ] story [ 270 ms ] threadThe good news is that it happened on a Friday -- see you all next week!
Edit: Gmail Down
The dashboard neglects to mention Google Keep experienced a disruption too.
https://status.cloud.google.com/ https://www.google.com/appsstatus#hl=en&v=status
(In Google's own browser, too.)
Pretty annoying.
(If you paste the 2nd URL directly, you will not have problems.)
If you keep going to the site over and over, then potentially it counts as new views (albeit not unique ones)
That said Google probably exempts their own sites from page ranks since they you know, own the search engine
I love the history API when it's used approprately, but it is a double-edged sword that has to be used carefully. News sites injecting their homepage in history on an article sometimes make me wonder if it's worth it.
I don't bother clicking on TechCrunch articles for this reason.
If anything they'd probably want you not visiting it more.
> News sites injecting their homepage in history on an article sometimes make me wonder if it's worth it.
They even bring up a news site as an example.
Hopefully the answer isn't another permissions request :)
We really need a "update uri without side-effects, at least in #scope.
https://i.imgur.com/CTxvExS.png
https://news.ycombinator.com/newsguidelines.html
The apis tested by the automated system could all still be working. Probably the only way would be to have a constantly running vm testing everything via the web ui.
https://www.google.com/appsstatus#hl=en&v=status
https://status.cloud.google.com/
I've suggested to our team they copy/paste locally anything they're currently editing.
"Sometimes, it's probably almost approximately around what we think could be along the spectrum of known values." - Me, as CEO.
You need to draw a line in the sand somewhere, but whatever measure you choose is going to be somewhat arbitrary so I think it's reasonable to have a human make the final call (based on some known criteria).
Then put that on the dashboard! It certainly is an outage for the affected users (by your scenario).
It is frustrating that these pages are not updated in real-time, but I do understand wanting to be sure before publishing a message to your entire userbase that you are experiencing disruptions.
Gee, you mean we just discovered that area where Google admits that either automation or machine learning does not actually work?!
Google doesn't need an internal party to hold responsible for the concerns of free mail users or their own contractors who are called "content creators." In part because Gmail users and YouTubers don't lead fortune 50 companies and play golf with Google Execs like Google Cloud customers do.
https://mobile.twitter.com/uhoelzle/status/13093135569956618...
So, "kind of" confirmed, I think. Assuming a bad update is the most likely culprit for a whole pool of servers to all crash at the same time.
To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.
A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality.
Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now".
Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.
I think I'd prefer a status page that reflected that, or even no status page at all with an explanation, rather than a lying one, though.
The on-caller for the specific service decides on how to produce.
The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.
It's complicated, and the human factor is massive in an unpredicted scenario.
I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it.
One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same
All my mail sync is down, Analytics isn't loading, Sheets and Docs are all stuck in "Reconnecting". I wonder what could cause this.
"a pool of servers that route traffic to application backends crashed"
Although I'm not sure if this is what stack exchange actually uses or if there's some private API for big partners.
For example, if you type a search in the homepage or address bar, you start getting suggestions instantaneously. Think about this... if Google decided to only give you suggestions every second keypress, or every third keypress, would you really notice?
There are two mechanisms helping keep search up when considering ads:
-> Search engine pages can be served w/ no ads, or even no personalization if components are down or lagging
-> Ad logs can be lost if necessary, since we'd under-bill in that case, which would be our own problem. That happened a small handful of times while I was there.
The billing system is indeed transactional, but has less need for 100% up-time and 0 latency in comparison to search. Not nearly as big of a deal if there's an hour delay on billing ads as compared to search being down for an hour.
It annoys me a lot that Google doesn't use the "real" URL in their search result links.