HN Explorer

Google outage – resolved

2316 points by abluecloud ↗ HN
various services are broken

- youtube returning error

- gmail returning 502

- docs returning 500

- drive not working

status page now reflecting outage: https://www.google.com/appsstatus

--------------

services look to be restored.

869 comments

[ 1.3 ms ] story [ 193 ms ] thread
pachico
Yup, suffering it right now.
manquer
Yeah hangouts/meet as well
niklasmtj
Same here in Germany. Gmail says that my account can not be found. Plus Google's support page is also down
nashtheman
Yeah, drive and youtube seem to be down as well
_1tan
Drive too. Status board still green though.
0x54MUR41
Yes, I can confirm this. I got an error when accessing Google Docs, Google Meet, and Gmail.
clusmore
Australia, same problem here.
domoritz
Yep, drive and gmail give me 500 errors.
noutella
Alors, Search queries taking up to 3 seconds here in France.
degosuke
Anyone else has this in the technical details? Numeric Code: 7444
dkjaudyeqooe
Pretty much anything that requires login is broken.
maxekman
Meet is also down. From Sweden.
Avalaxy
The status page has proven itself useless again. According to that page, everything is working perfectly fine.
kiechu
What if it is a static website of corporate propaganda?
daliusd
It works fine. You can't login only.
proto-n
Hungary - gmail loads, can't send email (error #2014)
lwhi
Google home down too!
ivan1783
Can confirm from Germany, youtube is broke.
tosh
Just checked https://www.google.com/appsstatus

all green, which does not reflect reality for me (e.g. Gmail is down)

edit: shows how incredibly difficult introspection is

faeyanpiraat
HN is my goto status page when these things happen, never failed to provide up-to-date reliable information.
jansan
Google should write an AI application that checks on HN if their systems are working and display the result live on their status page. That would be more reliable than the current system they have in place (which is obviously not working at all).
jaimex2
It would probably just end up banning Googles own account and leave them with no human escalation option.
gostsamo
won't help when the ai service itself is down.
polote
Seems like everyone going on HN now. The site is so slow
davnicwil
Interesting cascade effect of sites going down! I wonder where people will go next to check if HN is down?

Also wondering if this is perhaps the fastest upvoted HN post ever? 8 mins -> ~350 votes, 15 mins -> ~750 votes. I wonder if @dang could chime in with some stats?

Update: looks like it hit 1000 upvotes in ~25 mins!

Update: 1500 in ~40 mins

Update: 2000 in ~1 hour 20 mins (used the HN API for the timestamp)

TonyTrapp
The recent youtube-dl takedown had a similar number of votes but still slightly slower, I think. I think it was >500 in 30min and >1000 in 60min if I remember correctly.
SSLy
reddit, nanog, 4chan, twitter — pick your poison
devenblake
"poison" is a particularly apt description for all of the above
Macha
Some politics topics have shot up pretty quickly in the past but user and/or mod flags send them back to page 2 about as quickly.
throwawaylolx
Funny this post is already #7. Are we seeing many reports again?

Stats right now: 1985 points | 1 hour ago | 597 comments

Macha
Maybe, could have just seen a dropoff in upvotes once the issue was resolved also
[deleted]
(comment deleted)
tuxone
Accessing the site anonymously fixes it (logged sessions are not chaced)
marcbarbosa
HN uses Firebase, maybe that is the reason.
qwertox
Does this site also use Firebase, or do they only feed Firebase's Database so that other developers can build upon it?
ratww
Any source for that? Last information I have is that it's written in Arc Lisp and uses files rather than a database for storage.

There is a public API on Firebase, but AFAIR it's just a mirror rather than the main storage.

ksec
HN was having problem showing anything for the past 10 min or so. I was going to ask if HN was down, then I saw this.
sammorrowdrums
I, and probably many others here are like a status page for friends and family too... I had my wife thinking that the internet was broken, and tried tethering to phone and things and still didn't work, then showed me and I saw the status code errors and was like "it's actually parts of Google that are down".

I love (and am deeply scared by) the dependence of Google and the confusion of it with the entire internet.

throwawaylolx
It gets even worse: https://twitter.com/joemfbrown/status/1338452107419148290

>I’m sitting here in the dark in my toddler’s room because the light is controlled by @Google Home. Rethinking... a lot right now.

Some people are compiling more relevant events: https://twitter.com/internetofshit

mlindner
If your home automation system requires a network connection to work then something is very wrong.
m4rtink
Right? Home automation is perfectly fine and mught even expose a secure authenticated API to the internet so you can say check and adjust the temperature remotely, but it shoukd never ever go down for lical use if the Internet connectivity or the remote service backend goes down.
tibu
Home Assistant rulez with non-cloud sensors. I never by sensors, cameras, switches which require connection to an external host. But I'm sure that most people will do it even with such outages too.
fileeditview
I still like my physical switch on the wall to turn the light on and off very much.. for me it's hard to beat. I can even turn it on/off semi-remotely with e.g. a tennis ball.
i_am_new_here
If my Alexa fails, then I can also just turn the light off with the switch and when flipped back "on" again, then it will be on.

Fallback to 'classical mode' works for me.

ketamine__
There are other sites that can check for you
espadrine
It is all green if you do not need to be logged in.

If you are logged in, the page crashes with an error.

You can still browse all services from Incognito (which for some is not an option).

randompwd
This is not true. i am logged in to gmail and getting 500 err

edit:parent updated comment

StavrosK
That's what the GP is saying.
randompwd
the parent comment to mine at time of writing was just:

> It is all green if you do not need to be log in.

Giving the impression if you were logged in/didn't need to log in, that everything would be ok.

StavrosK
Oh, I see. Looks like their auth (state) servers are down.
espadrine
I apologize for my ambiguous phrasing, and I feel bad that you were downvoted (I reupvoted you, but that’s only one vote).

I wrote too fast because I thought it could help people work around the problem.

skinnymuch
Are you saying the Google status page is correct then? Being logged in is part of being green I’d think.

Also, you can’t use many parts of Gmail, Drive, Photos, etc, without being logged in.

wffurr
I think it just means the status page itself is not logged in. Apparently it should be...
fireattack
Try YouTube, which is totally broken.

But I guess it's technically not part of /appsstatus

shawabawa3
youtube is working in incognito mode
eddytrex_
Yeah for me too: youtube and gmail works in incognito mode
oneeyedpigeon
> gmail works in incognito mode

I mean, that must be a generous definition of "works"! :)

Intermernet
YT seems to be working in Incognito / Private mode.
fireattack
Ah, thanks!

Kinda weird it would totally break if the auth failed, unlike other services like Search.

thunderrabbit
Youtube is down for me if I am logged in. I can watch videos through duckduckgo search or an icognito window.
Kye
Google made everyone get an account trying to force Google+ on to everyone. They don't get to pretend like that didn't happen by not factoring it into their status.
fny
https://downdetector.com/ is remarkably good at catching this too.

But funnily enough, a lot of the votes come from traffic that searches for "is ____ down?" on Google. XD

Intermernet
Amusingly, downdetector.com is now down. Gotta love that cascade effect :-)
wjdp
Found this relatively recently, seems to do a good job.

Do love how more consumer services (ISPs &c) always have some report of being down somewhere but its means nothing unless there's a big spike.

skinnymuch
Pretty funny. I would normally assume they are connected to some testing unit that real users interface with. Vs internal APIs or however Google is doing this with them still being green
abalos
Out of curiosity, what response time do you expect on a page like that? And what level of detail? I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.
edf13
It's a big group of teams... one team is responsible for monitoring (And status reporting) alone
havetocharge
There should be enough people on that to communicate out as well as resolve.
vineyardmike
If they know its broken, one of the _many_ engineers/support across Youtube+Gmail+etc that are all known to be down related to this bug should be able to update it in first few minutes. Especially if this isn't a 5 min fix.
wbhart
I'd expect within seconds that Google is alerted of a very large number of issues with their servers and that the status page would be updated (the green light going to red) within seconds. It's now quite some time after the start of the outage and everything is still green on that status page.
RichWalton
I'd expect that status page to be automated based on a number of metrics / health checks. Our equivalent is.
JimDabell
> what response time do you expect on a page like that?

Faster than a free third-party website’s response time. Google should know they are down and tell people about it before Hacker News, Twitter, etc. Google should be the source of truth for Google service status, not social media.

> And what level of detail?

Enough to not tell people that there are “No issues” with services.

> I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.

Google employs enough people to do both.

throwawaylolx
How many employees do you think Google has? Do you think they're all working on the same task?
pjc50
Well, probably a lot of them are sitting around doing nothing because Google's down right now.
ric2b
> I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.

It's not like they would be working on the status page right now, that work should have been done a long time ago...

kalleboo
It should be faster than it takes TechCrunch to write an article about it
pupdogg
What's the point of having a status page when it can't reliably tell you the factual status?
woofie11
Helps Google meet a 99.999% SLA!
koheripbal
Not really since now it's up, and the status page still says it's down.
keithnoizu
right after I shamed them on twitter. I take responsibility despite it unlikely having any impact.
jaimex2
Customer confidence. Same as how everyone reports they have 99.99% uptime.
petre
Then their status page is not unlike like target reporting in the USSR: completely fictional.
phaer
From personal experience and other Ops peoples stories, I'd assume that could be the case for many status pages ;)
szatkus
It's red already.
s_dev
No -- it needs to be red when it's having the outage for people to have confidence in it as an indicator. The reality is Google have no real incentive to provide an actual external status page that is accurate -- to do so is an admission of not upholding SLAs. These status pages are updated retrospectively. Use a third party one like DownDetector.
numpad0
Someone here in hn said at Amazon AWS they are required direct CTO approvals to reflect negative realities. Maybe that’s the case for anyone these days.
ThrawsAway
Not true. Only for written things like postmortems. Automated systems need to be vetted but can update things like status pages on their own.

The issue isn't "negative" realities, it is saying something while investigating that might break contracts only to find later that it wasn't true.

prof-dr-ir
Mistakes happen... that page is probably still correct 99,9999% of the time!

/s

rightbyte
These uptime numbers cloud providers gives out can't be nowhere near actual uptime.
entropie
Everything went red just now.
TimMeade
Solid red now. Everything down
[deleted]
(comment deleted)
INTPenis
I don't think it's difficult, I think they just lie.

Monitoring is very simple, I even learned this from a document released by the Google devops team many years ago.

Always alert from the end user perspective. So in other words have an external server test login to Gmail. Simple as that.

They manually update that status page to not scare away stockholders.

crosspanda
Definitely i agree with you. Although the statuses are updated later, it may cause many problems if it is not instantaneous.
senko
It reflects the reality for them.

The small print says: The problem with Gmail should be resolved for the vast majority of affected users. We will continue to work towards restoring service for the remaining affected users...

At Google scale, "remaining affected users" probably number in tens of millions. Sucks to be one of them, tho.

But hey, it happens. As a SaaS maintainer, I can sympathise with any SREs involved in handling this, and know that no service can be up 100% of the time.

rochacon
Google Meet and YouTube also 500 in Brazil