HN Explorer

197 comments

[ 5.3 ms ] story [ 215 ms ] thread
DiFronzo
Oh okay, thought something was on my end.
andydd
gcp load balancer responding with 404
terramex
Etsy is 404 too: https://www.etsy.com

Seems to be a bigger issue.

edit: Nest is down too: http://nest.com

Fitbit.com is 404 too: https://www.fitbit.com

Big GCP issue?

edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League.

GCP status page is still green all over the board: https://status.cloud.google.com

19:10 CET update: Some websites are coming back, including spotify.com, but their app still does not work for me.

information about outage just added to GCP status page, direct link: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-11-16 09:53 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-11-16 10:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

19:20 CET update:

Description: We believe the issue with Cloud Networking is partially resolved.

Customers will be unable to apply changes to their load balancers until the issue is fully resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2021-11-16 11:28 US/Pacific with current details.

Spotify desktop app still not working for me.

19:45 CET: Spotify app is back online for me.

panzagl
homedepot.com is 404, various pages on lowes.com are 404
ly
When Spotify went down I switched to Youtube Music. Funny how that's still running but Spotify isn't, if it is in fact a Google issue.
progbits
Internal Google infra and GCP are different beasts. Sure some failure modes take both down but more likely than not they are independent.
rmbyrro
That's why many choose AWS, they eat their own dog food.
Jugurtha
I have things running on GCP that are OK. I think it's related to nameservers.
profmonocle
It is (was?) definitely a load balancer issue. We had multiple load balancers on completely different DNS providers fail.
cscheid
https://deno.land down too. I guess a lot of CI actions will be failing just like ours...
sholladay
deno.land is working for me, while lots of other sites are not. Did they failover to another provider?
profmonocle
It seems things are gradually coming back. All of our load balancers started working again.
cscheid
Working now, yes. Though deno.land's current IP still nslookup's to something inside googleusercontent.com so maybe google has fixed their Side. But it was definitely offline earlier today (cf. our github CI failures and downforeveryoneorjustme)
Nextgrid
I’m assuming all these websites point to Google web frontend servers and for some reason it’s no longer able to map the Host header to the proper backend to proxy to.
jeffbee
This is a sort of repetitive outage for Google. They've wiped out the GSLB configs before. A year ago there was also that big outage where they blanked out the whole Gmail delivery configs and started rejecting all mails (even for gmail.com). Config safety is not their strong suit.
notreallyserio
I was seeing regular levels of health checks throughout the entire outage, so they still had configs in place. That seems plausible.
profmonocle
For what it's worth, not only HTTP(S) load balancers went down. We have a couple TCP proxy load balancers that went down too.
karimf
Gojek and Tokopedia too.
[deleted]
(comment deleted)
XCSme
All work for me in the EU apart from Etsy.
mkl95
Not Google's best week.
deberon
GCP outage? Status page shows green but a bunch of sites seem down (Rocket League most importantly).
carlhjerpe
Most importantly this game I play for leisure is down!
NaughtyShiba
I mean, that’s a fair point… leisure activity, whatever it is, is required for rest and to avoid burndown
mosen
I think it was a joke.
ukd1
We're also affected.
ukd1
and back up and running!
dyeje
Discord seems to be down.
contrahax
Seeing the same - I have projects in us-east1 that went offline first, then us-west1 went offline a few minutes after. Everything green on their status page and nothing in the dashboard - everything returns a 404 so I'm assuming a really high level LB just took a dump.
contrahax
Seems to have just resolved itself in us-east1 so I'm hoping us-west1 follows a few minutes after.
plg
overleaf.com also
samasblack
My heart goes out to all the math students who were frantically texing their problem sets last minute
tthebst
just wanted to start writing my thesis
jdleesmiller
(Overleaf co-founder here.) We're back up, so get writing :)
Jansin3
Rocket league perhaps epic games even
timdaub
- discord doesn't allow me to connect either.
nicebill8
GCP - my Cloud Run containers are giving 500's
dfxt8
Discord is down too.
crackercrews
CBS.com is down, and some NYT pages as well.
[dead] rasengan
heywire
Google’s 404 has always been like that
rasengan
That's noteworthy. Do you have an archived url example? :O
dymk
It’s been like that for all the time that I can remember (at least a decade)

Not super noteworthy

rasengan
You're right. It's just bad.
corobo
For what it's worth I don't think you should have been flagged or even downvoted just for being wrong. Corrected yes, absolutely. Sinking to the bottom of low contrast lake was a bit much.

Flags being gigadownvotes on this site suuuuuucks now. How do people learn if nobody can correct them? Daft.

I used to have a vouch option but I guess I used it too much haha

[deleted]
(comment deleted)
hexfish
That's just an inside joke and has been a default error page for Google/GCP for as long as I can remember.
lenniez
Also experience that with all my GCP related infrastructure (Europe)
mixedbit
Looks like perhaps an issue with Google Load Balancer. We have a load balancer in front of Google Storage Buckets, and can access resources directly from the buckets, but getting 404 when going through the load balancer.
te_chris
Yep, it's the GLB. Went down for us at 5:46 GMT. Just responding 404 and logs reporting an internal error.
htrp
I can confirm this on my side too
neom
Non-engineer here - Is there an easy way to multi-provider redundancy around this? Can you have LBs on multiple clouds and use dns to move around or something? Or does your LB have to be at the provider the app is at? Sorry if this makes no sense. :o
sparrc
yes it's 100% technically possible, the main issue is it would be significantly more expensive.
cinericius
I wonder if a legal discovery will ever find internal status dashboards that reflect reality rather than fictitious SLA liability-aware status pages.
hmrr
Your cynicism is justified and clearly based on the same experience I have :)
paxys
You don't need legal discovery for that. Every "X as a service" contract you sign will explicitly say that SLAs aren't dependent on dashboards/ping tests but rather a mostly subjective measure of "availability".
melling
I noticed Discord went down for me.
profmonocle
I wondered why our alerts started going nuts. Seems like basically every global Google Cloud load balancer went down. Doesn't seem to affect single-region network load balancers.

Edit: All of ours are back up. Some other services still seem down though.

1cvmask
Is it regional? Surprised Spotify is not active-active on other platforms like AWS and Azure.