HN Explorer

629 comments

[ 3.5 ms ] story [ 324 ms ] thread
walshemj
just had rolld20 in the USA blow out a game I wonder if it is effected
zoidq
Looks to be working in the UK
walshemj
Network peering probs in Canada - looks like an unrelated problem
horyzen
Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.
nkassis
The GCE console also affected, couldn't send a support ticket just getting errors.
darkof
That feeling when you open https://console.cloud.google.com and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.
vpontis
Gosh, this was so scary... I thought someone had hacked in and deleted everything...

I hope they come back. This is still pretty scary

stnmtn
Same, my Manager called my and said "everything is down".

So I wander over to my Firebase console, and there's no database loading. Thank god for twitter, and people also saying that they have the same issue or I would have for sure though we've been hacked.

I hope this is a good wake up call for everyone. I know that I'm going to think more about how we do backups and fail-safes

hanniabu
And here I thought I was having a bad day with Google Play not loading
jjeaff
Same. I was thinking, oh, my db cluster must be having trouble recovering. Couldn't get any response through kubectl. Logged in to the cloud console and it looks all brand new, like I have no clusters setup at all.

Of course, this is 2 weeks after switching everything over from AWS.

dpau
my vm instances are all still there, can even log in via SSH in the compute engine tab. looks like they got a reboot 15 min ago. just restarted some processes but lost my progress on about 12hrs of computing time, i'm guessing it's going to be hard to get a refund..
londons_explore
You'll get a 25 percent refund of all costs for the month if you ask support
flamingcow
[I am the Cloud SQL tech lead]

This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.

grumblez
That's good to know. What confuses me is why they're saying "We continue to experience high levels of network congestion in the eastern USA", when I'm in us-west2 (Los Angeles) and none of my CloudSQL instances, nor is my k8s cluster showing up or contactable...
captn3m0
Mumbai region here, and GKE seems to be fine. (Accessing it from Bangalore)
electron0zero
Have services running in europe-west1, everything loads and working fine for me. looks like europe-west1 is not affected.
digaozao
I cant see any gke cluster in Brazil, or any VM.
miller_joe
I'm seeing that with northamerica-northeast1. I can't access anything over the network in that region and most of the GKE clusters and VMs in that region aren't listed in the console
snisarenko
The status page took a while to show issues. My app was down, and Twitter knew google cloud was down before the official status page.
[deleted]
(comment deleted)
vinayan3
Looks like only GCE is down according to the status page now. I'm able to access my console for instances and GKE clusters.
snisarenko
My site runs on Google App Engine and its down as well.
sladey
I happened to be initializing a GKE pool upgrade just as this occurred. The upgrade is now stuck according to the console.

The interesting thing is that a couple of minutes before everything went wrong, kubectl returned a "error: You must be logged in to the server (Unauthorized)" error

murat124
GCP has been down since 11:50am and they acked it 35 mins later. They're great at leaving their customers in the dark.
copperx
Not much different from AWS, from what I've heard.
numbsafari
Definitely the case. Neither are super great at this. One issue is that issues that may 100% impact individual clients may only impact a vanishingly small amount of their overall service load. That mismatch between customer and provider experience is one of the ugly aspects of public cloud providers.
kazen44
Also, it's one which gets hugely understated when people "move to the cloud".

especially if you use your bussiness for B2B services. Stuff like this could make you loose your bussiness, especially if some entity like google doesn't communicate and as a result, you do not have a answer for your own customers.

Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.

zepolen
On the flip side - a customer is more willing to be understanding if 'Google is down' instead of 'our server is down'.
discodave
That's why AWS is all about their Personal Health Dashboard (PHD). They can post specific issues for your account in there. Also, they get to keep the public page looking nice and green to show to executives of prospective customers.
msbarnett
Yeah, Amazon is the master of having their status page read all Green while half of US-East is in the toilet
xerxes901
Couldn't load the support console to "me too" this one either!
macintux
And thus was ruined hundreds or thousands of pleasant Sunday afternoons.

I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.

xerxes901
Spare a thought for the pleasant Australian early Monday mornings too! Always a rude awakening...
jagtesh
Multi-cloud for those times when you really need that level of availability and can afford it.
jjeaff
It's not even about being able to afford it. Some things just don't lend themselves to hot failover. If your data throughput is high, it may not be feasible or possible to stream a redundant copy to a data center outside the network.
EugeneOZ
All parts of the system should be copied (if you decided to build multi-cloud system), not just some of them.
newsbinator
It's the Queen's birthday, a Monday off here in New Zealand...

... but not for everybody now.

Scoundreller
So what happens when the crown changes? They change the holiday? Immediately? For the next year? Sounds like a bit of a nightmare.
ojm
It’s not actually the queens birthday.

In Australia, many states have different dates for the queens birthday.

So not a nightmare at all.

technion
Australia celebrates the Queen's Birthday public holiday on different dates in different states already.
doubleunplussed
Nah, it's not even her actual birthday. Different countries with the same queen even celebrate it on different days. Presumably it'll be renamed to "king's birthday" but the day kept the same when the monarch changes. Or done away with/re-purposed - there's a general feeling in Australia at least that once the queen dies there will be less support for the monarchy.
NeedMoreTea
The holiday is on the official birthday. The sovereign's actual birthday has been separate from the official birthday for centuries, so the holiday does not need to change.
[deleted]
(comment deleted)
jacques_chester
The only response is to wait for Google to fix it.

Nothing you or I or the pager can do will speed that up.

I am aware some bosses won't believe that and I am not trying to make light of it. But there really isn't much else to do except wait.

warp_factor
Or you wait for Google or you are frantically trying to move everything you got to AWS.
jacques_chester
If you wait, you get back to 100% with no effort or stress on your part.

If you try to be heroic, you get back to 100% with a bunch of wasted effort and stress on your part.

Because it will be fixed by Google, regardless of what you do or don't do.

After the incident is over would be the time to consider alternatives.

paulfurtado
So, for some companies, failing over between providers is actually viable and planned for in advance. But it is known in advance that it is time consuming and requires human effort.

The other case is really soft failures for multi-region companies. We degrade gracefully, but once that happens, the question becomes what other stuff can you bring back online. For example, this outage did not impact our infrastructure in GCP Frankfurt, however, it prevented internal traffic in GCP from reaching AWS in Virginia because we peer with GCP there. Also couldn't access the Google cloud API to fall back to VPN over public internet. In other cases, you might realize that your failover works, but timeouts are tuned poorly under the specific circumstances, or that disabling some feature brings the remainder of the product back online.

Additionally, you have people on standby to get everything back in order as soon as possible when the provider recover. Also, you may need to bring more of your support team online to deal with increased support calls during the outage.

mandatory
Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.
vpontis
Praying isn't working. Now, I'll try sobbing :(
Havoc
Systematic problem solving. I like it
dodobirdlord
Reports claim that it's a network congestion issue primarily in the northeast United States. It seems doubtful that any data has been lost, but you probably can't see resources due to requests not getting through. I hope that hearing this helps you to feel better.
flamingcow
[I am the Cloud SQL tech lead]

This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.

ksajadi
GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later
JimboOmega
Just like AWS, then. "Some users are experiencing increased error rates" = "Everything has been down for hours"
bsimpson
"Everything is fine, unless you're Carl. There's a massage outage, but only at Carl's house. Sorry, Carl."
notatoad
I'm also experiencing a massage outage. Please send masseuse.
bsimpson
Goddammit some (most?) days I can't type. "Massive"
djsumdog
I remember when S3 was down and the status was green because the updates for the status page with pushed via S3.
chmhsm
That's not just ironic, that's stupid. How do you count on S3 to update S3 status? Isn't that a huge design fault?
Analemma_
Azure too. During the most recent outage a couple weeks ago their Twitter account acknowledged the incident an hour before the status page did.

So no matter where you go for your cloud services, you're guaranteed a useless status page. Yippee.

duxup
I swear most status pages are run by folks who aren't "there".
hinkley
I haven’t written a status page in a while, but the rest of my infrastructure starts freaking out if it hasn’t heard from a service in a while. Why doesn’t their status page have at least a warning about things not looking good?
duxup
In my experience public status pages are "political" and no matter how they start tend to trend towards higher management control in some way... that leads to people who don't know, aren't in the thick of it, don't understand it, and / or are cautious to the point that it stops being useful.
jsty
Not only political, but with SLAs on the line they have significant financial and legal consequences as well. Most managers are probably happier keeping the ‘incident declaring power’ in as few a hands as possible to make sure those penalty clauses aren’t ever un-necessarily triggered.
hinkley
That’s fraud in other industries.
Scoundreller
Same with most corporate twitter feeds. I’d like to follow my public transit/airport/highway authority, but it’ll be 10 posts about Kelly’s great work in accounting for every service disruption.

And No, I don’t want to install a separate app to get push notifications about service disruptions for every service I use.

duxup
A good Twitter account is a wonderful thing....the bad ones hurt so bad.
hinkley
Ugh. I guess that just goes to show that any metric can be politicized.
wongarsu
It's just Goodhart's law in effect: If a status page is used as a target metric in an SLA the status page ceases to be a useful measure.
mr_toad
Status pages are the progress bars of the cloud.
duxup
I worked on the networking side for years.

Now the web development side and I'm all "Wait a minute...are there any progress bars that are based on, anything real!?!?!"

I should have known...

prepend
It’s an easy problem to fix as basically services should emit performance data, openly, and the status page should just summarize that. So if a service doesn’t report out, it’s assumed down or erroring out.

Having an excel file where people enter statuses is not very useful to me as a customer. That’s more like a blog.

[deleted]
(comment deleted)
emilfihlman
Digitalocean has the same issue: status pages are actually manually updated and no live data is fed into them.
intellix
Was noticing massive issues earlier and thought that maybe my account was blocked due to breaching from TOS as I was heavily playing with Cloud Run. Then I noticed gitlab was also acting up but my Chinese internet was still surprisingly responsive. Tried the status page which said everything was fine and searched Twitter for "google cloud" and also found nobody talking about it. Typically Twitter is the single source of truth for service outages as people start talking about it
arach
Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter
tr33house
Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations
bilal4hmed
gitlab is slow too
amenod
Slow is understatement... some pages on gitlab.com take minutes to load, and jobs take tens of minutes to start.

EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?

dsumenkovic
Yes, it was definitely related to GCP services.

GitLab is no longer seeing errors and Google Cloud has resolved the issue as of 23:00 UTC yesterday. Any further information can be found on the issue at https://gitlab.com/gitlab-com/gl-infra/production/issues/862

amenod
The timeline seems a bit short, but ok. :)

(the problems with runners and the UI started at least at 2019-06-02 7:48 UTC, though they were hit-and-miss at the time)

Still, happy this is solved and we can use the (fantastic otherwise) service again!

klon
Anyone using both AWS and GCP that can form an opinion on availability of both? As a GCP customer I am not very happy with theirs.
pm90
GCP is incredibly bad at communicating when there are problems with their systems. Just terrible. Its only when our apps start to break that we notice something is down, then look at the green dashboard which is even more infuriating.
timdorr
AWS is often the same way. No one seems to be good at communicating outage details.
TallGuyShort
I suspect there's a correlation between outages that are easy to detect and communicate and outages that automation can recover from so easily that you hardly notice.
[deleted]
(comment deleted)
obeattie
I really don’t get this. There’s a huge number of complaints about poor communication from companies like Google and AWS during every outage. Yet they remain seemingly indifferent to how much customer trust they are losing, and the competitive edge the first one to get this right could gain.
atmosx
If 20% of Google Cloud's customers leave after this outage because of poor communication they'll prioritise accordingly and apply all that nice SRE theories to their infra. But this isn't happening, because <various reasons>, so... who cares?
obeattie
I mean, I care. All else being equal I’m not sure why you wouldn’t want good communication to your customers.
manigandham
How much cloud spend do you control? That's the reality of how decisions are made.
obeattie
Many millions of dollars per year. I care about how my providers behave when they have issues, and I can't see why you think it's not at all relevant.
manigandham
> "why you think it's not at all relevant"

Nobody said this.

> "I care about how my providers behave when they have issues"

We all do.

As the other commenters stated, the communication is poor because the clouds are still growing rapidly and there's not much reason to be better. We might also be underestimating just how much more better service would cost and whether it's worth the revenue loss (if any). Are you really going to shift all of your spend overnight because of an outage? And where are you going to go?

The reality of these decisions is far more nuanced than it may seem and the current state of support is probably already optimized for revenue growth and customer retention.

[deleted]
(comment deleted)
sakisv
I don't think they are losing any kind of customer trust.

Unless something is really fucked (like both GCP and AWS being down for us-east) incidents like these are not going to impact them at all.

The cost of either migrating to the other provider or, even worse, migrating to more traditional hosting companies is enormous and will require much more than "service was down for 2 hours in 2019". The contracts also cover cases like this and even if they don't, Google and Amazon can and will throw in some free treat as an apology.

On one hand I find this quite sad, but from a pragmatic point of view it makes sense.

hinkley
What aren’t these on separate systems? I never had the impression that google cheaps out on things but this sounds exactly like the sort of shit that happens when people cheap out. Not even a canary system?
pm90
The idea that Google spends big on expensive systems is a huge lie.

Google started using a Beowulf cluster that the founders wired themselves. From the very beginning, the goal of metrics collection was to optimize costs. While today it’s seen as the cash cow, the focus has always been on cheap components strung together, relying on algorithms and code for stability and making the least possible demands of underlying hardware.

To think that they won’t try to save money any time they can seems implausible.

tubaguy50035
Obviously we don't know what the extent of the issue is yet, but afaik there has never been an AWS incident that has affected multiple regions where an application had been designed to use them (like using region specific S3 endpoints). GCP and Azure have had issues in multiple regions that would have affected applications designed for multi-region.
votepaunchy
> like using region specific S3 endpoints

AWS had the S3 incident affecting all of us-east-1: “Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.”

https://aws.amazon.com/message/41926/

WaxProlix
There was a massive push after that to have everything regionalized. It's not 100% but it's super close at this point.
dodobirdlord
S3 buckets are a global namespace, so control plane operations have to be single-homed. As an example, global consensus has to be reached before returning a success response for bucket creation to ensure that two buckets can't be created with the same name.
solidasparagus
But isn't CreateBucket the single s3 operation where you need global consistency?
dodobirdlord
As far as I know bucket policy operations also require global consistency.
cavisne
The availability of CreateBucket shouldnt effect the availability of customers apps. This tends to be true anyway because of the low default limit of buckets per account (if your service creates buckets as part of normal operation it will run out pretty quickly).

The difference with Google Cloud is a lot of the core functionality (networking, storage) is multi region and consistent. The only thing thats a bit like that in AWS is IAM, however IAM is eventually consistent.

tango24
That's one region, not the multiple region that OP mentioned
AaronFriel
Services in other regions depended transitively on us-east-1, so it was a multiple region outage.
sterwill
Which services in other regions? I remember that day well, but I had my eyes on us-east-1 so I don't remember what else (other than status reporting) was affected elsewhere.
theevilsharpie
I find GCP quicker to post status updates about issues than AWS, but GCP also seems to run into more problems that span across multiple regions.

I'm overall happy with it, but if I needed to run a service with a 99.95% uptime SLA or higher, I wouldn't rely solely on GCP.

kenhwang
AWS has what feel like monthly AZ brownouts (typically degradated performance or other control plane issues) with the yearly-ish regional brown/blackout.

GCP has quarterly-ish global blackouts, and generally on the data plane at that which makes them significantly more severe.

jjeaff
Are there any services that track uptime for various regions and zones from various providers? It's rare that everything goes down and thus the cloud providers pretend they have almost no downtime.
kenhwang
I can't imagine that being easy or cheap to make given the staggering number of product offerings across even the few big providers and how subtle some outages tend to be.
luu
CloudHarmony used to track this at some level for free, but it looks like you now need to sign-up or pay to get more than 1 month of history?

The last time I looked at it (back when it showed more info for free, IIRC), AWS had the best uptime of the three big cloud providers, with Azure in 2nd and GCP in 3rd.

IIRC, the memorable thing was that, shortly afterwards, the head of Google Cloud made a big announcement that CloudHarmony showed that GCP had the best uptime when CloudHarmony showed that it actually had the worst. Google was calculating this by computing downtime = downtime per region * number of regions, but at the time, Azure had ~30 regions and AWS had ~15 vs. ~5 for Google and if you looked at average region downtime or global outage downtime, Google came out as the worst, not the best.

dimes
I use both services heavily at work. The networking in GCP is terrible. We experience minor service degradation multiple times a month due to networking issues in GCP (elevated latency, errors talking to the DB, etc). We've even had cases where there was packet corruption at the bare metal layer, so we ended up storing a bunch of garbage data in our caches / databases. Also, the networking is less understandable on GCP compared to AWS. For instance, the external HTTP load balancer uses BGP and magic, so you aren't control of which zones your LB is deployed to. Some zones don't have any LBs deployed, so there is a constant cross-zone latency hit when using some zones. It took us months to discover this after consistent denials from Google Cloud support that something was wrong with a specific zone our service was running in.

AWS, on the other hand, has given us very few problems. When we do have an issue with an AWS service, we're able to quickly get an engineer on the phone who, thus far, has been able to explain exactly what our issue is and how to fix it.

captn3m0
Just curious, is this on a specific region(s)?
jpmattia
> We've even had cases where there was packet corruption at the bare metal layer,

I'd love to know how this happens in the modern world. I've seen it myself only once (not GCP, but our own network with cisco equipment.)

Is something in the chain not checking the packet's CRC?

user5994461
Had something similar last year because of a core router fabric issue. A few years ago, there was a batch of new servers with buggy motherboards corrupting/dropping packets, can't begin to imagine how hard it was to diagnose.

That's in own datacenters, not cloud.

jpmattia
> can't begin to imagine how hard it was to diagnose.

Yeah, when it happened to me, it completely threw me for a loop. We had reports of corruption in video files, which started the debug cycle. It was shocking when we isolated the box causing the issue.

But I guess your bigger comment has to be right: About the only way to have this sort of error is at the hardware level, because basic CRC checking should otherwise raise some sort of alarm.

user5994461
Keep in mind that hardware run with a firmware. What is called a hardware issue can actually be a software issue.

It wasn't just one box for us. Basically, the part number was defective (motherboard NIC), every single one that was manufactured. This affected a variety of things, since servers are bought in batch and shipped to multiple datacenters, damn impossible to root cause.

CRC can be computed by the OS (kernel driver) or offloaded to the NIC. I think it's unlikely for buggy CRC code to shipped to a finished product, it would be noticed that nothing works.

rexarex
AWS has better customer service and I don’t remember the last time there was a huge outage like this besides the S3 outage
dodobirdlord
There was a terribly day 2-3 months back in us-west-2 where CloudWatch went down for a couple of hours and took out AutoScaling with it, causing a bunch of services like DynamoDB and EC2 to improperly scale in tables and clusters, and then 12 hours later Lambda went down for a couple of hours, degrading or disabling a bunch of other AWS services.
solidasparagus
I've heard from people who have worked with both AWS and GCP that AWS has far better availability.
thruhiker
I've also heard similar from a teammate who previously worked with GCP. That said I know several folks who work for GCP and they are expending significant resources to improve the product and add features.
landon32
u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.
colinbartlett
Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down.

It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.

hazeii
...and only the paranoid survive?
MrMorden
Just because you're paranoid, it doesn't mean they're not out to get you.
FPGAhacker
I guess we know what steam uses (the store at least).
colinbartlett
I don't know about Steam, but I know Apple must use Google Cloud: https://www.apple.com/support/systemstatus/
djsumdog
Guess they don't eat their own dog food; no racks of proprietary Apple servers anywhere (unless they somehow run Darwin images in Google Cloud)
londons_explore
Can't tell if you know the answer to your own question and just can't talk about it due to NDA...
runlevel1
Apple runs Linux on the vast majority of the servers behind their cloud offerings.
gowld
Less than 1% of users are affected

Is there any reason to presume these statuses are correlated?

Apple's issue is

> Users may be experiencing slower than normal performance with this service.

kalleboo
Could be the only users who were affected were ones caught right in the failover between redundant clouds
hhs
"a lot of the world depends on Google to stay up."

Yup, I'm trying to check the Associated Press News right now and it's having trouble connecting to "storage.googleapis.com".

nabla9
This feels like 80's.

When the mainframe is down terminals are useless.

pmlnr
Yep. The cloud is just a lot of cheap hardware acting together as a shitty mainframe.
ktpsns
Server hardware is actually quite expensive. End users "smart" phones are cheap hardware, running dumb software which renders them as terminals for the cloud. That's sad because smartphone hardware is quite capable of doing useful work.

(For instance, I have a 500GB MicroSD card in my phone which contains a copy of my OwnCloud)

tus87
So that's why I can't login to YouTube this morning...
different_sort
I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back.

There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.

The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.

[deleted]
(comment deleted)
derekhh
> We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

I'm not seeing anything at 12:47.

derekhh
https://status.cloud.google.com/incident/cloud-networking/19...

404 - Impressive

ryacko
Cloud status dashboards seem to be hosted on the same cloud, which doesn't say much about redundancy.
chupasaurus
AWS changed internals of Service Health dashboard after they couldn't update it when S3 went down in us-east-1 (https://aws.amazon.com/message/41926/)

edit: wording

ryacko
Someone had to design the status page, and failed to anticipate the issue about how the status page depends on the systems it is reporting on.

Unlikely the rest of AWS, a cached web page does not require much complexity.