HN Explorer

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

90 points by 7tech ↗ HN
TLDR; never use google cloud systems for production.

Google cloud suspended all our projects due to the billing issue in their system they had.

Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.

All the account payments were made and the billing cards are valid. There are no outstanding bills.

Never use GCP for production.

---- Edit full story by request, long read: ----

Previous month billing didn't went through. Not sure if this was due to the billing outage google had (https://status.cloud.google.com/products/oLCqDYkE9NFWQVgctQTL/history) or financial transaction issue, however we went ahead and made a manual payment covering all the outstanding amount + extra. Despite the payment made, about a week+ later we suddenly started receiving threatening emails "Your Projects are at risk of suspension". Edited and updated the billing cards. Opened a billing support request clearly mentioning this is a production environment and all the bills are paid. They were "investigating" the issue and assured the project will not be suspended.

23 comments

[ 3.7 ms ] story [ 70.6 ms ] thread
senttoschool
Not a good look for Google.

I still don't think Google understands customer service yet. They think they can solve everything by being more technical than everyone else.

Google's engineering led culture is both a blessing and a curse.

ssmw
I’m sorry but without more context I won’t upvote this. We hear your side of the story but we won’t hear Google’s.

You say payments were made and billing cards were valid. Your account wouldn’t be suspended for that. I’m convinced there are important nuances you aren’t mentioning.

thematrixturtle
Yup. What exactly was the earlier problem? Why did you need assurances that projects would not be suspended in the first place?
[deleted]
(comment deleted)
hericium
Google are either completely delusional in thinking that at this point ML-based solutions are capable of replacing human decision making or they are fully aware that it's not working well and getting worse over time. It won't change though as savings made on humans are driving their YoY.

Pichai sacrificed (what was left of) Google's good press in favor of YoY. Google is proving time and again that they are unstable and unreliable as a business partner and can't be trusted with anything but being a growing stock market product.

[deleted]
(comment deleted)
CyberNoodle
Had a similar experience with GCP. I don’t remember the exactly how, but somehow we ended up with a $0.01 bill. This bill was apparently too small for Google to deduct, instead all our api services were disabled :/
Operyl
How weird, we get charged 0.01 all the time, and I laugh thinking that it cost them more to charge us a penny than it would've been to just roll it over.
blind_oracle
Huh, my Swiss telco operator does not even bill anything less than 3 bucks...
hericium
> I laugh thinking that it cost them more to charge us a penny than it would've been to just roll it over

In the long run it's not them covering those costs but you and other customers.

Operyl
Perhaps, but certainly not from me. For the past 2 years it has been a penny a month bill. My statement was mostly that their processing fees are pennies and they could have just waited a bit.
superchroma
Seems like the lesson is that there is no reason to use Google products if you ever expect you will need support in the future, because it is just not there. They evidently don't care enough to work with anyone and their revenue is high enough that customer retention has zero priority for them. This is unsurprising to me as they're an ad company first and foremost. Maybe do it if you know someone inside or are famous on Twitter?

Anyway, people pay a lot of money and just seem to be screwed time and time again.

wolczek
Had a similar experience with GCP about 5 years ago. They tried to charge $0.00 and then suspended the account. It took 11 hours to get the services back.
[deleted]
(comment deleted)
lbriner
Why are you running a business without any Business Continuity or Disaster Recovery? In this case, Google might have screwed up but what about if the account had been hacked, or Google had some big technical problem that killed off the production system for another reason?

We all know it is infuriating and frustrating but that is why you need BC/DR in the first-place.

piva00
That's great advice that doesn't really cover the core issue here: GCP with a business contract is trusted to not do this, yet they did it. Yes, you can prepare disaster recovery and business continuity procedures, they have a cost and it's not trivial. If you postpone this cost as a technical debt due to an assumption that Google should be trusted to have a business contract with and got burned by Google I do believe it's not the victim's fault.

Your comment sounds a lot more like victim blaming than just a sensible procedure to have in place as much as I agree with the need for BC/DR. BC/DR are not free, paying Google (as they've done) should've been enough for this case.

Schroedingersat
> Never use GCP for production.

Never use any google product if there is any alternative even a significantly worse one. This will keep happening and much worse until we say no to the abuse.

lfkdev
You should sue them. You have all the evidence on your side
23Ro
Disclaimer: not paid, neither working for google.

Ouch, this is a very bad situation. As a business owner of a SaaS offering myself this sounds like a nightmare (and we are on GCP).

Few months ago (in a different company) I had a very positive experience with GCP. We've unfortunately had a situation where a over-provisioned service account leaked. It has, of course, been used to mine crypto by some malicious actor. We've ramped up a very high six figure bill in just a few hours of this going unnoticed. After reaching out to google, and a months long review process, the bill we've ramped up has been dropped. During the waiting period we have NOT paid any bill for the entire time. Granted: we've been told to not pay by the customer service, until the situation has been sorted out.

What I'm trying to say is: there's good and bad, at every major cloud provider.

Dr_ReD
While I empathise with the OP, I would add:

- never-ever really trust any company assurances or contracts. Especially if they're way bigger than you are, always assume they WILL fail you and you'll have no recourse outside of litigation—and even then... :-/

Always have a backup plan for important stuff. Always. No exceptions.

More in general, avoid every single point of failure you can afford to.

splix
Same happened to us the last Thursday, but in our case they suspended our GCP account without a prior notification. Just learned about this from our external monitoring.

Turns out they wanted to check my IDs, and it's after I had been their customer for 14 years. And it took them 5 days to verify that ID. An hour ago they activated it back.

I've been running a few projects on GCP, with active customers, which I lost now. Those 5 days of outrage is basically a catastrophe for us.

And no apology was given by Google. There was barely any communication from their side.

Never use GCP for production.