I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
Because there's money in letting people make mistakes on their own dime.
For every story about a mistake in the tens of thousands, I wonder how many there are in the single-thousands, hundreds and even tens where people just suck it up and pay the bill. When you provide such a limited billing support surface, employ a thousand lawyers in-house, and hold as many cards as Google does (losing my G account would ruin my year).
Multinational service providers need better regulation to ensure consumers' rights are protected. In the UK utilities and banks have to absorb (or insure) against some leaks, theft and external fraud. If Google (and others!) were subjected to similar regulation, problems like this would evaporate overnight.
Sounds like a great mistake for Google to find a way to repeat. Why innovate when you can abuse users and hide behind "complexity" as "plausible deniability"?
We got a $12,000 bill a few hours ago on a presumably leaked gemini API which I very much doubt, and we are trying to resolve the issue with a real support agent. I think they messed something internally and customers are getting these bills.
I never got a surprise bill myself, but reading a few cases like this motivated me to cancel my GCS account and remove my CC. Now if I try to use it it fails immediately with an error.
As author of HashBackup, I know people are using it with GCS, and I'd like to be able to test against it, but not enough to swallow a large surprise Google bill.
Right now there are about 4 concurrent threads on the googlecloud subreddit about people getting hosed with life changing bills. Some no doubt through stupid mistakes (happens), but still bewildering that Google is insisting individuals like students are subject to the same scale to infinity bills as huge corporations and are unwilling to provide any mechanism to protect hobbyists.
And then people tell you but there are quotas and then:
>Google automatically upgraded it to the next level when the account crossed the $1,000 threshold during the incident.
Looks like another instance, I am dealing with a very similar issue. I didn't even notice any sort of tiers, as I had 17 grand in costs accumulate at a similar pace.
I (a hobbyist running a small side project for a dollar or two a month in normal usage, so my account is marked as "individual") got hit with a ~$17,000 bill from Google cloud because some combination of key got leaked or my homelab got compromised, and the attacker consumed tens of thousands in gemini usage in only a few hours. It wasn't even the same Google project as for my project, it was another that hasn't seen activity in a year+.
Google refuses to apply any adjustments, their billing specialist even mixed up my account with someone else, refuses to provide further information for why adjustments are being rejected, refuses any escalation, etc. I already filed a complaint with the FTC and NYS attorney General but the rep couldn't care any less.
My gripe is not that the key was potentially leaked or compromised or similar and then I have to pay as a very expensive "you messed up" mistake, it's that they let an api key rack up tens of thousands in maybe 4 hours or so with usage patterns (model selection, generating text vs image, volume of calls, likely different IP and user agent and whatnot). That's just predatory behavior on an account marked as individual/consumer (not a business).
This problem exists to some degree for all cloud services. Is there a solution based on the credit card used? Can you get a CC that has a hard limit guarantee?
EDIT: I guess that you'd still be responsible for the charges though.
16 comments
[ 2.5 ms ] story [ 48.8 ms ] threadFor every story about a mistake in the tens of thousands, I wonder how many there are in the single-thousands, hundreds and even tens where people just suck it up and pay the bill. When you provide such a limited billing support surface, employ a thousand lawyers in-house, and hold as many cards as Google does (losing my G account would ruin my year).
Multinational service providers need better regulation to ensure consumers' rights are protected. In the UK utilities and banks have to absorb (or insure) against some leaks, theft and external fraud. If Google (and others!) were subjected to similar regulation, problems like this would evaporate overnight.
You'll all keep using them either way.
Yes nuclear option, but I’ll take an hour down time over a $100k unexpected bill
As author of HashBackup, I know people are using it with GCS, and I'd like to be able to test against it, but not enough to swallow a large surprise Google bill.
Right now there are about 4 concurrent threads on the googlecloud subreddit about people getting hosed with life changing bills. Some no doubt through stupid mistakes (happens), but still bewildering that Google is insisting individuals like students are subject to the same scale to infinity bills as huge corporations and are unwilling to provide any mechanism to protect hobbyists.
And then people tell you but there are quotas and then:
>Google automatically upgraded it to the next level when the account crossed the $1,000 threshold during the incident.
I (a hobbyist running a small side project for a dollar or two a month in normal usage, so my account is marked as "individual") got hit with a ~$17,000 bill from Google cloud because some combination of key got leaked or my homelab got compromised, and the attacker consumed tens of thousands in gemini usage in only a few hours. It wasn't even the same Google project as for my project, it was another that hasn't seen activity in a year+.
Google refuses to apply any adjustments, their billing specialist even mixed up my account with someone else, refuses to provide further information for why adjustments are being rejected, refuses any escalation, etc. I already filed a complaint with the FTC and NYS attorney General but the rep couldn't care any less.
My gripe is not that the key was potentially leaked or compromised or similar and then I have to pay as a very expensive "you messed up" mistake, it's that they let an api key rack up tens of thousands in maybe 4 hours or so with usage patterns (model selection, generating text vs image, volume of calls, likely different IP and user agent and whatnot). That's just predatory behavior on an account marked as individual/consumer (not a business).
EDIT: I guess that you'd still be responsible for the charges though.