Tell HN: Google Cloud suspended our production projects at 1am on Saturday
Google cloud suspended all our projects due to the billing issue in their system they had.
Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.
All the account payments were made and the billing cards are valid. There are no outstanding bills.
Never use GCP for production.
---- Edit: full story by request, long read: ----
Previous month billing didn't went through. Not sure if this was due to the billing outage google had (https://status.cloud.google.com/products/oLCqDYkE9NFWQVgctQTL/history) or financial transaction issue, however we went ahead and made a manual payment covering all the outstanding amount + extra. Despite the payment made, about a week+ later we suddenly started receiving threatening emails "Your Projects are at risk of suspension". Edited and updated the billing cards. Opened a billing support request clearly mentioning this is a production environment and all the bills are paid. They were "investigating" the issue and assured the project will not be suspended.
530 comments
[ 3.2 ms ] story [ 276 ms ] threadI'm referring to these
https://cloud.google.com/support
What I hear is that unless you have at least standard support, they treat you like garbage.
I swear this website is slowly starting to become a way to contact support for big tech more than anything else.
Backblaze has previously locked our account on a saturday.
They claim we were putting too much pressure on their cluster.
They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads.
Also the workload has been quite consistent for days/weeks, so they could've detected it earlier and/or address it in a better way.
They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.
If everything is metered it seems kind of odd to complain about a lot of activity. Since it's literally the thing you make money with.
Minor nitpick, but this is categorically not true. As a customer, one who proactively worked with AWS AMs, SAs, and S3 engineers directly before making the change, I still took down S3 by just changing how new files were stored in our S3 bucket. Yes, this S3 bucket was at the time the largest bucket (based on number of files), but still not “simply unbreakable”.
I mean it sounds like a dick move by Backblaze, but comparing it to what OP is describing is downplaying how bad this looks for GCP IMO.
Going through with official contact at GCP, being assured he did everything right and that it was taken care of - and still getting shut down. In my book that's just inexcusable - if you can't count on that kind of support to work then how can you rely on that platform for anything ?
It has been well documented that any of the big tech companies will thankfully flush you and all you've build down the drain whenever they feel like that. You'll also probably never be able to talk to someone (high enough up the ladder) to figure things out.
Nevertheless lots of people are building their tech and their lives based on the services provided by these companies, when a maliciously flagged YouTube comment for a linked account can kill _everything_ in seconds.
Stop building stuff on their platforms. Now.
edit: Yeah, probably not very "helpful" but how many of the "Google killed my X for no reason" posts will we have to see until either regulatory bodies step in (doesn't look like that) or people will just stop using those services. Yeah, it's cheap and it's easy to start with, but so are some drugs, too.
Regulation, litigation, or split it into pieces.
I should not be able to get customer service from a random McDonalds franchise over a chicken sandwich easier than a company that we spend thousands, tens, or even hundreds of thousands of dollars with.
They do not get to use "We're too big with too many customers" as a defense. They can figure it out.
Google needs to be broken up as badly as banks do. It's a battle that keeps on coming up and here it is again.
[0]: https://news.ycombinator.com/item?id=31317641
I'm yet to hear a single convincing argument of why we need to have organizations of such a large scale nowadays. Could you make a case of what we would be missing if we just "threw the whole thing out"?
If your branch manager incorrectly adds your teller to an industry wide list of do not hire, the board and CEO are personally criminally responsible for it. No hiding behind the corporate veil. I don’t know how to prove whether someone acted in their own and not on the instructions of the employer though. What I’m afraid of is equifax will simply create many companies and divvy up it into equifax backend web services llc that has fewer than 150 employees.
I'd argue that if the split child company ends up only working for the parent, it is easier for the governments to prosecute them. So the child company would have to get other customers and act as a real independent unit or would have to shut down altogether and the parent company would have to go back to the market to find a proper "web service" provider.
I'd also argue that there is no way that a company like Equifax would be able to exist only with 150 people. The whole national level corporation would simply be broken down into many localized offices. Their databases would no longer be shared. The damage that each unit could make would be reduced, and their customers would be much closer to the managers responsible in ensuring that their data is managed properly.
In other words, my argument is that Capitalism and market forces do work. We just need to have these systems deployed in the correct scale.
I don’t believe in measures to radically change the world. That’s both a direct example of the Chesterton fence, and applied game theory - CEOs, board members, or investors, they are all gamers. Give them any set of rules, they will analyse and subsequently game them.
1) Walmart may be a better "business model", but is it a better model for a society that we want to live? I surely don't want to live in a place where big box stores are the norm, and the only alternatives are crazy expensive "niche" stores. And I surely don't want to live in a world where the quality of my food is determined by Walmart's weight, who will favor products that can last longer, are cheaper to produce and are easier to store and transport - i.e, ultra processed crap. If the price to pay to have better groceries is the "inconvenience" that I will have to go to 2-3 separate shops, it's absolutely worth it and I'll gladly accept the "inefficiency".
2) Optimizing for efficiency is a recipe for systemic, catastrophic failures. Just as an example: think of the semi-weekly github outages that are happening. Everyone is convinced that the cost of paying for a SaaS is negligible compared to the cost of operating your own, so they don't even try to set up their own CI and code repository and have to pray everyday to make sure that will be able to work with it. Meanwhile, my self-hosted gitea/drone/docker repo has been running for almost 4 years already with no issues and it requires minimal maintenance. It took me some time initially to set things up in a way that I was satisfied, and it probably doesn't save me any money compared with a off-the-shelf solution, but thanks to my initial investment and willingness to accept these costs I am more resilient than any competitor.
3) Taken to an extreme, focusing on "efficiency" could be used to justify authoritarian governments and the most dystopian worlds. I really do not share this techno-utilitarian worldview that thinks that maximizing economic output can justify the existence of corporations that reduce us to nothing but consumers that can be placed in a segmented box. It's this worldview that is brought us Big Data, the invasion of our privacy, the "gamification" of everything, and so on. Google/Apple/Microsoft/Amazon may all be trillion-dollar companies and may have built incredible products, but the societal/environmental/civil cost is just too much to be worth it. If I knew in 2004 that by accepting the invite to 1GB of Gmail I would be contributing to the emergence of Surveillance Capitalism, I'd never had done it.
Anything else you'd like to try?
Of course efficiency is the goal. Wouldn't you argue our current use of natural resources is unsustainable long term? The only way out of that is to improve efficiency.
Ignore Walmart. If we were to rollback farming to the methods used just a few decades ago, the world would starve. So, yes, we need to continue to improve efficiency. It's required due to resources being limited.
Or by stopping/reducing consumption of things that we don´t really need and have been shoved on us by the corporations. Stop focusing on the symptoms and focus on getting rid of the disease. Let's get rid of fast food, fast fashion, let North America abandon the failed experiment that is called "Suburbia" and get rid of its car dependency. Let's stop expecting overnight shipping for whatever stupid gadget we want to buy, etc...
If the US reduces its consumption to the average of the developed world, instead of having growth-addicted Corporations and Goverments pushing to make the rest of the world consume like the average American, I can bet that we can improve our overall quality of life even on a smaller GDP per capita.
Also, even if you disagree with that: the argument is not against "improving efficiency". The argument is against optimizing for it while ignoring other costs and especially ignoring the fact that most of these optimizations have diminishing returns. E.g: airline travel is something that has been optimized beyond imagination, but it will never be as sustainable as an economic activity as traveling by train or boats. Between trying to "optimize" air travel even more, I'd rather we just decided to ban short haul flights altogether and reallocated our resources to the (re-)construction of decent rail roads and passenger boats.
Do you really think an industry like oil refinery can be managed with a limit of 150 people?
Think of a oil refinery as a monolith, and that such a mandate would enforce that you'd have to build the whole system a set of independent microservices. The industry as a whole would still employ a large number of people, but now they would be forced to coordinate through specific interfaces (the smaller business units) instead of centralizing under a larger corporation.
So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.
His Big Mandate went something along these lines:
1) All teams will henceforth expose their data and functionality through service interfaces.
2) Teams must communicate with each other through these interfaces.
3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.
5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
6) Anyone who doesn't do this will be fired.
— https://gist.github.com/chitchcock/1281611
Maybe the government could do something similar?
Huh, here’s an idea: Maybe Jeff Bezos did what he did because he was fearing just such a goverment-mandated breakup, and wanted Amazon to be prepared for it?
Also, a lot of the work done by people in Big Corp is something that is not outsourced merely because they don't want to let their competitors to have access it. A lot of these redundancies would be eliminated and replaced by smaller units that can serve multiple companies. Companies that have internal "business systems" developers would simply get rid of that and looking into third-party SaaS and/or open source solutions that could be used, etc.
What does all companies mean, are you proposing the entire world follow this approach or just the US. If its just the US then other refineries would just offer cheaper oil and US consumers would buy it from overseas while the US refining goes bankrupt.
1) The price of oil (and basically everything downstream of crude) has little to do with the cost to produce it, and a lot more to do with current market conditions. So even if their productions costs were (slightly) higher, I don't see why a local, smaller, refinery would have too much trouble to keep themselves profitable even if they had to have lower margins to stay competitive.
2) Smaller companies would mean that you'd be consuming a lot less primary resources and your business relationships would be on a much more local scale. You wouldn't be getting a networks of branded gas stations that receive their gas from the same mega-distributor, you'd be more likely than many different gas station owners, each of them too small to be interesting to a foreign refinery.
3) Even if that is not enough and "US refining goes bankrupt"... so what? Why do Americans need to have their own refining industry, if they can be served by others?
Looks like you did not think it through.
You need to have energy independence. When you depend on other countries, those countries can use your weakness to get better deals or control you. If there is a pandemic and all the supply chains breakdown your people can still survive.
You are too deep in "Big State" thinking. Forget about that.
> If there is a pandemic and all the supply chains breakdown.
Under a localist model, your people can still survive because they do not depend on global supply chain and can still have a functioning economy.
What I am asking you is to imagine a less radical version of an Amish community. Less focus on economic output and more focus on resiliency. Try to keep your economy as local as possible and do not over-generalize. Do not think of industries as something that has "strategic value" and instead let them develop as much as it is required to fulfill the needs of your citizens. This way, your community will have plenty of redundancy and no one will be at the mercy of any other external entity.
> How "energy independent" is Switzerland? Singapore? The UK?
Exactly, atleast the UK depends on middle east and Russia which funds their murderous regimes. You should be supporting human rights not murdering innocents.
> your people can still survive
Tell that to the people who died from Covid because they could not get masks which were all manufactured in China.
> Try to keep your economy as local as possible and do not over-generalize.
I think you getting confused with your own talking points, in the previous comment you wanted want the US to be served by others and now you are saying keep the economy local.
What I am asking you is to live in reality and not live in fantasy land, you sound exactly like a politician who declares war on drugs. The current globalized world cannot just cannot function with your fantasy 150 people limit.
You getting confused with your own talking points is one the proof that your limit does not work.
> in the previous comment you wanted want the US to be served by others
No, read again with the whole context: what I said was that the focus should be on keeping the industry local, on a smaller scale and more worried about robustness than profitability. Then I said that "even if after that they are still being out-competed, it doesn't mean that they all is lost".
The point was that plenty of countries can be successful even if they are not 100% self-sufficient about key resources. I am not advocating the end of trade. I am advocating for smaller/stronger communities and stronger/more explicit interfaces between them.
> You should be supporting human rights not murdering innocents.
Please stop with this absurd rhetoric. Not only is BS, it could easily be turned on you ("So, you buy things from China? This means that you support the genocide of Uyghurs!")
> The current globalized world cannot just cannot function with your fantasy 150 people limit.
First, this "current globalized world" is precisely the thing that is so full of systemic issues that we should be working to avoid. In a less-globalized and not hyperconnected world, Covid would not even be a thing, so the whole "people died of Covid because they didn't have masks" is complete rhetorical bullshit .
Second, there is no limit on people. The only limit is about the size of a single corporation. I don't know what is so hard about it to understand. You keep mischaracterizing the argument to the point that it is making clear you are not interested in a good-faith conversation, which should be a signal that I am done here.
> it doesn't mean that they all is lost"
I dont know if you understand capitalism but out competed companies eventually run out of money and get bought out by the more successful one. I think that is the context which you dont undestand.
> it could easily be turned on you
Its not rhetoric, I am being serious, I would love to buy things made locally but that is just not possible with your 150 people limit. You want things to be less efficient and more expensive.
> In a less-globalized and not hyperconnected world,
Again you sound like a politician who declares war on drugs, your ideas sound great on paper but the globalized genie is out now, its not going anywhere for a long time.
> Second, there is no limit on people.
You are getting confused again with your talking points, you are the one who is advocating for 150 people.
> I don't know what is so hard about it to understand.
and yet who are the one getting confused or "misspsoke"
> You want things to be less efficient and more expensive.
There is a difference between wanting things to be more expensive and accepting that this may happen as part of the trade-off being made. Specially so if the idea is that this type of policy could potentially eliminate the concentration of power on the hands of a few conglomerates and create an incentive for automation and to eliminate "bullshit jobs".
> I would love to buy things made locally but that is just not possible with your 150 people limit
Why? Go to any farmers market, is there any step on the production chain that requires 150 people? Do you think (e.g) a municipal ISP to serve 10-20k people can't be operated with less than 150 people? Can't we buy fabric and materials (from small scale producers) and have a small textile manufacturing co-op making clothes?
Also, consider that we are used to having products being completely assembled, but there is nothing stopping companies in a "human scale" economy to work as provider of components that get to be assembled by the final consumer. These components could be made by separate companies. So, instead of having "Google Assistant vs Amazon Echo vs Apple Siri", we would pick-and-choose different speakers, different software providers, different enclosures, etc. The hard work here would be one of coordination - i.e, all these companies and providers would benefit if they worked on a "AI speaker device" common standard - but once that is set in place it reduce the average company headcount. The same logic could be potentially applied to any big consumer industry: clothing, furniture, home appliances...
Finally, let's talk about the software industry. Take any big product from the big companies and you can bet that you can find a small ISV (with certainly less than 150 people) who can deliver and profitably operate an equivalent service. Even though Gmail and Outlook dominate the mass market, smaller email providers still exist and they haven't "ran out of money" and got bought out by the more successful ones. We don't need the big players to serve the population, we could have more of these ISVs acting independently (*). These ISVs would likely invest in opensource as a way to outsource as much as they can to keep their overhead low, which would lead to a even more pulverized industry.
> your ideas sound great on paper but the globalized genie is out now
Again, why? There is no magical force stopping us from preferring local products. There is nothing forcing us to consume indiscriminately. I get that "the system" is too big for any of us and that our individual actions will barely have any impact. But feeling apathetic is not a justification to just accept things as they are. You can not say "I would love to buy things locally" and blame "Capitalism" when you end up buying things at a big-box store.
(*) "Oh, but Gmail/Facebook/etc are free to the user, people won't be willing to pay for it!" Well, the argument could be that these people would either have to find a smaller provider willing to do the service for them (their employer, some non-commercial community, the tech savvy family member who wants to self-host?), or they would indeed have to learn about TANSTAAFL.
I find it very hard to believe that this can not be automated to the point of requiring much more than a dozen controllers (4 groups of 3 people, on a 12-on/36-off work schedule). And there is nothing stopping this "top-level" department to be a company in itself.
But anyway, let's say that I'm underestimating the amount of human brainpower that is needed to get this work done safely. Not to be a Luddite, but shouldn't we be asking ourselves if there really is any type of (isolated) economical activity that is so fundamental for us that can only be done with more than 150 people involved?
It's not like we can not refine oil with less than 150 people. It's just that we can not do it at the scale and efficiency that we are used to, right? Instead of having one "giant" refinery that requires thousands of people, perhaps the limitation would force us to have smaller plants spread out around the world, or to have the downstream industries actually doing the refinement themselves, etc.
So, how about instead of looking for maximizing the efficient usage of resources in order to keep the economy growing at an accelerated pace, we start to focus on a constant growth rate and better distribution?
Until then, EU regulation will be just a demonstration that money and power still talks just like in the US, the only difference is that Europeans are better at fooling themselves about their moral superiority.
Anything less than that and all you are getting is an incentive for the make a risk calculation between profit and losses of being caught.
> Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well.
Which would be a sign that the regulations are not really working and consumers are just getting fooled into believing that their beloved EU is oh-so-awesome.
We could spend the whole day in this pointless display of "my dad is better than your dad", but can we just skip it please?
Having lived in the US and now living in Germany for over 9 years, I have no intention to leave. But this idea that "regulations" could fix the consumer hostility of Big tech is naive at best and damaging at worst. To assume that the differences between US and Europe are due to lack of regulations is a terrible mistake of reversing cause-and-effect. Cultural differences between and Europe can explain a lot better why things work different, while "regulations" assume that people are just automatons who can do nothing but follow orders established by some higher authority.
Regulation has its costs. Predictable.
> Cultural differences between and Europe can explain a lot better why things work different
Yep, the cultural differences that lead to even a "left" US government to be a lot more hands off than any EU government :)
But that logic applies to anything that any company does. You want to effectively kill any company for any infraction?
That sounds a little too severe.
My memory may be hazy[1], but I don't think Dieselgate was ever proven intentional; your proposed policy wouldn't have kicked into effect anyway.
[1] So feel free to post links to the findings if you have them handy
[0]: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
Dieselgate is a case of corruption in Germany, a particularly horrific and embarrassing one sure. But the existence of corruption in a single member state is not sufficient to disregard EU-wide regulatory efforts in a broad swath of consumer-facing industries.
"Strict emission laws" did not stop EU car makers to focus on ICE cars, it worked only to give consumers a false belief that their cars were not as clean as they believed. It's not regulation that is changing the industry, it is Tesla.
GDPR did not (and will not) stop big companies to collect and exploit user data.
"Right to repair" did not get the big manufacturers to stop producing devices that are consumer friendly and don't stop planned obsolescence.
Europeans have this stupid sense of moral superiority, but the people still buy from China, still look at cost above ethics and just use the "but our govt is strong" as an excuse to redeem themselves of personal responsibility.
Unregulated auto tend to have a bit more lemon markets than regulated auto.
Unregulated healthcare tend to have a bit more snake oil, blood letting and cocaine than regulated healthcare.
Unregulated food sector tend to have a bit more Listeria and Salmonella then regulated food sector.
No regulation are not know for being that nice either, which is to say that some regulation is good and some regulation is too much, but one can't dismiss regulation as just being bad.
The issue of regulating big industries and big tech in particular is what outcomes we want to avoid and how a healthy market can be created where one does not exist. Words like infrastructure, platforms, and lock-in are all associated with a lack of market choice and competition. Bad support could be a symptom of that, but the target of regulations wouldn't be about support but rather to avoid having a situation where citizens are unknowingly building their homes and companies on quicksand.
I prefer modern medical system with antibiotics, anti-virals and anti-inflamatory drugs.
No one said that.
2) No, regulation (or the lack of it) has nothing to do with it.
3) No one is arguing for the end of regulation.
There have been too many stories on HN and twitter over the years of business critical google accounts being suspended for no reason or google cloud products massively increasing their price overnight.
I don't trust google as a provider for business-critical infrastructure. I'm far from alone.
I would imagine it would be more than 0.1%.
I suspect it's far more than 0.1%. They have a bad reputation and word-of-mouth is now working against them.
Perhaps even offer to pay court costs.
They can say:
> We suspended your account for violation of XYZ rule. You can appeal internally, and if that is denied, you can ask a court to rule on if you broke this rule. If the court finds you did not violate the rule, we will reinstate your account. We will provide any data from your account to the court to make its decision, including proprietary data that we cannot reveal directly to you, for example data about other linked accounts.
Of course human ops is originally about reducing actual traumatic jobs like reviewing actual evil images, but its basically now all brain work which techies hate.
1. Ad business should be one thing 2. Other products should be different companies: Gmail, GCS, Domains, etc etc. They should not be "cushioned" by Ad business revenue.
That way they will play an even playing field with similar companies. And I think Amazon/AWS should have the same split.
If that happens, these companies will have to differentiate to their customers with non-crappy customer support.
- technically amazing: BigQuery and GKE are great
- really half assed as a product: console is somehow slower than AWS despite the latter being the most tech debt of the lot for having invented everything, support half witted, and honestly I had a card declined and even Alibaba Cloud didn't suspend my shit
If you have one of their premium support things you'll get faster service but I think I'm not hyper keen on them for general stuff.
If you have a spend of over $2500/mo you can apply for invoiced billing directly from Google or if you have less than that you can contact a GCP reseller partner to get invoiced by them.
Some GCP reseller partners also have a close relationship with Google and the contacts that come with that relationship.
Why should a business relationship with Google (or any other cloud service) be any different? I don't use GCP but if I was going to start, I would ensure that I have a person to call or its a non-starter.
Depends on the size of the account surely? Maybe there's some support person to talk to, but there's plenty of services I subscribe to fo $10-100/month and I dont have an account manager for any of them.
Google however is infamous for not providing support at all on all their services, even if your complete digital identity got killed off because some AI can't recognize that the "CSAM" you sent was in fact communication with your child's doctor [1].
[1] https://www.heise.de/news/Nacktscanner-Unbedachte-Fotos-vom-...
Because google choose to make this as difficult as possible. A couple of years ago I worked on a third party IAAS database cloud product (think Atlas or Elastic Cloud, but for a different database), and while Azure had the worst tech overall, Google were ultimately the hardest to deal with by a VERY long way because they don’t acknowledge the existence of people as anything other than ad targets.
2. Unlike a small hosting company, Google is so big it will suffer essentially no financial consequences if their robots make a wrong decision for credit card customers a few times per month.
So there is no cost to Google for leaving the system as is, and a high cost for your solution. The decision is a no-brainer from their point of view.
Perhaps the latter is the more practical solution indeed, though.
Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.
Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto.
Essentially the hosting company is offering a cash equivalent for unreliable credit.
This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose.
This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target!
The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash.
Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks.
So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops.
Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1]
An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale.
Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine.
Axed.
[1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.
If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png
They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.
Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/
https://en.wikipedia.org/wiki/Bystander_effect
Having multiple people that receive billing notifications is strictly better than having a single person - what happens if that person is on vacation, sick, overloaded? No one even has a chance of catching this. If at least one other person receives those increasingly important emails, they at least have a chance to catch this before it breaks.
It seems to work—at this moment, anyway—even in private mode for me now though. ¯\_(ツ)_/¯
Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case.
Though I'll agree that weekends should be considered off-limits regardless.
Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.
I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...
The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.”
And they had “just conducted a manual review” and confirmed the suspension.
Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues.
I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.
It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy.
Incidents like this turn customers off for life.
Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.
They proceeded to DDoS my droplet.
Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy.
As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.
Although crimes and thefts are rare, all developed societies invest millions into law enforcement and justice systems to measure and mitigate them.
Although device and equipment failures are rare, both private and government systems exist to address them fairly.
It seems like it's only in the tech industry that statistical insignificance of failures is assumed and then wielded as justification to avoid even minimal levels of accountability.
Stealing this, it's far more broadly applicable than just to this sort of billing issue.
If I had €1 for every time a politician has dismissed some incident as a random incident I could probably buy a high-end MacBook.
Most backup systems can't do efficient incremental diffs either - so you end up sending your companies entire database every 12/24 hours, burning through a lot of bandwidth.
And when you support snapshots in your backup software, it's very tempting to then drop support for replication...
Replication logs also have another problem - since they include every data change, certain types of database updates, for example a large field that changes hundreds of times per second, can work out far less bandwidth to just snapshot periodically.
Overall, all this stuff is a lot of complexity most businesses don't want to have. They just want to have a simple backup solution and get back to building their product. And that usually involves using some managed database where backups are just an option in the control panel. Or a self-managed database where backups are disk snapshots. Or a script that copies a database dump to S3 every day. But it hardly ever involves streaming change logs...
Not free, certainly. But it's a system requirement there to be able to rebuild the system in a non AWS environment.
If that’s how you feel then you should be applying that logic to any provider. Even if you rent rack space in a data center. Because unless you own everything through and through. This could happen anywhere.
> "We're fully redundant across AZs, regions, and even cloud providers!" crows the engineer with a single corporate credit card backing the entire house of cards.
https://twitter.com/QuinnyPig/status/1288275701389389825
You could backup to a different account, possibly in a different region, with the same provider, as long as the other account has a different payment method. Using one provider would in most cases reduce the amount of effort to get back up and running.
The key piece would be how DNS is structured and hosted/billed.
And yes I think people should engage lawyers in cases like this. Even in small cases court if it's the case
That's not true in the long run. Legislators can make arbitrary laws that ban or prescribe arbitrary things.
See eg how there are special laws in Spain and Australia that (try to) force specific behaviour from Google News. Behaviour that you can't override with a contract.
Take a look at the Digital Markets and Digital Services Acts on top of the already active GDPR.
Hmm, I feel this is opposite of what people normally complain about here. Every time EU does something about big tech, HN users will pop up saying it's just because they're jealous and wants to harass US companies. And other's saying the US are just afraid of hurting their own companies and look the other way so they can grow big.
https://en.m.wikipedia.org/wiki/Antitrust_cases_against_Goog...
$8 billion afraid. If you consider that 'minor', i rather suspect you are being anti-EU just for the hell of it.
Back to this, had any political authority (the EU/US, doesn't matter) applied the same principle in here, i.e. "we won't allow you to sell any cloud-related services anymore until you don't have proper customer service" let's say, I can assure you that would have done a lot more to alleviate problems like this one.
Yeah, and shut down everything that everyone has running on Google's servers on the process. Killing millions of small to large businesses in the process. That does make sense. The way to enforce food regulations is to shut down the food stores even at the cost of leaving people hungry.
Look at all the banks shifting their infra to the cloud. Haven't heard anything bad happen from that. And their uptime and business requirements are as stringent as anyone else, even further scrutinized by regulators and auditors.
That said, I think cost is more likely to drive change here than fear over billing - most people pick lover cost over risk mitigation far more often than they'd like to admit.
It won't go all back to on prem because cloud providers will slash their margins once they struggle to maintain growth, and doing so they'll diminish the advantages of on-prem/hybrid setups for a lot of people, and so we'll find some equilibrium or other.
I'd expect more emphasis on hybrid between multiple clouds rather than cloud vs. on prem to happen first. E.g. layers to threat more parts of cloud services as commodities.
I'm all for hybrid solutions, but mostly because it means you usually end up being able to cut the cost of on prem, colocation or managed servers even more relative to cloud setups because you can increase the utilisation rate (e.g. run your base load on cheap Hetzner servers but being able to spin up EC2 instances on short notice to take spices or handle failures up to and including all of Hetzner falling off the face of the earth). Most larger managed providers today also offers cloud services and many also offers colo services, so you can often easily mix and match as long as you're conscious off egress pricing which is often the biggest barrier to such setups today (especially if the big cloud providers are in the mix, though it's better than it was).
That said, I prefer not putting my eggs in one basket, and instead going the direction of putting in place layers to treat a multi-provider setup as one. In the past I've e.g. done zero downtime migrations between AWS->GCP->Hetzner that way, and also had systems split between actual on prem (as in a data room at our office), multiple colo's, dedicated servers at Hetzner, and a couple of VM providers where everything was transparent to our engineers (they didn't need to know on what continent a service ran unless doing performance optimisation or reliability engineering). We prepped to tie in AWS resources too, but it never become cost effective for that company to do so vs. the prices of the other providers used - it would have been if we had sudden extreme load spikes, but it was a business where traffic was linked closely to physical capacity at restaurants and nightclubs, and so traffic was very predictable.
Most companies aren't actually at the scale where they need cloud. Cloud comes with the benefit of minimal to no need for ops/devops, high uptime (when nothing goes wrong lol), integrated DDoS protections, APIs your junior devs can string together, and the illusion of infinite scaling.
In reality, it turns out you do need someone to effectively do ops, you don't have 99.999% uptime, your don't need most or any of those APIs, and you didn't need that promise of scaling after all.
With DDoS protection, on-prem hosting or independent dedicated hosting might be more economical and practical.
The other big benefit is auto scaling, in 2012 lots of companies would launch and be unable to meet traffic from a big push from reddit or HN or others. Today it’s rare to see a new product’s site go down due to a “hug of death.”
If you do in-colo hosting with leased servers you've already cut the big initial spend drastically (e.g. deploying to a new colo for a past employer involved buying a pair of switches and some cables), though you're still usually tied in to a lengthy contract. Doing managed servers usually comes with little to no upfront cost (sometimes a low setup fee) and often month-by-month contracts.
With respect to the on demand pricing benefit vs. monthly, my experience is that the price differential is so steep vs. the cheaper managed hosting providers that it's only usually viable for loads that run less than ~6 hours a day. Most sites do not have a variable enough load to justify the engineering cost to use cloud to handle their normal day/night cycle that way - usually the variations are too small. Some do.
But note that this only pays for itself if your base load is not on a cloud setup, as the base load tends to dominate and the base load cost of most cloud setups is high enough not to be outweighed by the increased flexibility vs. a pure managed/colo/on prem setup.
It gets worse for cloud: If looking for the most cost effective, they're not competing against a pure managed/colo/on prem setup, but against a hybrid setup which can auto-scale into the cloud but usually won't. If you e.g. deploy containers, all your need is to tie cloud instances automatically into an overlay network and feed data from your monitoring system into a tool that adjust min/max instances in for an autoscaling group in AWS when load exceeds a threshold, for example.
For a typical "pure" on prem setup you'd usually aim for the daily peaks to rarely exceed 50% of on prem capacity. Maybe a bit more if you have a very predictable business, or even less if your traffic is very unpredictable.
But if you add capability to that setup to spin up and tie in cloud instances to handle peaks, and you can push that into 90%+, or even above 100% - basically whichever number turns out most cost effective. Usually if you optimise that for cost, this means you'll end up with a setup which almost never spins up cloud instances, but which is now vastly cheaper relative to a pure cloud setup because it's gained the benefit of auto-scaling at a far higher load factor than would be safe in a pure on prem/managed setup.
For a 10-20 person company that's a huge expense with not that much ROI.
Some examples:
Suspended GCP even when everything in order - https://news.ycombinator.com/item?id=32547912
Ban app even when competitors are similar - https://www.vice.com/en/article/z3vdpj/google-play-store-ban...
Play Association ban - https://old.reddit.com/r/androiddev/comments/ts6jfg/google_h...
Ban due to wrong wording - https://twitter.com/hermaritz/status/1371383715381805061
Terraria banned - https://twitter.com/Demilogic/status/1358661840402845696
New project banned - https://medium.com/@amton15127/why-you-should-not-use-fireba...
Google bans company - https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_gettin...
Google bans mail - https://talkingpointsmemo.com/edblog/a-serf-on-googles-farm
Ban app for communicating changes during covid - https://news.ycombinator.com/item?id=23221447
Adwords ban - https://news.ycombinator.com/item?id=23224791
Serverpunch bad support - https://news.ycombinator.com/item?id=17431609
Delete app - https://news.ycombinator.com/item?id=20826618
Google bans game with pandemic - https://news.ycombinator.com/item?id=23229073
Google bans dev with no recourse - https://news.ycombinator.com/item?id=15197357
The business continuity risk of using google is just too dangerous, while the odds are low, the consequences are disastrous for a business and there is never any recourse.
This isn't really 100% true. It's just that Google uses third party contractors for the grunt work and they are working off a script. You can contact them but the comms with them is weird you won't get a direct response only canned responses as they are not allowed to talk to you directly. I'm pretty sure that these contractors are working out of India due to the times they would "respond" to the ticket.
I went through this recently with one of my apps because you can use the Google picker to load files. Very strange and stressful process I was this close to just deleting that feature due to the insane process.
Example was I had a spelling / grammar mistake on my landing page (minor) and they knocked back the application by sending me a premade email with all the rules that need followed which was full of spelling mistakes. Then once I corrected this they said I had a broken link (it was a link to instagram which didn't work and I took it off) again the email they sent to me was full of broken links. Very weird.
This only happens if you're using personal google accounts / @gmail accounts to manage gcp resources. Even if you don't use Workspace as your main directory, you can set it to be a SAML consumer so you still get full Workspace data governance.
I don't recommend anything else but have you heard any cases about digitalocean or amazon or fly or other platforms? also it's possible to write to support and get information about any kind of the issues that you are getting for any kind of the reasons.
Account suspensions are manual. If you have billing issues at most you will be limited on what you can do (deploy, scale...).
They went from: (Remember, companies are people!)
We have to make choices about big expensive things. - 0 people to blame besides them. Vendors / Consultants make choices about bigger and more expensive things for them - 1 person to blame besides them. Consultants + "The Cloud" to blame - 2 people to blame besides them.
It is a feature to have many people to blame, in these type of environments.
https://medium.com/hackernoon/why-and-how-we-left-app-engine...
https://dev.to/codenameone/google-play-kafkaesque-experience...
There's this story from a couple of days ago which is shocking: https://news.ycombinator.com/item?id=32538805
Google is a terrible, dystopian company.
The previous month's billing didn't 'went through' and you are unable to zero in on whether this is a Google issue or a you issue.
I get that it is easier to complain on the internet and often cathartic do so as a coping mechanism.
Having an account suspended because of a billing issue is not exclusive to Google.
Have you moved your projects off GCP now as voting with your wallet would be the sensible thing to do. Until you encounter a billing issue with AWS and move again to Azure and then to DO.
I'm sorry but :
>Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.
If he contacted someone at customer support, after making sure the mistake was corrected (on whatever side it was) and they assure him that the account will not be suspended - what the fuck is he supposed to do about it beyond that ?
Imagine having your business/job fucked up over an incompetent customer support system.
How you do that can vary, but ... billing issues are always a risk. Having two cards, with two different banks, and two different names, is a pretty basic precaution to take. Having multiple cloud providers closes out a bunch more existential risks.
DNS is the one unavoidable SPoF for most Internet businesses: you cannot have a "backup registrar". The best you can do here is pay for a quality service provider: someone who will call you if there's a problem, before they cut you off. That's going to cost a lot more than GoDaddy, btw.
So it must have been automation. And there is a very good chance that automation simply hadn't been blocked by the support agent which had been giving reassurances.
In our case our bank blocked transactions to AWS and Google one night. I contacted the bank and Google, made arrangements and successfully paid the account within a day. I just retried the AWS payment and had no further issues with them.
I had issues with Google for months. Google kept sending me threatening messages about how our accounts suspension is imminent. We'd then follow up with a human, have it resolved, only for the same thing to happen weeks later. Note that there were no outstanding payments, and no failed payments again.
After ranting to friends in startups it became apparent that this was not an isolated incident, and it's always Google.
You have to be joking or ignorant. I have never met someone who had a billing issue with AWS that wasn't able to be resolved without shutting off access.
I have never seen a story bubble up on HN about someone who had their account terminated by AWS for no obvious reason.
Perhaps Google is subject to an ongoing smear campaign, or they just attract people who don't know how bill paying works and Google has been completely reasonable. What I do know is that both AWS and Azure have been completely open, honest and will send engineers, account managers or whenever was required to get problems resolved without business impact when I was at both large corporation and early stage startup.
I have never encountered this with any interaction with Google, so it can't be part of their corporate culture, which makes it very unlikely that Google Cloud can be much different.
No, they attract and select people who see the world only from an engineering perspective. Combined with Google's cultural roots that come from MIT, where projects and services are things that are funded by the government, military or megacorporations in which you can pull the plug whenever and the other side would not flinch, this makes them totally blind to the existence of humans on the other side of their services. Resulting in an false reality in which you, me and the millions of other people who depend on their services do not exist.
So you see things like this happen. Things like 'deprecating' products in people's faces. Things like 'combining and merging and unmerging products because it makes more engineering sense' and creating messes like "Google Gmail Hangouts Chat Rooms Meet" or whatever they are calling the frankenstein they made by combining all those.
They're playing catch-up, and customers are at risk while they do so.
(1) AWS had a head-start on Google and Google's been chasing AWS's monetization strategy and away from their Google App Engine approach, which was fundamentally different. Azure may be more contemporary to Google's Cloud, but Microsoft has a tried-and-true large-scale corporate customer support engine that Google's only now being forced to build out to support its Cloud offering (the customer support story for ads was fundamentally different, and Apps for Domains didn't have the scale Cloud now does).
That was really all I had: expressions of astonishment at the prevalence of Google apologist comments here today.
In an effort to be constructive let me say, it's a matter of scale. Google can't be fair because of their scale. Contrariwise, I use small-scale services and get outstanding customer service.
E.g. Tarsnap, cperciva is a god among men when it comes to customer service. Or sr.ht (Sourcehut) ddevault is right there on IRC most days. Or Podia, I wanted a certain font on my site, they didn't have it, I emailed them about it and they literally added it within the hour. Give these folks your money, eh?
IF you can get ahold of someone... and IF they comprehend your issue... and IF they know how to solve it... and IF they can communicate with other departments involved... and IF they remember to respond to you... and IF their internal documentation is up to date... then you MIGHT stand a chance of getting your problem solved.
It's at the point where you will get better support for Google's own services from companies that resell them, because they have on-call internal contacts at Google and actually care about their customers.
It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the company. That's why I had to step in to try to save them.
Google, AWS are still charging people mid-2000s bandwidth prices. Its outrageous. Digitalocean's egress price is 1/10th of Google's cheaper egress pricing. Same with Hetzner. And with Hetzner dedicated servers? Its unmetered.
here it is: https://news.ycombinator.com/item?id=32242987
> - The link says "my friend got this", so even the person who authored that post isn't claiming to have received this notice. This is a second hand report, at best. For all we know, their friend received this notice years ago, and mentioned it to the OP, who didn't realize it was old news.
There have been people confirming in recent HN discussions that they were using full 1gbps of their dedicated box for a while since Hetzner declared their true-unmetered policy a year or so ago.
https://twitter.com/pontifier
What a fascinating concept.