What left a bad taste in my mouth is any quota increase getting rejected unless you got on a phone call with sales and listened to an upsell speech. I'm trying to give you money and you're putting roadblocks in front of me.
I had a similar problem. We wanted to get on a call to resolve it quickly, but was instead forced into an email chain where I received a reply every 3 weeks.
I had it happen three times on the same account including a hidden quota that the UI didn't even list. Sales person said that it might get better if we moved to invoice billing instead of a credit card (why?). After that I just gave up and moved back to AWS.
edit: It's also not a new account, very consistently paying for some services (maps, etc.) for years before I decided to ramp up.
edit2: It was also a tiny quota increase from the default one so not like we suddenly asked for five hundred instances.
I don't know why all the cloud companies make you jump through so many hoops to get more quota - but Google has consistently been the fastest for me. Fill out some random Google Docs form (why does it have to be so dodgy?), receive quota within 24 hours.
Contrast:
Microsoft - flat out refused me more quota despite spending 10k/mo with them. Required me to convert to invoice billing, and then wanted a bunch of proof of incorporation and when my trading name didn't match my registration name they were unable to proceed.
Oracle - took 3 months of escalations and deliberations, required me to explain on the phone to a VP why I needed the quota.
AWS - frequently requiring me to write up a spiel about what I'm going to do with the quota before they approve it, increasing the RTT to 72hrs+ - do they actually verify this? How would they? Why do they care so much? We've spent 50k+ and always paid the bills, what's the issue?
There are several valid reason to do this - concerns about customer solvency and making sure you aint gonna crash tbheir house of cards architectures to name a few. The bigger problem is when those requests get stuck in the bureaucracy hell and all three major cloud provider companies are known to be extremely bureaucratic
Microsoft was the most baffling - happy to give me 25k+ credits for free in their startup program, unable to let me pay them for the same setup going forwards.
My quota requests weren't outrageous - never more than 20 VM's, albeit very large VM's.
It is not your quota that is the problem, it is the credit risk.
If you paperwork is not up to spec, they are running the risk of credit exposure when you the customer doesn't' pay. Higher risks for any provider when there is no legal entity to sue etc, which is why they want you to convert to invoice billing.
Wait, really? Quota are recorded by code that must be changed by pull request, not an entry in a db? That sounds like an insane waste of engineer time.
- creates a PR with the project ID and the requested value as an exception in a file (this requires OWNERS approval, so at a min one eng/pm to approve)
- file would update a DB the next time it's picked up
It's not as bad as you think, the PRs are written on their own and the engineer in question probably blanket approves all of them every morning while they're checking email (quickly scanning them looking for automated red flags).
Not generally. There is a quota service for all modern gcp APIs that handles per region, per user, and per project quotas that internally you just need the right permissions to update for a customer request, no PR required.
I was really impressed by how simple it is to increase quota on AWS. It was just simple elastic IP quota increase and few things related to VPC quotas. I haven't used Google cloud as much, so I can't comment on it.
If you don't mind me asking, what quota increase was rejected?
A relatively small GPU increase so we could do some machine learning project testing. We asked for I think 8 T4s which are like $180/month each if run 24/7.
I don't know, seems to be mostly filled with speculation and anecdotes. I can find similar anecdotes of bad behavior by Amzon, for example claims of using AWS to steal its users' business ideas[1].
Admittedly, the original author's title of "Why I distrust Google Cloud more than AWS or Azure" much better describes their position than the editorialized title of the HN submitter ("Why Google Cloud is less trustworthy than AWS or Azure").
> Admittedly, the original author's title of "Why I distrust Google Cloud more than AWS or Azure" much better describes their position than the editorialized title of the HN submitter ("Why Google Cloud is less trustworthy than AWS or Azure").
I thought the new title better but I did not want to change the original (because I had already shared the link with friends), so I just changed when submitting. Should've gone with the original...
I can't speak to Azure, but while AWS has a policy of never discontinuing services or features, even if they are replaced by something else, whereas GCP does discontinue entire services (although if it is generally available they have to give 12 months notice according to the terms of service).
RBE was discontinued during alpha. They tried a thing, it didn't work out for some reason, so they decided not to bring it to market. This hardly fits the bill of the typical Google deprecation.
“Reliable”. In addition to major outages that normal dcs have they also have dozens of small ones that never make it to the status pages. Even for simple stuff, more so for managed stuff. They are some good reasons to pick public cloud but if you’re picking it for reliability you will be disappointed
I see the Google hate on HN is strong enough that an article that just regurgitates the contents of a few articles and blog posts can be upvoted to #1 in short order.
The issue here is that the majority of criticisms apply to Google products and services _other than_ GCP. By and large, many commenters here (myself included) have had very good experiences with GCP. Less so with Google Reader.
```
I don’t know what would happen if the media starts picking up a theme that Google is secretly building AI weapons or AI technologies to enable weapons for the Defense industry,” she continued. “Google Cloud has been building our theme on Democratizing AI in 2017, and Diane and I have been talking about Humanistic AI for enterprise. I’d be super careful to protect these very positive images.
```
Not being judgemental about if defense is right or wrong , but its fair to say we are in the running for it. What is not okay is "we care for humanity and freedom of speech" and then do backchannel discussions on how dissent can be quelled with government agencies.
So to me distrust is for the company possibly and probably not so much on the reliability of its services
And the odd thing is that Google has been significantly better for consumers than other FAANG companies.
Apple has historically been anti consumer and anti developer with a huge marketing budget to wash it. Facebook intentionally makes us sad. M$ and their anti competitive practices should be well known.
What? Apple has been pro-consumer to the detriment of everyone else. Developers are still screaming that they aren't allowed to install malware on my iphone.
"Let" Russia? I am pretty sure that Russia is the one firmly in charge of that decision, not Apple.
It isn't inconsistent to push back on government overreach where it is legal to do so, but conform where they have no other legal choice except dropping an entire market.
There's definitely some voting manipulation going on on HN last few years. At the same time number of irrelvant or barely relevant negative comments with no insight sky-rocketed. This trend coincided with the appointment of new mod dang. I guess the current mod is trying to make HN more popular and using the age-old outrage playbook to get to that end.
Google generally does not come off to me as having a great customer support culture. To appreciate the scale of the issue, take a look at the hoops and blackboxes merchants who want to be listed on Google Shopping have to go through and all the horror stories online. I did notice on Google Workspace and Google One that they do seem to be trying to improve the support experience.
Gotta love a typical hackernews comment begging for links instead of just searching for it yourself. On the contrary I don't think your comment is useful either.
Google cancels products so frequently that no matter now cool it may seem I rarely even bother checking them out - I can't stand becoming reliant and then having the rug pulled out from under me
GCP does deprecate products (former App Engine PM here, deprecated many APIs), but it's definitely less frequent than the "Google deprecates everything" memers want you to believe, and there's a minimum 12 month deprecation period before literally anything is deprecated.
I'm not sure what the author (or Yegge, for that matter) expect a good timeframe to be for deprecating services.
Anecdotally, Google announced the changeover of Cloud logging API versions in October of 2016, with a 5-month ramp (October to March) to switch from the v1 beta API to the v2 beta API. Five months is nearly two quarters, which is quite a long window for a beta API IMHO.
That having been said, Google's habit of leaving things that are pretty much mission-critical in beta is unwise, but it should be unwise for them, not end-users. End-users that need reliability and low churn shouldn't be developing on beta-anything.
Giant changes like that are worth capturing in long term planning processes, and then you need time to get ramped up on the new stack, design and implement the replacements, run all your backfills, and also, still have a couple years to figure out the parts that don't migrate nicely. With enough time available, you also don't have to drop actual business related improvements, even if your progress slows down a bit
I thought Yegge was rather specific about what he thought. He specifically mentioned a bunch of things (Java, emacs, AWS, Android, browsers) that take the approach to deprecation that he wished to see.
I would sum up his ideas as: "If you offer a service I pay for, and deploy code on, you should not break my code while you still offer the product."
I also don't think he'd have bothered to mention one or two minor things. His examples were rampant and incredibly bad, like breaking their own offerings.
About deprecation: the gold standard is what AWS is doing with SimpleDB: essentially, never.
The thing here is that if you run hundreds of services in production - many of which work smoothly and you don't need to touch often, you will find that Google's habit of forcing you to change how to use their tooling will generate a huge burden...
They discourage you from using it and make it clear that for every use case some other tool at AWS would be best, and have been doing so for several years now... They wont even list it anymore under https://aws.amazon.com/products/databases/ ..
Still, they support it ( https://aws.amazon.com/simpledb/ ) because there are customers with legacy systems that depend upon this service.
When I was doing deprecations on GCP, it was minimum 12 months for any deprecation. Five seems shockingly fast!
As for the beta thing: GCP's definition of beta was basically everyone else's definition of GA, since the GA requirements were so insane (e.g. 99.999% internal availability) that getting there would take literal years (see the GCF beta to GA taking like 18 months?). I totally agree that it's weird that things would stay in beta for so long, as opposed to hitting industry standard levels so users can have confidence in them, but setting GA as far higher than the industry was part of Google Cloud's plan.
Seems pretty thin. A handful of price hikes that made the news is not something that would bother an ordinary company. Maybe a little mom and pop that's running on tight margins, but they would be better served by squarespace anyway. Everyone in business understands that sometimes you raise prices.
The only reason I didn't try Google cloud is payment. I can't even run a couple of VM, some test db without using the credit card. This means that if someone hacks me, if the documentation isn't clear enough or if I make a mistake I will fully pay for it (and immediately). This is incredibly haunting for beginners and people that cannot afford such scenarios.
With AWS at least I can use a prepaid card: if something bad happens for any reason, at least I know I can afford to eat the day after.
This seems like a particularly strange reason to not try their service and to me is akin to saying "I won't use CarRentalCompany because they require a credit card. What if I'm new to driving and total the car by wrapping it around a tree?"
If anything, this is just AWS being overly generous and forgiving.
I'm shocked that any cloud provider lets you use prepaid cards... all of the major providers have problems with crypto mining and abuse, so it's crazy that AWS would allow prepaid cards that might be laundered, etc.
And IMO, if you're a real customer, all the providers are fairly forgiving, provided you can get in touch with a real human who works there.
I hate google and would NEVER use GCP for anything remotely meaningful (I'd use Azure and even Oracle first). Nevertheless I had a domain issue (through google.Domain ) and I was pleasantly surprised by the customer support. Almost as good as AWs paid support.
Otoh I find AWS quite straightforward, but I've been using it for several years now. Their support is worth their weight in gold.
While I agree with the automation piece, guessing first.last@google.com is going to get someone a large % of the time. You can reach an engineer or a PM with about 10 minutes of stalking LinkedIn.
Also, if you're paying GCP any reasonable amount of money, you have an account manager who will respond in < 24 hours to connect you with the right PM to deal with the issue. Google deals with lots of random humans, GCP mostly deals with actual businesses. As much as I hated the support offshoring Google (and sometimes GCP) did, most actual businesses could get a human fairly quickly.
Answering tickets is partly the kind of support I look for, Azure will get their account managers of my customer connected to me, Both AWS/Azure are okay to come on customer calls with me for a large enough deal. I never got that kind of support from Google.
I'm one of those "you'll get my baremetal and systemd out of my dead cold hands", kind of guy.
But I have reasonable exposure to both AWS and GC and I can say that, by far, Google Cloud is easier to reason about. As a consequence, it's much harder to misconfigure. The 2 large AWS deploys I've seen have, at best, had billing issues no one really understood (incl AWS), and at worse, security issues.
Complaining that maps prices went up re Cloud Hosting is, to me, like complaining that Amazon raised the price of the Kindle, e.g., not particularly relevant.
I just completed a migration from AWS to GCP. My experience was previously entirely AWS, but GCP has been really nice. GCP has fewer features (e.g., no scheduled filestore backups), but GKE is far and away better than EKS and the overall console UX is far better as well. There’s also generally less to understand and thus misconfigure, and far fewer half-baked features to wade through to find the happy path (I’ve spent way too much time using CloudFormation).
I haven't used Kubernetes on either platform - so there may be more to that.
One thing I really dislike about GCP is how expensive it is for personal or hobby use. I burned through $300 for a simple vm on GCP in a few weeks because their cheapest instances are so expensive.
Yeah, navigating either platform is tough for hobbyists. I can get more lambda invocations than I’ll ever use for free but a single load balancer is like $30/month, never mind instances.
That is incorrect. It costs $75/mo and they give you that as credit. Also why use gke if you’re trying to learn kubernetes? Single instance kubeadm cluster is perfectly fine for that purpose (even better)
If you roll your own cluster without GKE, I believe you have to configure your own load balancers, ingress controllers, etc. Having some of that ready to go out of the box allows you to learn Kubernetes concepts more gradually.
I used to think the problem with AWS was pricing and hidden costs etc. But in reality it’s because companies just let developers run wild without restriction on AWS and end up over provisioning or pulling in expensive services to solve dead simple problems.
The issue is definitely not AWS. It’s always the developers. You really need a gate keeper to AWS to question why you need a service and ask for a price estimate on cost and usage.
Funny, since AWS has much more granular control over IAM roles and users than GCP does, so that the infrastructure/security group should be able to provision devs with the ability to roll their own IAM in a scoped way to prevent issues.
In my experience trying to configure AAD policies, AWS IAM and (to a very limited extent GCP IAM), it does not generally require a large investment in time. It does require a development account in which the developer has full access to IAM/AAD.
At my employer, we have a gatekeeper team who is terribly overworked and hardpressed to push back too much when business outcomes are at stake. One of the more successful things theyve done is create a terraform repo anyone can contribute to. They will review PRs and manually apply changes for production accounts. Whats great is that these folks can take my PRs that are 80% right and they are able to help me achieve least privilege better than I could on my own. However, other devs really dont care about least privilege and they tend to go for large open policies.
AWS's IAM policy is far and away the most sophiscated and granular, and even has a nice UI now. Trying to achieve this in Azure is next to impossible because you must have extremely high permissions to even be able to make new roles/policies that are super granular.
Also, permissions boundaries are specifically made for the use case of "IAM teams delegating some control to devs".
IAM team creates a "developer admin" role/user that can only create users/roles that have a permissions boundary on it. That way, no matter what policy the dev admin grants, the dev user can only do what the permission boundary allows.
(A) Not necessarily, and (B) if so, okay so what? The whole point of the parent comment was about keeping devs in check, and I submitted an anecdote that AWS in fact has better tooling to keep people from doing things they aren't supposed to. Not related to billing oversight, but permissions.
It's not that one-sided. On AWS, you have to go hunt to find the pricing for everything. On GCP, it's right next to the instance that you're starting. The GCP dashboard also provides recommendations to down-size VMs if they are too large. On AWS it's also super easy to spin up a VM and never see it because it's in a different region. These little things add up.
I'd implement a good cost attribution strategy before trying a gatekeeping approach. Companies generally have at least semi-functional mechanisms for managing department budgets. Once there's a clear picture how much each service/webapp/product costs to run then they can feed that into the existing budget infra and let things shake out.
Until you know both halves of the ROI calculation it's difficult to focus effort on trimming the right things. e.g. It seems silly for a team to spend $2k/mo on naive/managed solutions for simple things but maybe it's worth it if it helps them avoid hiring another $10k+/mo engineer.
What does a billing issue look like? Is it something trivial, like they charged you for $X+X, but you only used $X (e.g., they double billed you -- should be solvable with a phone call)? Or more complex, e.g, they charged for more egress than you actually used (kind of hard to prove or disprove after the fact)?
Not GP, but in my experience, AWS bills run away from you if you're not careful. They don't have great tooling (or, at least, accessible or intuitive tooling) to determine what your bill is going to be, or to set limits.
Pair that with a misconfiguration because of their horrendous web interface, and you're in for a surprisingly large bill at the end of the month.
Google, on the other hand, has some of the best tooling in the industry when it comes to billing and cost management. I dislike Google as much as the next guy but I'd feel more comfortable with them over AWS if I ever needed to choose.
The last startup I was at repeatedly ended up with $25k AWS bills due to runaway elastic search clusters or dynamodb. The only reason we resolved them was due to us having hired our former AWS account rep.
I got my fair share of those from customers while at GCP, but I agree that in the past several years GCP has gotten much better at billing infra given all the problems we heard of...
Honestly there are tons of examples of runaway bills on both, and neither provides much better of a way to handle visibility of cloud billing than the other. We could discuss the limitations of AWS' billing estimation systems ("only visible when you look!") or GCloud's budgeting system (which has notoriously questionable "limitations" https://www.theregister.com/2020/12/10/google_cloud_over_run...), but neither of the two are particularly better than the other at avoiding surprise billing.
There is this bias effect that is not common only to this part of the thread but this entire thread, and perhaps any discussion of "which cloud is better" where people who are clearly invested in one platform or another show biases that help them to justify their (or their company's) lock-in decisions.
This is not to say that cloud itself is a bad call, but it's crazy how many people out there don't realize how their situation and fear of "making the 'wrong' decision in the past" affects how they discuss the options (or even how they reinvest in a particular option later!), and how they claim "actually that vendor is worse than mine"
I have larger development investments in both AWS and in Google Cloud. They each have pros and cons but runaway billing is a gotcha of minute-by-minute rental billing of compute, storage and network services (the "cloud") and how we use it, and not really something specific to one vendor or another. It's just something that you have to be constantly aware of, constantly monitor, and work to avoid.
It's 100% our mistake(s). It's only AWS fault indirectly, in that AWS is complicated and requires a lot of non transferable knowledge.
As a small example, we currently pay $750 for Route53. We don't know why (it isn't traffic). It has something to do with Route53 resolvers that our "lead sre" setup before leaving. AWS support doesn't understand how it's setup, and since $750 is relatively small, we've just left it.
Is that true? Google Projects correspond to AWS accounts. You can have as many AWS accounts as you like. If I'm not mistaken (very much possible), you can even inherit permissions to AWS accounts comparable to Google's org/folder hierarchy.
Probably would have been better for the author to have taken the "don't sell out to a single cloud vendor API" angle. But I guess no clicky-baits for that approach. I've been screwed over by both AWS and Azure, and never by GCloud, but that doesn't mean I trust any of them.
The best (silliest) business model I've seen is the new thing of vendors promising to make you cross-cloud...via locking you into the platform offered by the vendor (which is usually entirely hosted on one specific cloud provider anyway).
There's valid reasons to support multiple providers, but that is definitely not one of them.
Indeed. The thing that caused me to actually move stuff off AWS to GCP was that you could deploy generic Node Express apps as Firebase Cloud Functions (and then Cloud Run). I knew I could move those anywhere if I needed to.
Honestly, it's kind of brilliant. Convincing people that they need to pay you outrageous sums to not change the fact that they're locked-in at the same level is really clever.
Kinda scummy? Probably. But brilliant nonetheless.
I'd like to hear arguments on this topic that aren't just moronic. If you'd launched your company on Google App Engine 10 years ago, what harm would have come to you in the meantime?
To be honest, since I've heard literally all of these stories/anecdotes, and in some cases have been affected by them, and yet there is still nothing new in this article to rehash, it actually makes me feel good having gone with GCP.
I certainly have felt what I thought were missteps by GCP in the past, but over the past couple years have been an extremely happy customer, and I still feel I've architected my applications so that if worse came to worse I could migrate off GCP if needed.
Look where the cash cows are. For MS and Amazon, their cloud platforms are top of mind for senior management. Even more so with Amazon's new CEO having been with AWS since inception.
For Google, anything not driving search and ads is a side show. Does anyone think Sundar's staying up at night worrying about Asian egress pricing when he's about to spend the next day being accused by performatively outraged senators about censorship and election influcence?
Can HN please add a filter for these increasingly lame "Google cancels all the stuff" posts?
Yes, Google has cancelled services, but they've all been free things that they had every right to decide would never increase revenue. Why should Google have to keep everything they ever built running for ever?
If you pay for services from Google, then it's a completely different story. We've used Appengine for 12 years now, and every time they've decided to deprecate services, there's always plenty of notice, a superior replacement, and usually lower costs.
>We've used Appengine for 12 years now, and every time they've decided to deprecate services, there's always plenty of notice, a superior replacement, and usually lower costs.
That doesn't remove the cost and time of updating your code and migrating.
That should be a decision for the customer to make and not one that is forced on them. Many larger companies tend to have legacy systems that are basically black boxes to them. The cost of updating one can be orders of magnitude more than any cloud costs for running it.
100% this. Deprecating things your customers use always sucks for your customers. No matter what. It doesn't matter how long the notice is. It doesn't matter how good the replacement is. You have made work for your customers that they wouldn't otherwise need to do.
The thing is they change/deprecate/retire paid-for services too (in a brutal contrast with the competiton).
Two quotes from one of the posts referenced in the submission (from Steve Yegge):
...I know I haven’t gone into a lot of specific details about GCP’s deprecations. I can tell you that virtually everything I’ve used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don’t even get me started) to Cloud Endpoints to… I dunno, everything, has forced me to rewrite it all after at most 2–3 years, and they never automate it for you, and often there is no documented migration path at all. It’s just crickets. And every time, I look over at AWS, and I ask myself what the fuck I’m still doing on GCP. ...
... Update 3, Aug 31 2020: A Google engineer in Cloud Marketplace who happens to be an old friend of mine contacted me to find out why C2D didn’t work, and we eventually figured out that it was because I had committed the sin of creating my network a few years ago, and C2D was failing for legacy networks due to a missing subnet parameter in their templates. I guess my advice to prospective GCP users is to make sure you know a lot of people at Google… ...
On the first, I've worked as a PM on Firebase, App Engine, and Endpoints.
Nobody has been forced to migrate from the Firebase RTDB to Firestore (and AFAICT the Firestore API hasn't deprecated anything?), App Engine deprecations (https://cloud.google.com/appengine/docs/deprecations) are basically "you can't do new things using these old things, but the old ones will continue to run" (though other deprecations I've done have provided clear explanations of why we're deprecating and how someone can work around it), and Endpoints is still around despite being comically out of date (it's even getting a managed version!).
Just got kicked off of google music as a paying customer, so nope, got a shitty youtube music replacement.
As a paying google fi customer got transferred to hangouts then that got canceled and I apparently need to change my phone number if I want to make an outgoing voip call again because ??? google.
> Just got kicked off of google music as a paying customer, so nope, got a shitty youtube music replacement.
YouTube Music isn't available in my area. Got kicked off Google Play Music with a "download your content, we're deleting it" and couldn't pay even if I wanted to.
Announced discontinuation in August, full deletion of my Music Library in February.
> On 24 February 2021, we will delete all of your Google Play Music data. This includes your music library, with any uploads, purchases and anything you've added from Google Play Music. After this date, there will be no way to recover it.
Same here. I ended up on Spotify instead, which is still shitty although less than I remember (no music player should ever display the notice "can't play the current song", that is its core feature).
However I don't think the two can be compared: we don't need to launch a giant project to make serious infrastructure changes to switch provider.
Just responding to the OP "Yes, Google has cancelled services, but they've all been free things that they had every right to decide would never increase revenue. Why should Google have to keep everything they ever built running for ever?"
Which to be clear they are not responsible for running all their services forever, but it puts a lie to the idea that google is fine with stability in a service, they'll take something commercially successful enough and sacrifice it for something else in hope it attains "hypergrowth".
At the end of the day google doesnt give a flip about its customers, the structure of google will always incentive new services over fixing/maintaining existing stuff and anyone who has a pollyannish view of this needs to wake up.
>>but they've all been free things that they had every right to decide would never increase revenue
If my service provider were to hold this opinion, I would not be able to trust them, actually I would start to search for an alternative immediately. It sounds like a service provider who is okay with turning customers into lab-rats, to experiment on, and once done just discard them away.
> every time they've decided to deprecate services, there's always plenty of notice
Ah, but with AWS, if something is deprecated, generally they tell you you should use something else, but the old way will continue to work indefinitely. You can switch over on your own timeframe.
>If you pay for services from Google, then it's a completely different story. We've used Appengine for 12 years now, and every time they've decided to deprecate services, there's always plenty of notice, a superior replacement, and usually lower costs.
Really? I've had the complete opposite experience on AppEngine as a paying customer.
I was using Python2 AppEngine with ndb and the Users API. Cloud Datastore + ndb automagically cached your data and worked pretty nicely. When they moved to Firestore, they dropped that feature and recommended you buy your own Redis DB and manage caching yourself. They got rid of the Users API entirely and forced apps onto OAuth, which is much more complicated to integrate.
They old AppEngine emulator worked really nicely as well, in that you could emulate a pretty full AppEngine environment locally. When they moved from Python 2 to 3, they dropped most of the emulator's features. True, AppEngine apps require less AppEngine-specific code, so there's less need for an emulator, but it's still useful for testing certain scenarios. I checked recently and it seemed like they had improved their emulator, but I believe there was about a year where there was no Admin UI for their emulators like there had been for AppEngine Python2.
It's all caused me to move away from AppEngine and rely more on vendor-agnostic stacks.
This sounds like a mis-characterization of the situation. It is not Google who have moved from Python 2 to 3, it's you. Google still offers and supports the legacy Python 2.7 runtime for App Engine, and will continue to do so indefinitely. The same is true about ndb and Firestore. It may be that you moved from NDB to Cloud NDB (Firestore), but nobody forced you to do it.
But I think it's not such a "free choice" when Google announces a service is deprecated given that they're notorious for shutting off their deprecated services. Once Google announces a deprecation, I think it's fair to assume an EOL could come at any time, and I don't want to be caught on the back foot when that happens with not enough time to migrate.
I see that Google now has clear messaging that they will support Python 2.7 AppEngine indefinitely,[0] but I don't recall seeing that messaging in 2019. Internet Archive only has snapshots of that page[1] going back to April 2020, which makes me think they hadn't made it clear until then that this was their policy.
In 2019, I just remember seeing scary warnings everywhere in AppEngine docs of "we strongly recommend you get off of Python 2.7." I talked to Google DevRel folks at PyGotham 2019 and asked them what was going to happen to Python 2.7 AppEngine. They said it was going away but they hadn't picked an EOL date yet.
Woof, sorry. I worked on the Users API deprecation a while ago (2018-2019, prior to any announcement), and there were a few features that just couldn't be migrated reasonably to OAuth (e.g. the `admin` functionality). We did consider things like moving to Cloud IAM (e.g. what we did for GCF and Run) as well as Firebase Auth, but couldn't replicate everything :/
Hi-5 fellow "got caught across the boundary" traveller. This happened to me when I moved jobs to be devops lead on a new product for a company - the google recommended contractor implemented regular AppEngine with ndb, and about 6 months later we were staring down the barrel of everything being deprecated (and had to do the same: add a redis instance to the stack) and then hope that the new AppEngine was going to be ready when we actually went live.
It eventually came together but we ended up having to do a whole lot of refactoring while we were on a tight launch schedule.
What about all the data they gobble up for free from its users during the time its active?
Google has never been free, its bread and butter is the data we give it. We were it's product.
I thought this was going to be written by someone who uses it a large scale.
from a customer service point of view, is Microsoft the best ?
Is Google the most secure?
Is Amazon the simplest to enter?
GCP doesn't really support IPv6 at all. Sure you can terminate IPv6 on a load balancer and then proxy the connections over IPv4 back to your instances, but you can't get native IPv6 on an instance.
It's that type of shortcoming that leads me to believe Google does not see a future in this product.
The network virtualization scheme and likely many other parts of the stack need to change in order to support v6. It's not that there aren't investments in this area, it's just a non-trivial effort.
Were they surprised that IPv6 existed? It's not as though this is a new technology, or that the necessity hasn't been obvious since around the time Google was born.
This week it’s ICE and Exxon-Mobil. Next week, who knows. It’s not worth the risk when Google has a demonstrated history of catering to its employees demands to drop customers.
> The clock is ticking for Google Cloud. The Google unit, which sells computing services to big companies, is under pressure from top management to pass Amazon or Microsoft-currently first and second, respectively, in cloud market share-or risk losing funding. While the company has invested heavily in the business since last year, Google wants its cloud group to outrank those of one or both of its two main rivals by 2023, said people with knowledge of the matter.
If they pulled this off, they would be hailed as gods of marketing for eons to come.
Tactically, what would this look like? Getting large numbers of AWS or Azure customers to move over? Companies have higher priorities than changing their cloud provider, they want to focus on growing their business.
Indeed - in fact Microsoft recently tried to buy Pinterest just to get a large company that they can move from AWS to Azure:
> The deal, which would have been Microsoft’s largest acquisition to date, confirms that the tech giant is continuing to pursue an acquisition strategy aimed at amassing a portfolio of active online communities that could run on top of its Azure cloud computing platform. Pinterest - which boasts more than 320 million active users - currently relies on Amazon Web Services (AWS) as its infrastructure provider.
If they could onboard themselves on Google Cloud they'd probably be the biggest cloud. Seems like that's what they should be trying to do, it'd also future-proof them if they have to split up the business
This is very real. After a production outage caused by excessive health check failures the day after a massive GCP outage (Sept 2020) -- where we quickly hit our already-oversized quota (quotas, another GCP issue) during a traffic spike -- we've moved all of our sensitive workloads to AWS.
We continue to use GCP for less sensitive workloads and for GKE, but our entire ops team has unspoken distrust. This is totally an infra-specific opinion, ignoring the fact that we've had to rewrite apps entirely after breaking changes from Google products.
GCP has a great UI, the project structure makes much more sense, and billing is way easier, but after having a massive outage during a pretty standard scaling event, we just can't justify the risks.
250 comments
[ 3.6 ms ] story [ 255 ms ] threadedit: It's also not a new account, very consistently paying for some services (maps, etc.) for years before I decided to ramp up.
edit2: It was also a tiny quota increase from the default one so not like we suddenly asked for five hundred instances.
Contrast:
Microsoft - flat out refused me more quota despite spending 10k/mo with them. Required me to convert to invoice billing, and then wanted a bunch of proof of incorporation and when my trading name didn't match my registration name they were unable to proceed.
Oracle - took 3 months of escalations and deliberations, required me to explain on the phone to a VP why I needed the quota.
AWS - frequently requiring me to write up a spiel about what I'm going to do with the quota before they approve it, increasing the RTT to 72hrs+ - do they actually verify this? How would they? Why do they care so much? We've spent 50k+ and always paid the bills, what's the issue?
My quota requests weren't outrageous - never more than 20 VM's, albeit very large VM's.
If you paperwork is not up to spec, they are running the risk of credit exposure when you the customer doesn't' pay. Higher risks for any provider when there is no legal entity to sue etc, which is why they want you to convert to invoice billing.
- either they're automatically approved because you fill in a form requesting more and it just becomes a PR for an engineer to approve, OR
- it can't move because you've hit an internal service limit
- google form submission
- creates a PR with the project ID and the requested value as an exception in a file (this requires OWNERS approval, so at a min one eng/pm to approve)
- file would update a DB the next time it's picked up
If you don't mind me asking, what quota increase was rejected?
"We are setting up more customer environments and are running out of X"
It was usually increased within the hour.
Admittedly, the original author's title of "Why I distrust Google Cloud more than AWS or Azure" much better describes their position than the editorialized title of the HN submitter ("Why Google Cloud is less trustworthy than AWS or Azure").
1: https://news.ycombinator.com/item?id=23929044
Agreed. What is with the retitling?
common in other cloud environments, they just give you a longer runway / head's up on it
2) price increases
true
3) shitty customer service
same as azure and aws -- google, though, have a crap reputation to begin with. With the other two, it was a surprise.
4) continued existance of google cloud
article is over a year old; and - frankly - aws is likely to spin-off from amazon sooner than google cloud being shutdown
no sources, that is just my gut feel.
The market is easily large enough to have 4 - 6 global players *AND* 2 - 3 regional / country-specific ones as well.
What i hate though is more about the company , is what they project they are vs what they are
https://theintercept.com/2018/05/31/google-leaked-emails-dro...
``` I don’t know what would happen if the media starts picking up a theme that Google is secretly building AI weapons or AI technologies to enable weapons for the Defense industry,” she continued. “Google Cloud has been building our theme on Democratizing AI in 2017, and Diane and I have been talking about Humanistic AI for enterprise. I’d be super careful to protect these very positive images. ```
Not being judgemental about if defense is right or wrong , but its fair to say we are in the running for it. What is not okay is "we care for humanity and freedom of speech" and then do backchannel discussions on how dissent can be quelled with government agencies.
So to me distrust is for the company possibly and probably not so much on the reliability of its services
Apple has historically been anti consumer and anti developer with a huge marketing budget to wash it. Facebook intentionally makes us sad. M$ and their anti competitive practices should be well known.
What? Apple has been pro-consumer to the detriment of everyone else. Developers are still screaming that they aren't allowed to install malware on my iphone.
And bend to china.
It isn't inconsistent to push back on government overreach where it is legal to do so, but conform where they have no other legal choice except dropping an entire market.
Or do you propose another solution?
Even for non-vague comments, it's often much better for one person to post links than to make 50 people repeat the same search.
https://killedbygoogle.com/
Honestly the list hilariously long!
GCP does deprecate products (former App Engine PM here, deprecated many APIs), but it's definitely less frequent than the "Google deprecates everything" memers want you to believe, and there's a minimum 12 month deprecation period before literally anything is deprecated.
Anecdotally, Google announced the changeover of Cloud logging API versions in October of 2016, with a 5-month ramp (October to March) to switch from the v1 beta API to the v2 beta API. Five months is nearly two quarters, which is quite a long window for a beta API IMHO.
That having been said, Google's habit of leaving things that are pretty much mission-critical in beta is unwise, but it should be unwise for them, not end-users. End-users that need reliability and low churn shouldn't be developing on beta-anything.
Giant changes like that are worth capturing in long term planning processes, and then you need time to get ramped up on the new stack, design and implement the replacements, run all your backfills, and also, still have a couple years to figure out the parts that don't migrate nicely. With enough time available, you also don't have to drop actual business related improvements, even if your progress slows down a bit
I would sum up his ideas as: "If you offer a service I pay for, and deploy code on, you should not break my code while you still offer the product."
I also don't think he'd have bothered to mention one or two minor things. His examples were rampant and incredibly bad, like breaking their own offerings.
The thing here is that if you run hundreds of services in production - many of which work smoothly and you don't need to touch often, you will find that Google's habit of forcing you to change how to use their tooling will generate a huge burden...
They discourage you from using it and make it clear that for every use case some other tool at AWS would be best, and have been doing so for several years now... They wont even list it anymore under https://aws.amazon.com/products/databases/ ..
Still, they support it ( https://aws.amazon.com/simpledb/ ) because there are customers with legacy systems that depend upon this service.
As for the beta thing: GCP's definition of beta was basically everyone else's definition of GA, since the GA requirements were so insane (e.g. 99.999% internal availability) that getting there would take literal years (see the GCF beta to GA taking like 18 months?). I totally agree that it's weird that things would stay in beta for so long, as opposed to hitting industry standard levels so users can have confidence in them, but setting GA as far higher than the industry was part of Google Cloud's plan.
Among other edge cases that caused pain.
With AWS at least I can use a prepaid card: if something bad happens for any reason, at least I know I can afford to eat the day after.
If anything, this is just AWS being overly generous and forgiving.
And IMO, if you're a real customer, all the providers are fairly forgiving, provided you can get in touch with a real human who works there.
Otoh I find AWS quite straightforward, but I've been using it for several years now. Their support is worth their weight in gold.
But I have reasonable exposure to both AWS and GC and I can say that, by far, Google Cloud is easier to reason about. As a consequence, it's much harder to misconfigure. The 2 large AWS deploys I've seen have, at best, had billing issues no one really understood (incl AWS), and at worse, security issues.
Complaining that maps prices went up re Cloud Hosting is, to me, like complaining that Amazon raised the price of the Kindle, e.g., not particularly relevant.
I haven't used Kubernetes on either platform - so there may be more to that.
One thing I really dislike about GCP is how expensive it is for personal or hobby use. I burned through $300 for a simple vm on GCP in a few weeks because their cheapest instances are so expensive.
I will say - reserved instance pricing is great on AWS if you know you will be there for a while.
How'd you manage that? My bill is always super lower every month.
The issue is definitely not AWS. It’s always the developers. You really need a gate keeper to AWS to question why you need a service and ask for a price estimate on cost and usage.
This requires money and time that often only large corporations have the luxury of.
At my employer, we have a gatekeeper team who is terribly overworked and hardpressed to push back too much when business outcomes are at stake. One of the more successful things theyve done is create a terraform repo anyone can contribute to. They will review PRs and manually apply changes for production accounts. Whats great is that these folks can take my PRs that are 80% right and they are able to help me achieve least privilege better than I could on my own. However, other devs really dont care about least privilege and they tend to go for large open policies.
AWS's IAM policy is far and away the most sophiscated and granular, and even has a nice UI now. Trying to achieve this in Azure is next to impossible because you must have extremely high permissions to even be able to make new roles/policies that are super granular.
IAM team creates a "developer admin" role/user that can only create users/roles that have a permissions boundary on it. That way, no matter what policy the dev admin grants, the dev user can only do what the permission boundary allows.
I've learned much more with 5 minutes posts (which I truly dislike) in other sites than going to AWS docs.
Comparing to Digital Ocean docs and Q&A which is so much easier
Edit reason: Added DO as example of simplicity and really helpful price model.
Until you know both halves of the ROI calculation it's difficult to focus effort on trimming the right things. e.g. It seems silly for a team to spend $2k/mo on naive/managed solutions for simple things but maybe it's worth it if it helps them avoid hiring another $10k+/mo engineer.
Pair that with a misconfiguration because of their horrendous web interface, and you're in for a surprisingly large bill at the end of the month.
Google, on the other hand, has some of the best tooling in the industry when it comes to billing and cost management. I dislike Google as much as the next guy but I'd feel more comfortable with them over AWS if I ever needed to choose.
I got my fair share of those from customers while at GCP, but I agree that in the past several years GCP has gotten much better at billing infra given all the problems we heard of...
There is this bias effect that is not common only to this part of the thread but this entire thread, and perhaps any discussion of "which cloud is better" where people who are clearly invested in one platform or another show biases that help them to justify their (or their company's) lock-in decisions.
This is not to say that cloud itself is a bad call, but it's crazy how many people out there don't realize how their situation and fear of "making the 'wrong' decision in the past" affects how they discuss the options (or even how they reinvest in a particular option later!), and how they claim "actually that vendor is worse than mine"
I have larger development investments in both AWS and in Google Cloud. They each have pros and cons but runaway billing is a gotcha of minute-by-minute rental billing of compute, storage and network services (the "cloud") and how we use it, and not really something specific to one vendor or another. It's just something that you have to be constantly aware of, constantly monitor, and work to avoid.
As a small example, we currently pay $750 for Route53. We don't know why (it isn't traffic). It has something to do with Route53 resolvers that our "lead sre" setup before leaving. AWS support doesn't understand how it's setup, and since $750 is relatively small, we've just left it.
There's valid reasons to support multiple providers, but that is definitely not one of them.
Kinda scummy? Probably. But brilliant nonetheless.
I certainly have felt what I thought were missteps by GCP in the past, but over the past couple years have been an extremely happy customer, and I still feel I've architected my applications so that if worse came to worse I could migrate off GCP if needed.
For Google, anything not driving search and ads is a side show. Does anyone think Sundar's staying up at night worrying about Asian egress pricing when he's about to spend the next day being accused by performatively outraged senators about censorship and election influcence?
Yes, Google has cancelled services, but they've all been free things that they had every right to decide would never increase revenue. Why should Google have to keep everything they ever built running for ever?
If you pay for services from Google, then it's a completely different story. We've used Appengine for 12 years now, and every time they've decided to deprecate services, there's always plenty of notice, a superior replacement, and usually lower costs.
That doesn't remove the cost and time of updating your code and migrating.
Two quotes from one of the posts referenced in the submission (from Steve Yegge):
...I know I haven’t gone into a lot of specific details about GCP’s deprecations. I can tell you that virtually everything I’ve used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don’t even get me started) to Cloud Endpoints to… I dunno, everything, has forced me to rewrite it all after at most 2–3 years, and they never automate it for you, and often there is no documented migration path at all. It’s just crickets. And every time, I look over at AWS, and I ask myself what the fuck I’m still doing on GCP. ...
... Update 3, Aug 31 2020: A Google engineer in Cloud Marketplace who happens to be an old friend of mine contacted me to find out why C2D didn’t work, and we eventually figured out that it was because I had committed the sin of creating my network a few years ago, and C2D was failing for legacy networks due to a missing subnet parameter in their templates. I guess my advice to prospective GCP users is to make sure you know a lot of people at Google… ...
Nobody has been forced to migrate from the Firebase RTDB to Firestore (and AFAICT the Firestore API hasn't deprecated anything?), App Engine deprecations (https://cloud.google.com/appengine/docs/deprecations) are basically "you can't do new things using these old things, but the old ones will continue to run" (though other deprecations I've done have provided clear explanations of why we're deprecating and how someone can work around it), and Endpoints is still around despite being comically out of date (it's even getting a managed version!).
As a paying google fi customer got transferred to hangouts then that got canceled and I apparently need to change my phone number if I want to make an outgoing voip call again because ??? google.
YouTube Music isn't available in my area. Got kicked off Google Play Music with a "download your content, we're deleting it" and couldn't pay even if I wanted to.
Announced discontinuation in August, full deletion of my Music Library in February.
> On 24 February 2021, we will delete all of your Google Play Music data. This includes your music library, with any uploads, purchases and anything you've added from Google Play Music. After this date, there will be no way to recover it.
However I don't think the two can be compared: we don't need to launch a giant project to make serious infrastructure changes to switch provider.
Which to be clear they are not responsible for running all their services forever, but it puts a lie to the idea that google is fine with stability in a service, they'll take something commercially successful enough and sacrifice it for something else in hope it attains "hypergrowth".
At the end of the day google doesnt give a flip about its customers, the structure of google will always incentive new services over fixing/maintaining existing stuff and anyone who has a pollyannish view of this needs to wake up.
If my service provider were to hold this opinion, I would not be able to trust them, actually I would start to search for an alternative immediately. It sounds like a service provider who is okay with turning customers into lab-rats, to experiment on, and once done just discard them away.
Ah, but with AWS, if something is deprecated, generally they tell you you should use something else, but the old way will continue to work indefinitely. You can switch over on your own timeframe.
Really? I've had the complete opposite experience on AppEngine as a paying customer.
I was using Python2 AppEngine with ndb and the Users API. Cloud Datastore + ndb automagically cached your data and worked pretty nicely. When they moved to Firestore, they dropped that feature and recommended you buy your own Redis DB and manage caching yourself. They got rid of the Users API entirely and forced apps onto OAuth, which is much more complicated to integrate.
They old AppEngine emulator worked really nicely as well, in that you could emulate a pretty full AppEngine environment locally. When they moved from Python 2 to 3, they dropped most of the emulator's features. True, AppEngine apps require less AppEngine-specific code, so there's less need for an emulator, but it's still useful for testing certain scenarios. I checked recently and it seemed like they had improved their emulator, but I believe there was about a year where there was no Admin UI for their emulators like there had been for AppEngine Python2.
It's all caused me to move away from AppEngine and rely more on vendor-agnostic stacks.
But I think it's not such a "free choice" when Google announces a service is deprecated given that they're notorious for shutting off their deprecated services. Once Google announces a deprecation, I think it's fair to assume an EOL could come at any time, and I don't want to be caught on the back foot when that happens with not enough time to migrate.
I see that Google now has clear messaging that they will support Python 2.7 AppEngine indefinitely,[0] but I don't recall seeing that messaging in 2019. Internet Archive only has snapshots of that page[1] going back to April 2020, which makes me think they hadn't made it clear until then that this was their policy.
In 2019, I just remember seeing scary warnings everywhere in AppEngine docs of "we strongly recommend you get off of Python 2.7." I talked to Google DevRel folks at PyGotham 2019 and asked them what was going to happen to Python 2.7 AppEngine. They said it was going away but they hadn't picked an EOL date yet.
[0] https://cloud.google.com/python/docs/python2-sunset
[1] https://web.archive.org/web/*/https://cloud.google.com/pytho...
It eventually came together but we ended up having to do a whole lot of refactoring while we were on a tight launch schedule.
It's that type of shortcoming that leads me to believe Google does not see a future in this product.
Source: Worked in GCP networking.
What if some Google employees decide my company is bad? Will Google cave in and fire us as a customer? That’s an existential risk.
Fossil fuels are a maybe.
This really shouldn’t be a problem for 99% of customers though.
If they pulled this off, they would be hailed as gods of marketing for eons to come.
> The deal, which would have been Microsoft’s largest acquisition to date, confirms that the tech giant is continuing to pursue an acquisition strategy aimed at amassing a portfolio of active online communities that could run on top of its Azure cloud computing platform. Pinterest - which boasts more than 320 million active users - currently relies on Amazon Web Services (AWS) as its infrastructure provider.
source: https://www.forbes.com/sites/carlypage/2021/02/11/microsoft-...
We continue to use GCP for less sensitive workloads and for GKE, but our entire ops team has unspoken distrust. This is totally an infra-specific opinion, ignoring the fact that we've had to rewrite apps entirely after breaking changes from Google products.
GCP has a great UI, the project structure makes much more sense, and billing is way easier, but after having a massive outage during a pretty standard scaling event, we just can't justify the risks.