This is why, as a systems designer, I refuse to use "wrapper" services proved by cloud providers. If it doesn't have a shell where individual services can be run using my own choice of stack, I don't like it - for this exact reason.
how do y'all look at systems like AWS Aurora, GCP Spanner, etc which offer big speedups and cost savings, but lock you into the platform in practice?
this isn't academic for me - I've got a service that's on vanilla RDS PostgreSQL but growing fast, and I'll soon need to decide whether to move to something cheaper/faster but proprietary or blow my budget trying to stay generic...
I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic?
Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution.
So does it come down to the proprietary option or fail?
This would only come from someone who has never done or managed a migration at scale.
There are so many moving pieces doing any migration at scale, your code is the least of the problems.
I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that.
Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
There is so much involved with any migration at scale - data migration, regression testing, network changes even where a company many have a physical connection to their cloud provider, security, compliance , training, and the list goes on.
You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
Because in the grand scheme of things “your code” and even changing your code is just a minuscule part of what it takes if a migration did ever happen.
So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
No — changing your code makes every one of those aspects harder, more time consuming, and thus a worse option. (Again, except network topology.)
And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable:
Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc.
- - - -
Edit to include a specific example:
Let’s imagine I’m migrating a dataset, A. And my copy is A’.
In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface.
So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests.
In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’.
But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing.
In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
Again, you’re looking at your little piece of work that is a speck in an entire organization. Your CTO is working with your CIO, compliance department, finance department, product owners, the PMO organization, etc.
They are talking to the board about strategic decisions and what will give them a competitive advantage. He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
A dependency always slips in somewhere unless you are constantly testing and preparing for portability like Uber does.
Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
Let alone a dependency on all of the arns always having “aws” in them not thinking they wouldn’t work in China or gov-cloud. I hardcoded the partition.
> Again, you’re looking at your little piece of work that is a speck in an entire organization.
> He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
Of course not — he’s paying experts to avoid that situation, aligning with his strategic priorities.
That’s my job.
You also resorted back to personal attacks because you can’t make your point directly.
> If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
Like this ridiculous comment to someone whose job is being hired by CTOs looking to migrate.
They literally come to me, for help doing exactly that.
>
Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
> I hardcoded the partition.
> “Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”
Yes — that’s why companies pay me: I’m used to doing things that can’t depend on a region, service, etc while AWS engineering frequently makes that mistake (eg, all the US-East-1 magic).
That’s precisely my point:
Quality coding avoids and bounds mistakes like that, which slow your migration.
- - - -
So you’ve resorted to personal attacks, but didn’t actually respond to my specific example, and were insistent on my credentials but still haven’t shared the largest system you personally have migrated.
But please keep it up: I make my money from people like you causing CTOs to hire people like me to clean up the mess — “whoops, accidentally wrote regionalized code!” or “we need over a year to migrate 100 VMs due to self-induced complexity!”
Scenario: I have $aLotOfData in S3 and I need to move it to a MySql database.
Are you going to:
A) write an ETL job that is going to be slower and more complex to avoid the “lock-inz”
Or
B) just call run
“load data from S3…”
As a sql extension?
Are you going to tell them that they can’t use Glue? Athena? Redshift? Quicksite?
When your company needs a call center are you going to tell them not to use Amazon Connect? Now they need to integrate the call center with Salesforce - are you going to tell them they can’t do that because they have to use Lambda?
They want to have speech to text - now they need Amazon Lex.
Then they want a reporting dashboard that comes from their contact trace records. Now they need Kinesis which is going to stream to S3.
The typical company uses over 250 SaaS products that they have some form of integration with. Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
If you work in the health care industry are you going to tell them not to use Epic?
That question depends on their priorities as a company: how do they value developer time vs lock-in vs complexity?
Which contrary to your repeated personal attacks is the sort of conversation I have with corporate leaders: they determine the strategy; I make it work technically.
But yes — there are clients who insist on avoiding those services (and generally would, if they’re migrating out of S3).
And in this particular context, we were explicitly discussing the decision of how to migrate a product into the cloud where I advocated using the cloud solution but taking a small step which would both ease that transition and any future ones — and you flipped out.
- - - -
To address your dramatic nonsense:
You don’t have to use Lambda to integrate with Salesforce; and you know that non-AWS call centers are used by most AWS customers.
> Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
This is just ridiculous dramatics completely unrelated to anything I said.
- - - -
Let’s ask a few questions to refocus:
- why were you so insistent on my credentials, but won’t share yours? …are you embarrassed by your largest migration?
- why can’t you address my specific example? …because it shows I have good advice and you spouted nonsense?
- why are you melting down into unrelated dramatics? …or pretending I advocated against using cloud solutions? …or integrations at all?
I think you’re just throwing up arguments at random because you can’t admit that on the original issue I was right.
You absolutely do have to use Lambda to integrate your call center with Salesforce. Your call center agents use their familiar Salesforce screen when a customer calls in.
But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
None of this is “making the beer taste better”.
I’ve shared by credentials up thread - 2 years heading a migration and integration when a private equity owned health care company was acquiring companies to get big and go public, 3 years leading the application modernization effort for another company that was acquired at 10x revenue and three years working at AWS in the Professional Services department and now building out a professional services practice for an MSP for applications modernization.
And before that over 20 years as a professional developer
> But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
This is a fantasy you invented about me because you cannot respond to my actual point — a strawman.
My original advice was to use the cloud — and I was addressing someone else’s concern about lock-in, while giving advice that would aid their initial migration to using cloud services.
I think that it’s very telling you can’t be honest about what I said.
> It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
As is this — another strawman you invented.
- - - -
Why do you have two strawmen, but haven’t in several posts actually addressed my specific example?
Are you unable to?
- - - -
What you’ve said is that you spent two years migrating just a hundred VMs, which sounds like a migration delayed by self-induced complexity.
I appreciate arrogantly incompetent developers like yourself — who can’t address technical points and instead resort to fallacious thinking:
You did see the part about integrating Salesforce with a Connect call center didn’t you?
So I’m an “incompetent developer” who someone managed to get hired at AWS Professional Services not a third party partner - you to consult the largest organizations on the planet?
I’m sure an AbstractFactoryRepositoryFacade is going to prevent cloud lock-in while you advise people to create a complicated workflow when a simple SQL statement using an AWS extension would be faster to develop, more performant and less maintenance?
It’s not a straw man it goes to the core of the efficiency gains you get by taking advantage of your vendors functionality and integrating into it tightly instead of trying to maintain a leaky abstraction.
And “your code” is probably a tiny part of the entire enterprises architecture. The average enterprise has 250+ SaaS dependencies. Do you really think the CTO of a large organization is staying up late at night because a developer chose to use DynamoDB and didn’t put a facade over it to “avoid lock-in”?
If they're getting screwed in pricing or availability vs a competitor, yes. They're asking "how's long do we have to stay with this bad option before we can switch?"
I agree with you, but I think the idea is that the underlying storage engine affords you functionality that you can't mimic yourself / elsewhere and therefore your capabilities to leave are limited.
You're always "locked in" to some degree and it's an almost worthless thing to invest into if you never actually need to actually migrate off.
The only thing that has changed is more stuff has been deprecated. I got an email from Google only last week that something in Analytics is changing (don't care - I stopped using it ages ago when it broke because of some earlier deprecation).
Just a rant about having to react to things that the author didn't want to react to. But these are facts of life. I find GCP policies pretty reasonable. I was never caught off guard by them.
> I find GCP policies pretty reasonable. I was never caught off guard by them.
For some individuals, sure. But other people and especially businesses don't really want change for change's sake (and if possible want to have a more-or-less set-and-forget thing), and the article clearly explains why it is better with AWS if that's the goal.
I think this is a good point. Maybe GCP is failing at communicating why things are changing.
As a customer, I'd be fine with GCP doing things to keep their costs down so GCP pricing stays the same or decreases. But I understand this is hard to quantify outside GCP and requires a bit of faith (which I accept as inherent, since full transparency is impossible).
I am guessing that you either aren't using it for business, or if you are, you are very small.
The author's take is brutal, but totally on-point. The cloud business is about asking businesses to use your system as a foundation. Google doesn't take that seriously enough.
When you have 30 engineering teams all using different parts of the platform and your Product team is crushing your whole org with their aggressive roadmap, you will no longer feel like GCP policies are reasonable.
When everything takes 20% longer to ship than you planned because every other week some random team(s) is forced to do unplanned, immediate rewrites by a vendor, that vendor starts to look like a dick and also a huge liability to your business.
Also Google's own documentation is full of grenades. Just yesterday I found in their GKE docs something they tell you is the default and you must use it...except it doesn't apply to clusters created a few versions ago, even if they're updated, because they didn't roll their change out to existing clusters -- just new ones created after their change.
Also google's docs reflect their deprecation policy -- there is no document versioning. You can always and only see latest.
That backwards compatibility strategy to update defaults on new versions while leaving existing the same is to avoid the scenario you're complaining about by not changing behavior for existing customers.
But then they don't have versioned documentation, so you only know how to properly operate the newest version of things and for the existing stuff you have to figure it out on your own.
> When everything takes 20% longer to ship than you planned because every other week some random team(s) is forced to do unplanned, immediate rewrites by a vendor
This vendor is certainly not GCP in this case.
Every other week? Immediate rewrites? Is this even based on reality?
GCP is the best of Google's offerings and they still manage to do this. If you're using Google APIs like Places, AdWords, Geocoder, etc, they frequently make breaking changes with no announcement.
I've had many, many days in my career of having to drop everything to patch mission-critical glue code to keep up with Google APIs.
Also keep in mind that the documentation for their SDKs is frequently outdated and incorrect, especially around authentication and to the point that I often just deferred to reading the source code over reading their docs. And that's even if their SDK was in a working state, because often their own SDKs would lag behind their own API changes and you'd have to try out ruby, python or php to find one that worked properly (although golang always works, because that's what Googlers actually give a shit about).
Maybe its gotten better in the last few years, but that was solidly my experience from 2010-2018 and those were a lot closer to "Peak Google" than now. Now I write less code against their APIs, but I also have about 10M/yr in GCP infrastructure that I manage and it's also a treadmill even though we stick to the basics (VMs, GCS, Cloud SQL, GKE).
This story resonates with me so much. An avid cloud platforms user in the past, now I just use Dockerfile + fly.io combo. Dockerfile gives you a total freedom without vendor lock-ins and forceful deprecations. I have never been calmer and happier before that switch. Now, only I decide when to upgrade - and usually it is as easy as editing a text file pointing to a new package version.
IMHO, the only thing cloud providers are genuinely useful for is a bottomless data storage. But even that aspect is now isolated by Rclone in all my apps since migrating off the cloud. Apps just talk with the file system, which is then abstracted to a cloud storage of choice by Rclone.
And now you’re going to spend more money on cloud just to treat it like an overpriced collocation center without any of the benefits of moving to it.
Have you ever done a large migration at scale? I have - even moving over a lot of VM hosted services involves your project management organization, regression testing, networking, firewall, training, security, working with outside vendors, and the list goes on.
The average enterprise uses 254 SaaS services. I have never in 5 years heard any CTO of any company of any size worry about “lock in” when considering their cloud choices.
Source: 2 years as an architect responsible for integrating acquired companies before a private equity owned company went public, two years leading the “application modernization” effort at a startup that exited 10x revenue, 3 years working at AWS in the Professional Services department and now working at a smaller consulting shop.
And I got Amazoned a couple of months ago. I have no particular loyalty toward Amazon
Maybe they don't worry about it because they don't think there's anything to worry about it. The most expensive part of switching cloud providers is data transfer.
That's where the lock-in is, Kubernetes vs VMs isn't that big a deal at all; annoying sure. For the record, my employer uses both Kubernetes and VM. The deployment is pretty much the same.
Totally. Migrating anything is a PITA, even if it's simple. If it's complex and involves tons of code changes to eliminate "lock in", or more likely, move from one flavor of lock-in to another, you better think seriously whether it's even worth bothering.
Developer time is very, very expensive, and any time you spend "migrating" is time not spent building features that add value. Except in rare cases, the end customer probably doesn't care if your stuff runs on AWS, GCP, or Azure.
I guess the point is then: once you’re migrated over, you live happily. For a while. The next migration will come. It’s only a question of when. What then? You’re now so far removed from simple Unix-like principles, tools and platforms that migration will be harder by an order of magnitude (and you noted how much work it is!).
And what about the 250 other SaaS services that your company uses?
Are you going to suggest that your medical software doesn’t integrate with third party EMH/EMR systems? That your call center and sales team doesn’t integrate with Salesforce?
What about your dependency on Workday? ServiceNow? Your project management software? All of the Microsoft software?
Yes even VMWare. There is a reason that AWS has a VMWare offering.
> You’re now so far removed from simple Unix-like principles, tools and platforms that migration will be harder by an order of magnitude
Exactly my experience. Some services I manage are still entrenched in Big Co. Cloud, and it's a hell. This lock-in costs us additional money (thousands per year), but that amount is less than it would cost us to migrate those apps immediately.
Unix is cool not because it's a Unix, but because it's simple, aka Unix Philosophy.
It does depend on the managed services you're using. Most SaaS companies do offer the "conventional" services managed; if all you need is a database, a cache and some sort of place to dump a docker container for your backend/frontend and not deal with the wires of manually setting up, connecting and securing all of it against the outside world, their offer makes sense. (And like... this describes the majority of internet tech stacks).
The hairy stuff is when you get into the highly specific offerings for each SaaS platform. They all offer a ton of them and they can certainly be attractive for some cases (ie. Filehosting) or useless in others (there's a lot of stuff that amounts to "our version of cache/database" and little else) but that's where the lock-in issues tend to play up.
GCP is uniquely terrible though from what I've heard in terms of support lifespan and utter unhelpfulness in terms of communicating to their customers when some SaaS offering of theirs are on the chopping block.
Have you seen or been involved with overseeing the architecture of a large enterprise or state government? There is such a morass of built up complexity it takes years to complete a migration and many just give up and install a direct connect to their cloud provider and claim they always meant to be hybrid.
I've seen the (broad) architecture of my local government yes (due to keeping tabs on government spending on IT as a personal curiosity). I'm aware of the morass you speak off.
The reality though is that most tech stacks aren't that complicated. Not every company is a venture capital "go big or go bust, we change the entire industry" darling (nor are they necessarily looking to grow that way, especially outside the SV bubble) or a government with 2000 different contracts and enough yellow tape + bureaucracy around those contracts that signing a new contract is just easier than possibly extending an existing one with new features.
For the rest of those tech stacks (which are definitely "enterprise", simply by definition, since they get used in companies and are bespoke enough to be their own tools), it's rare to see a stack more complex than the one I mentioned (backend, cache, db) unless a technical admin decided to play runaway with Azure/Google/Amazon because it's shiny and exciting.
Yes, in those cases the tradeoffs of lock-in are worth it. With that in mind though - F500 are the absolute peak of the economic market (since you know, that's how Fortune makes that list). They represent less than 1% of the total market in terms of actual needs.
In those cases, the vendor lock-in from using a bespoke AWS or Azure solution are absolutely worth it because if AWS or Azure are an active problem they also will have the manpower to migrate off of AWS/Azure. My suggestion/advice is for the other 99%. You still want AWS/Azure for CYA, the managed part (not having to deal with ops) and backups but you don't need all the extra solutions they offer.
It's why "you are not Google" is an advice that exists. Newbies often tend to assume that because [big company/VC startup] is willing to eat the potential risk of a lock-in gone awry that the [smaller company] they work for/founded is able to eat it as well. There's definitely a usecase for all the specific services that GCP/Azure/AWS offer. Chances are though that you're not working for a company where that's the case and the more conventional tools will do just fine.
(VC Startups tend to ignore that advice mostly because they have a lot of financial resources to spend upfront, so they think it's worthwhile/that they will hit that point where GCP/Azure/AWS services become a necessity compared to a traditional tech stack.)
> I have never in 5 years heard any CTO of any company of any size worry about “lock in” when considering their cloud choices.
The company I got laid off just decided they are going multi-cloud and that anything on their original cloud offering which is vendor specific is verboten in the middle of multiple in flight projects which now probably will never complete because they basically just did it by fiat in an unfunded mandate; no new deadlines, no new thought around how.
No, its stupid and wrongheaded, but that doesn't mean people don't do it, just giving an example of my most recent experience but I have seen it multiple times. They believe they will be able to sell their SaaS product in different clouds using existing budget companies have invested in said clouds.
People are loss averse to decisions that they made about things that are completely unrealized, it makes no sense.
One of our engineering teams wanted to use fly.io instead of kubernetes that I was running in AWS for the company. They really liked its ease of use and they really wanted to use it. Unfortunately fly.io only allowed 5,000 concurrent TCP connections, and there's was a chat app that needed to maintain many more web sockets than that for each instance that was running (More like 50,000).
So they ended up switching to kubernetes and AWS.
Fly.io is nice, but as far as scale is concerned, it's entry level.
The TCP connection limits are in place only when a central HTTP/HTTPS routing/termination is used. But you can turn that off by a config parameter in fly.toml file and by allocating static IPv4/IPv6 addresses for your app. By doing so, your app will be then fully responsible for handling all HTTP/HTTPS traffic by itself without external caps.
It's funny how people spend a lot of time and, in the end, choose a worse (unfavourable?) product just because of their inability to find a config switch that would solve all their issues and enable them to use their tool of choice.
There's quite a few reasons why it happens though. Perhaps the feature was available, perhaps there was no documentation or poor documentation. Perhaps this is more complex in some way? We don't really know.
There's a another reason to avoid fly.io outside of platform capabilities: their outages caused by growing pains. If it were me, I'd circle back in a year or two.
While my team and I were already getting progressively more skeptical of Google products for all the often retread reasons, the Google Domains issue in particular solidified my team's feelings as well into a single example as well.
I know there are SLAs, that Domains wasn't necessarily targeted at the same audience, etc. But, I'm guessing lots of folks got burned by it and I don't know how to justify trusting Google at this point. Given their struggles with their reputation and cloud offering, that situation and how they handled it still blows my mind.
That was a poor decision. Every other cloud provider has domain support. Just left my CTO role at a public company to do a another startup, GCP wasnt even in the running. It was between Azure and AWS.
Also the first company i didnt pick GSuite or whatever they are calling it now because they fail to innovate on it. Office365 has been beating it in feature launches like adding an AI copilot and AI generated meeting summaries. Spend less effort renaming products and more time building features.
Google needs to fire their leadership. They will be the next IBM if they dont.
> Successful long-lived open systems owe their success to building decades-long micro-communities around extensions/plugins, also known as a marketplace. I’ve ranted about Platforms before, and how important they are, and how Google has never once in their entire corporate history ever really understood what goes into making a successful open Platform, not counting Android or Chrome.
That one statement deserves its own essay in the context of Chrome and Firefox. It'll be interesting for anyone around in 30 years to look back and see how both those browsers are doing against the Emacs web browsers just to score the thinking.
Google engineers just don’t seem to value back compat in their bones.
I’ve read they have tools and processes internally that let them make breaking changes and globally find / fix callers. That’s cool, but the rest of don’t have that.
No one likes getting an email that says “we’ve changed the way X works, you have until Y to change your code or it will break.” Not even if Y is a year from now.
A rule that says no breaks ever is probably not optimal either but it’s a big deal and should be treated as such. Maybe require a VP sign off or something.
Also: keeping your product in beta for a decade is cute but unconvincing.
Google culture in general seems to be "this is not fun or a very profitable, so we are just going to, like, not do it anymore".
Fundamentally it comes from the fact that they don't have to do any of it anyway.
Anyone using AdWords knows that it's a money-printing machine. Google is thousands of fresh CS graduates who want to do "cool green-field stuff" but not really the day-to-day grind we are all part of and what we call "software engineering".
I think that there are competing cultures, and ultimately only one of them has the power to fire people.
I'm currently losing a multi-year long siege to kill a project with significant external visibility almost entirely because of organizational structure. A series of reorgs and new VPs have ended up such that our VP doesn't give a crap about our project and the VP that does doesn't have a great relationship with our VP.
My experience is that people really do become proud of the systems that they build and that deleting them is something they avoid. This isn't coming from junior engineers wanting to do green field stuff. This is coming from VPs who swoop in and say "what I care about is this metric and if you aren't pushing on this metric then you are fired."
There are aggravating factors. Regulation sometimes means that core infrastructure needs to change out from under you, so the "it is stable and just needs a small amount of ongoing maintenance" doesn't always persist. I think that Google's monorepo also contributes to this. It has a lot of huge benefits but it also means that "keep the old version of library X around" isn't really possible.
I think the other problem with reorgs is that the incoming VP often wants a success they can call their own.
It's a bit like lion infanticide when a pride is taken over - the new leaders are not invested in existing projects - they want to give birth to a success of their own and your maintenance of an existing project is competing for resources.
I'm doing boring enterprise maintenance for a lot less than Google money, maybe I should just apply. "No, I'm not Ivy league or MIT, but I'm great at taking abuse and grinding away thanklessly". Any chance that buys me a pass on not being great at memorizing algorithims?
Nope, the whole point of the memorizing algorithms to gain access structure is to show that you're great at grinding away thanklessly. Just saying it isn't enough.
This has nothing to do with what Google engineers value and everything to do with what management above them values. At Google you don't get promotions or raises for maintaining products, you get them for launching new products. That's the full story of how you get all these deprecations. Google management has been trying it's harded to kill the company for a long time.
Sure, it’s almost always the case that a corporate culture is heavily influenced by the incentives put in place by management. But those incentives and that culture influences what engineers value. That’s how the mechanism works.
> A rule that says no breaks ever is probably not optimal either but it’s a big deal and should be treated as such. Maybe require a VP sign off or something.
Works for Microsoft, though!
See also: all of the super crusty VB6 and Win32 apps from 20+ years ago powering a double-digit percentage of everything on the planet
I think python would've been in more trouble if it never made the transition to version 3. You probably shouldn't be building massive apps in Python in the first place.
Yeah that's the boat I'm in. I started using Python for production projects after the switch. It sucks for those who came before, but it's just fine for those who came after, and I'm not sure it hurt the platform as much as this article suggests.
Plus, they clearly learned their lesson after that switch, which is still talked about in 2023.
Python 3 was released in 2008. Python 2.7 was supported until 2020.
They supported Python 2 alongside Python 3 for 12 years. They provided an automatic conversion tool. They provided libraries which made it possible to run the same code on both 2 & 3.
The transition sucked, but they did everything they could to make it as smooth as possible.
My impression is rather different. The automatic conversion tools and compatibility libraries were only provided several years and versions into the v3 transition because the community was demanding it so hard and most users refused to do anything with v3 without it. Most of it probably wouldn't have been necessary at all, or would have been there on day 1, if they had made backwards compatibility the priority instead of going hard on the everyone rewrite everything to the new v3 style right away. I'm sure they didn't want to support Python 2 for 12 years but were instead basically forced to because the transition was so difficult for so many major projects.
Not even close. They could've not broken compatibility in the first place. They could've let Python 2 and Python 3 modules import each other in the same process.
With what they provided, even 3-4 years ago it was difficult to run some codebases on Python 3.
The Android team did need to be told. There was a lot of internal dev pressure to abandon backwards compat (by which I actually mean reproducing bugs) even between immediate revisions.
At one point their (at the time brilliant) dev relations baited me into telling them “if the targetSdk setting isn’t for this exact problem then what is it for?” and never heard of them thinking of making such changes again.
The backstory was one product I had become responsible for was reliant on a bug in the inheritance implementation of the earlier VMs. I was stunned to say the least. That product happened to be FIFA which made my case somewhat easier.
Yep. Been navigating the firestore / datastore maze and even though it mostly works I still can’t quite tell how the bits relate to each other. There is firestore on fire base side. And then on gcp side also firestore. Except it’s called datastore there. And comes in native flavour and datastore mode. And then there is firestore compatibility with native mode datastore. But also datastore in datastore mode.
And they all have different documentation pages and APIs and pricing and honestly wtf google
I can understand the confusion. A brief history lesson.
Firebase: a paas that google bought that included a real-time nosql db (also casually called firebase)
Datastore: googles OG serverless nosql db that was tightly baked into appengine as part of their own paas
After buying firebase google decided the db implementation was shit and set out to write their own. In typical google fashion they picked the most confusing name and called it firestore.
Rather than maintain 2 different nosql databases google quietly broke datastore free from appengine and migrated everyone’s data to firestore. But to maintain compatibility they kept both sets of APIs so now you have Firestore native (to get people to move off firebase) and firestore in datastore mode (for anyone who was using the og datastore). The 2 APIs still have specific features and access patterns hence the differences in billing.
You can access and manage firebase both from the GCP console and the Firebase console but it’s the same db, same as how firebase storeage is really just GCS. So that adds another layer of confusion.
This is a reminder to update your policies to avoid changes to your access to AWS Billing, Cost Management and Account consoles. Our records indicate that you are still using retired actions to access these consoles.
If your policies are not updated with new actions by December 11, 2023, your users’ access to the AWS billing, Cost Management, and Account consoles will be affected.
The policies that need to be updated to include the new fine-grained actions are listed in the “Affected Resources” tab of the AWS Health Dashboard in the “Policy | Policy Name | Policy ARN | Type” format.
To help you with the migration, we have published a mapping between old and new actions in our user guide [1]. If you need to update policies across multiple member accounts in your organization, we have built bulk policy migration scripts to help you update all policies quickly and securely from your management account. See the bulk policy migration scripts user guide [2] for more information. You can find a detailed guide on how and which policies you need to update on our blog [3] and definitions of new IAM actions in Cost Management [4] and Billing [5] user guides.
AWS will not be able to grant further exceptions after December 11, 2023, so we strongly recommend that you act promptly to migrate your policies to new actions.
If you have more questions or need help make updates to your policies, please contact AWS Support [6].
Sincerely,
Amazon Web Services
========
Not only did they send me this email, they sent me 10 copies - one for each account we have. Pages and pages of documentation, steps to follow, Cloud Formation stacks to create, Python scripts to run, etc. I'm still not completely sure what the problem was or what is supposedly improved, or if I would've actually lost access to anything, but hopefully the emails stop now.
The automation all happens on your side of the control plane. AWS isn’t going to update IAM roles for any customer. The closest you get is that roles can consume managed policies, and those could change but you know that going in.
I run things at a mid sized company and I get AWS emails like this daily. There's always something that needs attention. Except sometimes it doesn't really need attention. Other times it's something absolutely critical. Sometimes can't tell unless I read 50 pages of documentation and talk to their support.
Meanwhile at my old job, we had actual servers and vms in house. We had a machine running some obscure service sitting in a corner for two decades with no problems. Magical.
Ok, so don't change or upgrade anything for years. Sounds like a good solution. /s What happens when your VM host craps out and nobody knows how to restore it?
I once had a contract job at place that hadn't updated their systems in over 5 years. One server, a VM like you describe, had a 1500 day uptime. The VP/CIO was in a panic because nothing was supported anymore.
I've built a startup on AWS, and same. Since we started about a year ago, I've gotten two "action required" messages from AWS.
One involved migrating our Go-based lambdas from the "Go 1.x Runtime" to a more generic one that AWS wants to use for compiled binaries across languages. They provided a very helpful guide, including the Cloudformation template changes. I implemented them, and they worked perfectly just as promised.
The other message I got was rather urgent and telling me I need to upgrade the SSL cert on my RDS. The due date on that action is August 22, 2024.
There are a few areas of AWS I've found to be pretty abandoned. (Don't try to send push notifications using the AWS Platform Application + SNS approach, woof.) But overall, I've been very happy with AWS as a developer.
Oh we are posting emails. Here is one from google I have been ignoring because updating the app is so far down my todo list it will just never happen.
========
This is your final reminder: By November 1, 2023 version 4 will be deprecated and any updates to existing apps will first require you to migrate to PBL version 5 or newer. We are sending you this reminder as one or more of your apps is still using Play Billing Library (PBL) version 4.
Important notes: If your app is targeting Android 14 or higher, you must update to PBL 5.2.1 or PBL 6.0.1 or higher.
In May we launched Play Billing Library 6.0 with updates to subscription features, in-app purchase logging, and new API insights. We highly recommend upgrading directly to version 6.0 to future-proof your integration for the next two years and take advantage of the latest tools and features our commerce platform has to offer.
To be fair, this email doesn't say they are going to stop collecting payments, or that they will remove the app from the play store, only that I will not be able to update it. And given I haven't updated this one is a very very long time, its not really a big deal.
I would be more upset if they actually remove the app from the app store. I'm looking at you Apple.
recently Youtube shows you a pop up that says they banned adblockers and even suggest that you pay for Youtube Premium. However, in my country if you click the Youtube Premium link, it says YouTube Premium is not available in your country"
I don't think Google takes their non ad products seriously. They only launch their products in few countries, the teams use them for promotion then abandon the products
It's concerning to read your comment on this post, since it suggests you were unaware of all these issues people were facing with Google cloud. Was there not much cross talk or discussion of what issues the customers were experiencing with the platform as a whole?
I don't agreee with the Android related tougths. Altrough they don't deprecate API's, permission related things are changed too often. If you have any non-trivial app, you have to make changes every year to remain in Google Play and the rules are awkard.
My latest pain: I have a voip phone app in the store and last time I had to add a big popup before touching the contact list, informing the users the purpose (which is obvious with a phone app). Also the app description now contains that "this app might upload your phone numbers", altrough it only list it for the user, so they can select a contact and call it.
There are many more idiot thing, but I stop here because I already start to become angry, thinkig about the many wasted hours at their stupidity.
No cloud provider is immune to this, Azure dropping support for MariaDB is going to cost us a lot of money in migrations. It's a pretty terrible migration at that.
> I’m not actively developing on AWS, so I don’t have as much of a sense for how often they sunset APIs that they have previously dangled alluringly before unwitting developers. But I have a suspicion it’s nowhere near as often as happens at Google, and I believe wholeheartedly that this source of constant friction, and frustration, in GCP, is one of the biggest factors holding it back.
AWS very rarely deprecates APIs. The last API they deprecated (that I can recall) was EC2-Classic, and that was after 15 years of service AND having a vastly better alternative (EC2 with default VPCs) that almost everyone was already using.
In fact, AWS S3 is another example of a well-built, time-tested platform that passes the backwards compatibility test. AWS S3 spun up in 2006 or thereabouts. Even though the version of their most-current API schema is from *2012*, IIRC, you can still use code that you wrote back in 2006 to create stuff into buckets and store them.
Same with EC2 VPCs and core networking.
> Go ahead, I dare you. Follow the link and click the button. Choose “yes” to get all the default parameters and deploy the cluster to your Google Cloud project. Haha, joke’s on you; it doesn’t work. None of that shit works. It’s never tested, starts bit-rotting the minute they roll it out, and it wouldn’t surprise me if over half the click-to-deploy “solutions” (now we understand the air quotes) don’t work at all. It’s a completely embarrassing dark alley that you don’t want to wander down.
Azure 100% does the same thing with their Marketplace AMIs, and I'm almost certain that AWS has a ton of quickstart guides that straight-up don't work. My guess is that a Solutions Architect or Presales Engineer wrote the solution for a small handful of customers to use, then published it for "impact" and completely forgot about it to move onto the next big problem to solve. (Guilty as charged.)
However, if someone contacted Azure and AWS after running the broken quickstart asking for help, if they have enough money for support, they will absolutely get an SA to help them get up and running. Not sure about GCP. I suppose it's probably the same.
> Larry & Sergey got rid of all the unhealthy snacks by 2006 though.
I will always tell people about how when I worked there in Corp Eng in 2015 (worst job ever, unfortunately), all of the doors on their (many) fridges had large frosted glass films covering their bottom halves. The top half had "heatlhy" soft drinks and juices, along with seltzers and other low-calorie beverages. The bottom half? Good ol' fashioned American sodas...including diet, zero-calorie sodas!
You could go absolutely crazy gorging yourself on their infinite frozen yogurt and snacks (they definitely 10000% had unhealthy snacks when I was there) twice/day, but good luck finding the Diet Dr Pepper, I guess?
personal anecdote: used dialogflow (when it got rebranded from api.ai after getting acquired by Google) to create a chatbot in 2018.
funny to track how it got placed inside Google's AI offerings since. i may have missed a few, but it got moved inside their GCP suite of offerings, and now lost somewhere in Vertex GenAI offerings probably.
advancements happen and i understand that. but when your entire platform to achieve x changes drastically within a span of a couple years, how can we consider creating stable systems like of the past with XaaS model?
Don't think I see much in terms of new content here so to add an anecdote about a deprecation post-dating the OP:
My wife and I created an app using the GCP Maps Platform for Unity (or some name similar to it, it was definitely a GCP project) 2-3 years ago. It was for an exhibition, not a long term project though there were aspirations to maybe do something like that. But exhibition ended, and a year or so later the API was sunsetted.
Fast forward to just last week, phones have changed since then, and she wanted to rebuild the app to demo it. I had to say nope, API's gone... But we're lucky to have not actually gone forward with trying to build a business on this despite there definitely must have been people that did and had to migrate to self hosting by force.
Still I use GCP, the UX compared to AWS is just much better, both the APIs (i.e. Terraform) and console (no region!!!). I do know the typical Googler personality and suspect server features like Cloud Run are safe while anything too close to application like analytics is at risk and I'll generally avoid it. I do wonder where auth falls on this spectrum, I'd like to avoid auth0 etc since just too expensive, but it is a bit hard to trust after seeing a shutdown of an app feature.
125 comments
[ 4.1 ms ] story [ 185 ms ] threadI'm spending a lot of my weekend hours racing against the clock to keep some old app engine apps online. I wish I had started in 2020 :)
I've learnt my lesson and will be sticking with boring technology I can take anywhere in future. Node and Postgres for me from now on.
this isn't academic for me - I've got a service that's on vanilla RDS PostgreSQL but growing fast, and I'll soon need to decide whether to move to something cheaper/faster but proprietary or blow my budget trying to stay generic...
I’d ask two questions — how possible is it to isolate the code which is proprietary? …is there a third option which allows not blowing your budget, but remaining generic?
Naively, isolating the proprietary interface in a wrapper in your code minimizes changes should you need to move — and blowing your budget isn’t a long term solution.
So does it come down to the proprietary option or fail?
There are so many moving pieces doing any migration at scale, your code is the least of the problems.
I have seen it take nearly a year to migrate hundreds of generic VMs with VM hosted databases from on prem to cloud. You can’t get more generic than that.
Yes, this was with AWS Professional Services (where I use to work) doing much of the work.
Perhaps you could justify your point rather than attack other people and appeal to authority?
How is any of that relevant to what I said?
You’re ignoring that maintaining your code without modifying your data access (changing only the wrapper) significantly simplifies virtually everything you listed - with the exception of network topology changes.
So the question remains, have you ever been a part of a large scale migration? Have you ever been in the room when the planning was happening?
And yes — I’ve helped migrate global systems on the peta-scale, for finance where compliance is legally mandated. The idea that changing your code at the same time is irrelevant is laughable:
Changing your software significantly while migrating has caused major errors in projects I’ve seen attempt it — you’re essentially doing a rewrite while trying to migrate, which introduces bugs in compliance, security, etc.
- - - -
Edit to include a specific example:
Let’s imagine I’m migrating a dataset, A. And my copy is A’.
In the scenario where I have some wrappers, W and W’, I can write my validation test on the data, T, to work against that interface.
So then T(W(A)) = T(W’(A’)) implies that W(A) = W’(A’), since T is a function (and the same code, run both places). My problem in demonstrating my data integrity reduces to ensuring that my wrappers faithfully marshal data — something I’ll get by writing unit tests.
In the scenario where I don’t have those data wrappers, I have to write two tests — T and T’.
But then what does T(A) = T’(A’) prove? Nothing directly — I need to prove that my two integrity tests are actually measuring the same thing.
In this way, isolating the complexity of your data marshaling directly impacts the complexity of downstream tasks, such as verifying the integrity of your data.
They are talking to the board about strategic decisions and what will give them a competitive advantage. He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
A dependency always slips in somewhere unless you are constantly testing and preparing for portability like Uber does.
Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
Let alone a dependency on all of the arns always having “aws” in them not thinking they wouldn’t work in China or gov-cloud. I hardcoded the partition.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
Another case is that “we are using MySQL to avoid lock in with AWS”.
Years later.
“Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”
> He’s not losing sleep over whether you put a facade over your data access layer because in some distant future long after he is gone, AWS may raise prices.
Of course not — he’s paying experts to avoid that situation, aligning with his strategic priorities.
That’s my job.
You also resorted back to personal attacks because you can’t make your point directly.
> If the spend is large enough, he’s going to talk to his dedicated sales rep to lower prices long before he comes to zmgsabt and asks him did he make his data access class “cloud agnostic”.
Like this ridiculous comment to someone whose job is being hired by CTOs looking to migrate.
They literally come to me, for help doing exactly that.
> Hell, I released code that was part of a major official open source “AWS Solution” and got complaints a few weeks later that it was dependent on a region and that’s not the first time I’ve seen that happen.
> I hardcoded the partition.
> “Oh shit. The data team is loading data into the database using the AWS extension that lets you load data from S3 using a sql query”
Yes — that’s why companies pay me: I’m used to doing things that can’t depend on a region, service, etc while AWS engineering frequently makes that mistake (eg, all the US-East-1 magic).
That’s precisely my point:
Quality coding avoids and bounds mistakes like that, which slow your migration.
- - - -
So you’ve resorted to personal attacks, but didn’t actually respond to my specific example, and were insistent on my credentials but still haven’t shared the largest system you personally have migrated.
But please keep it up: I make my money from people like you causing CTOs to hire people like me to clean up the mess — “whoops, accidentally wrote regionalized code!” or “we need over a year to migrate 100 VMs due to self-induced complexity!”
Are you going to:
A) write an ETL job that is going to be slower and more complex to avoid the “lock-inz”
Or
B) just call run
“load data from S3…”
As a sql extension?
Are you going to tell them that they can’t use Glue? Athena? Redshift? Quicksite?
When your company needs a call center are you going to tell them not to use Amazon Connect? Now they need to integrate the call center with Salesforce - are you going to tell them they can’t do that because they have to use Lambda?
They want to have speech to text - now they need Amazon Lex.
Then they want a reporting dashboard that comes from their contact trace records. Now they need Kinesis which is going to stream to S3.
The typical company uses over 250 SaaS products that they have some form of integration with. Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
If you work in the health care industry are you going to tell them not to use Epic?
Which contrary to your repeated personal attacks is the sort of conversation I have with corporate leaders: they determine the strategy; I make it work technically.
But yes — there are clients who insist on avoiding those services (and generally would, if they’re migrating out of S3).
And in this particular context, we were explicitly discussing the decision of how to migrate a product into the cloud where I advocated using the cloud solution but taking a small step which would both ease that transition and any future ones — and you flipped out.
- - - -
To address your dramatic nonsense:
You don’t have to use Lambda to integrate with Salesforce; and you know that non-AWS call centers are used by most AWS customers.
> Are you going to tell them not to use any of those either? Are you going to tell them not to use Salesforce? ServiceNow? Microsoft Office?
This is just ridiculous dramatics completely unrelated to anything I said.
- - - -
Let’s ask a few questions to refocus:
- why were you so insistent on my credentials, but won’t share yours? …are you embarrassed by your largest migration?
- why can’t you address my specific example? …because it shows I have good advice and you spouted nonsense?
- why are you melting down into unrelated dramatics? …or pretending I advocated against using cloud solutions? …or integrations at all?
I think you’re just throwing up arguments at random because you can’t admit that on the original issue I was right.
But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
None of this is “making the beer taste better”.
I’ve shared by credentials up thread - 2 years heading a migration and integration when a private equity owned health care company was acquiring companies to get big and go public, 3 years leading the application modernization effort for another company that was acquired at 10x revenue and three years working at AWS in the Professional Services department and now building out a professional services practice for an MSP for applications modernization.
And before that over 20 years as a professional developer
https://help.salesforce.com/s/articleView?id=sf.integrate_wh...
> But you’re so concerned with “cloud lock in” and you completely ignore the other 200+ services that the average enterprise is using. Why are you so concerned with the cloud, but not Salesforce, ServiceNow, Sharepoint, Okta, Azure AD, etc?
This is a fantasy you invented about me because you cannot respond to my actual point — a strawman.
My original advice was to use the cloud — and I was addressing someone else’s concern about lock-in, while giving advice that would aid their initial migration to using cloud services.
I think that it’s very telling you can’t be honest about what I said.
> It makes no technical sense to spend developer time and add complexity to create an ETL job to do what a simple SQL statement can do.
As is this — another strawman you invented.
- - - -
Why do you have two strawmen, but haven’t in several posts actually addressed my specific example?
Are you unable to?
- - - -
What you’ve said is that you spent two years migrating just a hundred VMs, which sounds like a migration delayed by self-induced complexity.
I appreciate arrogantly incompetent developers like yourself — who can’t address technical points and instead resort to fallacious thinking:
Your work gets me paid.
So I’m an “incompetent developer” who someone managed to get hired at AWS Professional Services not a third party partner - you to consult the largest organizations on the planet?
I’m sure an AbstractFactoryRepositoryFacade is going to prevent cloud lock-in while you advise people to create a complicated workflow when a simple SQL statement using an AWS extension would be faster to develop, more performant and less maintenance?
It’s not a straw man it goes to the core of the efficiency gains you get by taking advantage of your vendors functionality and integrating into it tightly instead of trying to maintain a leaky abstraction.
You're always "locked in" to some degree and it's an almost worthless thing to invest into if you never actually need to actually migrate off.
The difference being whether you are internally maintaining your special snowflake, or else paying for a solution maintained for you by a 3rd party.
For some individuals, sure. But other people and especially businesses don't really want change for change's sake (and if possible want to have a more-or-less set-and-forget thing), and the article clearly explains why it is better with AWS if that's the goal.
As a customer, I'd be fine with GCP doing things to keep their costs down so GCP pricing stays the same or decreases. But I understand this is hard to quantify outside GCP and requires a bit of faith (which I accept as inherent, since full transparency is impossible).
The author's take is brutal, but totally on-point. The cloud business is about asking businesses to use your system as a foundation. Google doesn't take that seriously enough.
When everything takes 20% longer to ship than you planned because every other week some random team(s) is forced to do unplanned, immediate rewrites by a vendor, that vendor starts to look like a dick and also a huge liability to your business.
Also Google's own documentation is full of grenades. Just yesterday I found in their GKE docs something they tell you is the default and you must use it...except it doesn't apply to clusters created a few versions ago, even if they're updated, because they didn't roll their change out to existing clusters -- just new ones created after their change.
Also google's docs reflect their deprecation policy -- there is no document versioning. You can always and only see latest.
This vendor is certainly not GCP in this case.
Every other week? Immediate rewrites? Is this even based on reality?
I've had many, many days in my career of having to drop everything to patch mission-critical glue code to keep up with Google APIs.
Also keep in mind that the documentation for their SDKs is frequently outdated and incorrect, especially around authentication and to the point that I often just deferred to reading the source code over reading their docs. And that's even if their SDK was in a working state, because often their own SDKs would lag behind their own API changes and you'd have to try out ruby, python or php to find one that worked properly (although golang always works, because that's what Googlers actually give a shit about).
Maybe its gotten better in the last few years, but that was solidly my experience from 2010-2018 and those were a lot closer to "Peak Google" than now. Now I write less code against their APIs, but I also have about 10M/yr in GCP infrastructure that I manage and it's also a treadmill even though we stick to the basics (VMs, GCS, Cloud SQL, GKE).
IMHO, the only thing cloud providers are genuinely useful for is a bottomless data storage. But even that aspect is now isolated by Rclone in all my apps since migrating off the cloud. Apps just talk with the file system, which is then abstracted to a cloud storage of choice by Rclone.
The lockin happens when DevOps gets lazy and start using a bunch of cloudy functions.
Have you ever done a large migration at scale? I have - even moving over a lot of VM hosted services involves your project management organization, regression testing, networking, firewall, training, security, working with outside vendors, and the list goes on.
The average enterprise uses 254 SaaS services. I have never in 5 years heard any CTO of any company of any size worry about “lock in” when considering their cloud choices.
Source: 2 years as an architect responsible for integrating acquired companies before a private equity owned company went public, two years leading the “application modernization” effort at a startup that exited 10x revenue, 3 years working at AWS in the Professional Services department and now working at a smaller consulting shop.
And I got Amazoned a couple of months ago. I have no particular loyalty toward Amazon
That's where the lock-in is, Kubernetes vs VMs isn't that big a deal at all; annoying sure. For the record, my employer uses both Kubernetes and VM. The deployment is pretty much the same.
Everyone here is completely ignoring the organization complexities.
Developer time is very, very expensive, and any time you spend "migrating" is time not spent building features that add value. Except in rare cases, the end customer probably doesn't care if your stuff runs on AWS, GCP, or Azure.
Are you going to suggest that your medical software doesn’t integrate with third party EMH/EMR systems? That your call center and sales team doesn’t integrate with Salesforce?
What about your dependency on Workday? ServiceNow? Your project management software? All of the Microsoft software?
Yes even VMWare. There is a reason that AWS has a VMWare offering.
Exactly my experience. Some services I manage are still entrenched in Big Co. Cloud, and it's a hell. This lock-in costs us additional money (thousands per year), but that amount is less than it would cost us to migrate those apps immediately.
Unix is cool not because it's a Unix, but because it's simple, aka Unix Philosophy.
The hairy stuff is when you get into the highly specific offerings for each SaaS platform. They all offer a ton of them and they can certainly be attractive for some cases (ie. Filehosting) or useless in others (there's a lot of stuff that amounts to "our version of cache/database" and little else) but that's where the lock-in issues tend to play up.
GCP is uniquely terrible though from what I've heard in terms of support lifespan and utter unhelpfulness in terms of communicating to their customers when some SaaS offering of theirs are on the chopping block.
The reality though is that most tech stacks aren't that complicated. Not every company is a venture capital "go big or go bust, we change the entire industry" darling (nor are they necessarily looking to grow that way, especially outside the SV bubble) or a government with 2000 different contracts and enough yellow tape + bureaucracy around those contracts that signing a new contract is just easier than possibly extending an existing one with new features.
For the rest of those tech stacks (which are definitely "enterprise", simply by definition, since they get used in companies and are bespoke enough to be their own tools), it's rare to see a stack more complex than the one I mentioned (backend, cache, db) unless a technical admin decided to play runaway with Azure/Google/Amazon because it's shiny and exciting.
In those cases, the vendor lock-in from using a bespoke AWS or Azure solution are absolutely worth it because if AWS or Azure are an active problem they also will have the manpower to migrate off of AWS/Azure. My suggestion/advice is for the other 99%. You still want AWS/Azure for CYA, the managed part (not having to deal with ops) and backups but you don't need all the extra solutions they offer.
It's why "you are not Google" is an advice that exists. Newbies often tend to assume that because [big company/VC startup] is willing to eat the potential risk of a lock-in gone awry that the [smaller company] they work for/founded is able to eat it as well. There's definitely a usecase for all the specific services that GCP/Azure/AWS offer. Chances are though that you're not working for a company where that's the case and the more conventional tools will do just fine.
(VC Startups tend to ignore that advice mostly because they have a lot of financial resources to spend upfront, so they think it's worthwhile/that they will hit that point where GCP/Azure/AWS services become a necessity compared to a traditional tech stack.)
Startups should use whatever technology that gets them to market fastest and the last thing they should be worried about is “lock-in”.
If they find product market fit and get customers, in normal times the money will come. Why worry about the “undifferentiated heavy lifting”?
Sure for Dropbox or BackBlaze, managing storage is their competitive advantage for instance.
But creating a AbstractFactoryRepositoryKeyValueFacade to avoid a DynanoDB dependency makes absolutely no sense.
The company I got laid off just decided they are going multi-cloud and that anything on their original cloud offering which is vendor specific is verboten in the middle of multiple in flight projects which now probably will never complete because they basically just did it by fiat in an unfunded mandate; no new deadlines, no new thought around how.
Is it giving your former company a competitive advantage?
People are loss averse to decisions that they made about things that are completely unrealized, it makes no sense.
So they ended up switching to kubernetes and AWS.
Fly.io is nice, but as far as scale is concerned, it's entry level.
I know there are SLAs, that Domains wasn't necessarily targeted at the same audience, etc. But, I'm guessing lots of folks got burned by it and I don't know how to justify trusting Google at this point. Given their struggles with their reputation and cloud offering, that situation and how they handled it still blows my mind.
Also the first company i didnt pick GSuite or whatever they are calling it now because they fail to innovate on it. Office365 has been beating it in feature launches like adding an AI copilot and AI generated meeting summaries. Spend less effort renaming products and more time building features.
Google needs to fire their leadership. They will be the next IBM if they dont.
Of the companies I've worked for, half had GSuite and half had office. Personally, I'll take GSuite any and every time.
I'd be okay with office365 if teams wasn't such a shitty piece of software.
slack so far has been the best corporate chat experience.
Which is still a pretty good position... endless amounts of money coming in from legacy services.
That one statement deserves its own essay in the context of Chrome and Firefox. It'll be interesting for anyone around in 30 years to look back and see how both those browsers are doing against the Emacs web browsers just to score the thinking.
I’ve read they have tools and processes internally that let them make breaking changes and globally find / fix callers. That’s cool, but the rest of don’t have that.
No one likes getting an email that says “we’ve changed the way X works, you have until Y to change your code or it will break.” Not even if Y is a year from now.
A rule that says no breaks ever is probably not optimal either but it’s a big deal and should be treated as such. Maybe require a VP sign off or something.
Also: keeping your product in beta for a decade is cute but unconvincing.
Fundamentally it comes from the fact that they don't have to do any of it anyway.
Anyone using AdWords knows that it's a money-printing machine. Google is thousands of fresh CS graduates who want to do "cool green-field stuff" but not really the day-to-day grind we are all part of and what we call "software engineering".
I'm currently losing a multi-year long siege to kill a project with significant external visibility almost entirely because of organizational structure. A series of reorgs and new VPs have ended up such that our VP doesn't give a crap about our project and the VP that does doesn't have a great relationship with our VP.
My experience is that people really do become proud of the systems that they build and that deleting them is something they avoid. This isn't coming from junior engineers wanting to do green field stuff. This is coming from VPs who swoop in and say "what I care about is this metric and if you aren't pushing on this metric then you are fired."
There are aggravating factors. Regulation sometimes means that core infrastructure needs to change out from under you, so the "it is stable and just needs a small amount of ongoing maintenance" doesn't always persist. I think that Google's monorepo also contributes to this. It has a lot of huge benefits but it also means that "keep the old version of library X around" isn't really possible.
Who is still in the same position or team 10, or even 2 years, or even 6 months later, especially in any faang?
It's a bit like lion infanticide when a pride is taken over - the new leaders are not invested in existing projects - they want to give birth to a success of their own and your maintenance of an existing project is competing for resources.
This is exactly my impression judging by the quality of their products and Google's interviewing process.
This describes me, and they hired me.
Works for Microsoft, though!
See also: all of the super crusty VB6 and Win32 apps from 20+ years ago powering a double-digit percentage of everything on the planet
Plus, they clearly learned their lesson after that switch, which is still talked about in 2023.
They supported Python 2 alongside Python 3 for 12 years. They provided an automatic conversion tool. They provided libraries which made it possible to run the same code on both 2 & 3.
The transition sucked, but they did everything they could to make it as smooth as possible.
With what they provided, even 3-4 years ago it was difficult to run some codebases on Python 3.
At one point their (at the time brilliant) dev relations baited me into telling them “if the targetSdk setting isn’t for this exact problem then what is it for?” and never heard of them thinking of making such changes again.
The backstory was one product I had become responsible for was reliant on a bug in the inheritance implementation of the earlier VMs. I was stunned to say the least. That product happened to be FIFA which made my case somewhat easier.
And they all have different documentation pages and APIs and pricing and honestly wtf google
Firebase: a paas that google bought that included a real-time nosql db (also casually called firebase)
Datastore: googles OG serverless nosql db that was tightly baked into appengine as part of their own paas
After buying firebase google decided the db implementation was shit and set out to write their own. In typical google fashion they picked the most confusing name and called it firestore.
Rather than maintain 2 different nosql databases google quietly broke datastore free from appengine and migrated everyone’s data to firestore. But to maintain compatibility they kept both sets of APIs so now you have Firestore native (to get people to move off firebase) and firestore in datastore mode (for anyone who was using the og datastore). The 2 APIs still have specific features and access patterns hence the differences in billing.
You can access and manage firebase both from the GCP console and the Firebase console but it’s the same db, same as how firebase storeage is really just GCS. So that adds another layer of confusion.
Hope this helps.
========
Hello,
This is a reminder to update your policies to avoid changes to your access to AWS Billing, Cost Management and Account consoles. Our records indicate that you are still using retired actions to access these consoles.
If your policies are not updated with new actions by December 11, 2023, your users’ access to the AWS billing, Cost Management, and Account consoles will be affected.
The policies that need to be updated to include the new fine-grained actions are listed in the “Affected Resources” tab of the AWS Health Dashboard in the “Policy | Policy Name | Policy ARN | Type” format.
To help you with the migration, we have published a mapping between old and new actions in our user guide [1]. If you need to update policies across multiple member accounts in your organization, we have built bulk policy migration scripts to help you update all policies quickly and securely from your management account. See the bulk policy migration scripts user guide [2] for more information. You can find a detailed guide on how and which policies you need to update on our blog [3] and definitions of new IAM actions in Cost Management [4] and Billing [5] user guides.
AWS will not be able to grant further exceptions after December 11, 2023, so we strongly recommend that you act promptly to migrate your policies to new actions.
If you have more questions or need help make updates to your policies, please contact AWS Support [6].
Sincerely, Amazon Web Services
========
Not only did they send me this email, they sent me 10 copies - one for each account we have. Pages and pages of documentation, steps to follow, Cloud Formation stacks to create, Python scripts to run, etc. I'm still not completely sure what the problem was or what is supposedly improved, or if I would've actually lost access to anything, but hopefully the emails stop now.
If automatable changes are required.. pop up a dialog and ask me to confirm them?
Meanwhile at my old job, we had actual servers and vms in house. We had a machine running some obscure service sitting in a corner for two decades with no problems. Magical.
I once had a contract job at place that hadn't updated their systems in over 5 years. One server, a VM like you describe, had a 1500 day uptime. The VP/CIO was in a panic because nothing was supported anymore.
One involved migrating our Go-based lambdas from the "Go 1.x Runtime" to a more generic one that AWS wants to use for compiled binaries across languages. They provided a very helpful guide, including the Cloudformation template changes. I implemented them, and they worked perfectly just as promised.
The other message I got was rather urgent and telling me I need to upgrade the SSL cert on my RDS. The due date on that action is August 22, 2024.
There are a few areas of AWS I've found to be pretty abandoned. (Don't try to send push notifications using the AWS Platform Application + SNS approach, woof.) But overall, I've been very happy with AWS as a developer.
========
This is your final reminder: By November 1, 2023 version 4 will be deprecated and any updates to existing apps will first require you to migrate to PBL version 5 or newer. We are sending you this reminder as one or more of your apps is still using Play Billing Library (PBL) version 4. Important notes: If your app is targeting Android 14 or higher, you must update to PBL 5.2.1 or PBL 6.0.1 or higher. In May we launched Play Billing Library 6.0 with updates to subscription features, in-app purchase logging, and new API insights. We highly recommend upgrading directly to version 6.0 to future-proof your integration for the next two years and take advantage of the latest tools and features our commerce platform has to offer.
Says as much in a few words as the original article.
I would be more upset if they actually remove the app from the app store. I'm looking at you Apple.
I don't think Google takes their non ad products seriously. They only launch their products in few countries, the teams use them for promotion then abandon the products
> ...GNU Emacs, which is a sort of hybrid between Windows Notepad, a monolithic-kernel operating system, and the International Space Station
I actually worked on GCP for several years and we always made sure our customer facing API changes were backwards compatible.
The decision to deprecate or “sunset” a feature was not taken lightly and needed to be announced O(years) ahead of time.
Apparently, my team was an exception rather than the norm.
I was aware that people criticized GCP for various reasons, but there wasn’t much I could do beyond my team/product area.
AWS very rarely deprecates APIs. The last API they deprecated (that I can recall) was EC2-Classic, and that was after 15 years of service AND having a vastly better alternative (EC2 with default VPCs) that almost everyone was already using.
In fact, AWS S3 is another example of a well-built, time-tested platform that passes the backwards compatibility test. AWS S3 spun up in 2006 or thereabouts. Even though the version of their most-current API schema is from *2012*, IIRC, you can still use code that you wrote back in 2006 to create stuff into buckets and store them.
Same with EC2 VPCs and core networking.
> Go ahead, I dare you. Follow the link and click the button. Choose “yes” to get all the default parameters and deploy the cluster to your Google Cloud project. Haha, joke’s on you; it doesn’t work. None of that shit works. It’s never tested, starts bit-rotting the minute they roll it out, and it wouldn’t surprise me if over half the click-to-deploy “solutions” (now we understand the air quotes) don’t work at all. It’s a completely embarrassing dark alley that you don’t want to wander down.
Azure 100% does the same thing with their Marketplace AMIs, and I'm almost certain that AWS has a ton of quickstart guides that straight-up don't work. My guess is that a Solutions Architect or Presales Engineer wrote the solution for a small handful of customers to use, then published it for "impact" and completely forgot about it to move onto the next big problem to solve. (Guilty as charged.)
However, if someone contacted Azure and AWS after running the broken quickstart asking for help, if they have enough money for support, they will absolutely get an SA to help them get up and running. Not sure about GCP. I suppose it's probably the same.
> Larry & Sergey got rid of all the unhealthy snacks by 2006 though.
I will always tell people about how when I worked there in Corp Eng in 2015 (worst job ever, unfortunately), all of the doors on their (many) fridges had large frosted glass films covering their bottom halves. The top half had "heatlhy" soft drinks and juices, along with seltzers and other low-calorie beverages. The bottom half? Good ol' fashioned American sodas...including diet, zero-calorie sodas!
You could go absolutely crazy gorging yourself on their infinite frozen yogurt and snacks (they definitely 10000% had unhealthy snacks when I was there) twice/day, but good luck finding the Diet Dr Pepper, I guess?
funny to track how it got placed inside Google's AI offerings since. i may have missed a few, but it got moved inside their GCP suite of offerings, and now lost somewhere in Vertex GenAI offerings probably.
advancements happen and i understand that. but when your entire platform to achieve x changes drastically within a span of a couple years, how can we consider creating stable systems like of the past with XaaS model?
My wife and I created an app using the GCP Maps Platform for Unity (or some name similar to it, it was definitely a GCP project) 2-3 years ago. It was for an exhibition, not a long term project though there were aspirations to maybe do something like that. But exhibition ended, and a year or so later the API was sunsetted.
Fast forward to just last week, phones have changed since then, and she wanted to rebuild the app to demo it. I had to say nope, API's gone... But we're lucky to have not actually gone forward with trying to build a business on this despite there definitely must have been people that did and had to migrate to self hosting by force.
Still I use GCP, the UX compared to AWS is just much better, both the APIs (i.e. Terraform) and console (no region!!!). I do know the typical Googler personality and suspect server features like Cloud Run are safe while anything too close to application like analytics is at risk and I'll generally avoid it. I do wonder where auth falls on this spectrum, I'd like to avoid auth0 etc since just too expensive, but it is a bit hard to trust after seeing a shutdown of an app feature.