Their problem isn't serverless, rather Cloudflare Workers and WebAssembly.
All major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.
For a best price-to-performance ratio create your instances and do whatever is needed on them. Software stacks are not that complicated to delegate everything to the Wizards of Cloud Overcharging.
The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.
- Eliminated complex caching workarounds and data pipeline overhead
- Simplified architecture from distributed system to straightforward application
We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions.
Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever.
Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.
As someone who worked with serverless for multiple years (mostly amazon lambda but others too) i can absolutely apporove the authors points.
While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems".
Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data.
Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks.
While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy".
For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.
I've observed massive back office pipelines using dozens of interconnected lambda, batching, streaming, distributed storage for ephemeral data and other rube Goldberg contraptions to build what was ultimately a cron job on a modest server running for 1 hour.
Being in the cloud doesn't mean you need to accept timeouts/limitations. CDK+fargate can easily run an ephemeral container to perform some offline processing.
Incredible that these kinds of services were hosted like this.
I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering.
Why didnt they go with a self hosted backend right away?
Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.
After building my first Serverless/Cloudflare worker app, this is why I migrated to Deno.
Deno enables you to run the same codebase in deno (self-hosted/local) and in deno deploy (serverless platform from deno).
I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo).
I think developers are drowning in tools to make things "easy", when in truth many problems are already easy with the most basic stuff in our tool belt (a compiler, some bash scripts, and some libraries). You can always build up from there.
This tooling fetish hurts both companies and developers.
I'm working on a project right now that should be two or three services running on a VM. Instead we have 40+ services spread across a K8s cluster with all the Helm Chart, ArgoCD, CICD pipeline fun that comes with it.
It drives me absolutely nuts. But hey if the company wants to pay me to add all that stuff to my resume, I guess I shouldn't complain.
I once started working at a company that sold one of those visual programming things, and during training I was tasked with making a simple program, I was a bit overwhelmed with the amount of bugs and lack of tools to make basic features, so I made a prototype of the application I wanted in python, the plan was to port it later, I got it in a couple of days.
The tool developers weren't keen of the idea, they told me "Yeah, I can solve the problem with a script too, the challenge is to do it with our tool". And I thought it was kind of funny how they admitted that the premise of the tool didn't work.
It's like this holy grail panacea that arises 20 times every month, developers want to invent something that will avoid the work of actually developing, so they sunk-cost-fallacy themselves into a deep hole out of which they can only escape if they admit that they are tasked with automating, they cannot meta-automate themselves, and that they will have to gasp do some things manually and repeatedly, like any other working class.
Agreed, but not Bash. Bash should not be used for anything except interactive use. It's just way too error-prone and janky otherwise.
I am not at all a fan of Python but even so any script that you write in Bash would be better in Python (except stuff like installation scripts where you want as few dependencies as possible).
If it's worth saving in a file, it's worth not using Bash.
"Self-Hosting : Being tied to Cloudflare's runtime meant our customers couldn't self-host Unkey. While the Workers runtime is technically open source, getting it running locally (even in dev mode) is incredibly difficult.
With standard Go servers, self-hosting becomes trivial:"
A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.
Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO.
Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards.
I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.
As someone with years of experience on serverless stuff on AWS I might be a bit biased BUT I'd argue serverless is the sweet spot for most applications. You need to remember however that most applications aren't your typical startups or other software products but simply some rather boring line of business software nobody outside the company owning it knows of.
Concerning how IT departments in most non-software companies are, the minimal operational burden is a massive advantage and the productivity is great once you have a team with enough cloud expertise. Think bespoke e-commerce backends, product information management systems or data platforms with teams of a handful of developers taking responsibility for the whole application lifecycle.
The cloud expertise part is a hard requirement though but luckily on AWS the curriculum is somewhat standardized through developer and solutions architect certifications. That helps if you need to do handovers to maintenance or similar.
That said, even as a serverless fan, I immediately thought of containers when the performance requirements came up in the article. Same with the earlier trending "serverless sucks" about video processing on AWS. Most of the time serverless is great but it's definitely not a silver bullet.
I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster.
Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.
> The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster
The problem here is that pretty much all services have dependencies - if they didn't, you could have already moved that logic client-side.
This bites edge-compute architectures really hard. You move compute closer to the client, but far from the DB, and in doing so your overall latency is almost always worse.
The most latency-friendly architecture is almost always going to be an in-memory DB on a monolithic server (albeit this comes with its own challenges around scaling and redundancy)
I think the "local maximum" we've gotten stuck at for application hosting is having a docker container as the canonical environment/deliverable, and injecting secrets when needed. That makes it easy to run and test locally, but still provides most of the benefits I think (infrastructure-as-code setups, reproducibility, etc). Serverless goes a little too far for most applications (in my opinion), but I have to admit some apps work really well under that model. There's a nearly endless number of simple/trivial utilities which wouldn't really gain anything from having their own infrastructure and would work just fine in a shared or on-demand hosting environment, and a massively scaled stateless service would thrive under a serverless environment much more than it would on a traditional server.
That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.
One thing I could not find in the write-up is the change in the expense. Did serverless save any money, compared to always-up VMs? Did much of their load run under the free tier limits?
Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)
49 comments
[ 2.3 ms ] story [ 72.1 ms ] threadAll major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.
Source work somewhere where you easily get 1ms cached relational DB reads from outside the service.
30ms makes me suspect it went cross region.
- Eliminated complex caching workarounds and data pipeline overhead
- Simplified architecture from distributed system to straightforward application
We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions. Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever.
Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.
"Down with serverless! Long live serverless!"
While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems".
Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data.
Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks.
While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy".
For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.
Being in the cloud doesn't mean you need to accept timeouts/limitations. CDK+fargate can easily run an ephemeral container to perform some offline processing.
I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering.
Why didnt they go with a self hosted backend right away?
Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.
I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo).
Here is a link to my reasoning from back then: https://github.com/K0IN/Notify/pull/77#issuecomment-16776070...
This tooling fetish hurts both companies and developers.
It drives me absolutely nuts. But hey if the company wants to pay me to add all that stuff to my resume, I guess I shouldn't complain.
The tool developers weren't keen of the idea, they told me "Yeah, I can solve the problem with a script too, the challenge is to do it with our tool". And I thought it was kind of funny how they admitted that the premise of the tool didn't work.
It's like this holy grail panacea that arises 20 times every month, developers want to invent something that will avoid the work of actually developing, so they sunk-cost-fallacy themselves into a deep hole out of which they can only escape if they admit that they are tasked with automating, they cannot meta-automate themselves, and that they will have to gasp do some things manually and repeatedly, like any other working class.
I am not at all a fan of Python but even so any script that you write in Bash would be better in Python (except stuff like installation scripts where you want as few dependencies as possible).
If it's worth saving in a file, it's worth not using Bash.
50 tools in ops/backend to make my life easier.
75 tools in the frontend to make my life easier.
45 different tools used by product to make my life easier.
20 used by HR to make my life easier.
10 used by office management to make my life easier.
None of them really do.
With standard Go servers, self-hosting becomes trivial:"
A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.
Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards.
I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.
Concerning how IT departments in most non-software companies are, the minimal operational burden is a massive advantage and the productivity is great once you have a team with enough cloud expertise. Think bespoke e-commerce backends, product information management systems or data platforms with teams of a handful of developers taking responsibility for the whole application lifecycle.
The cloud expertise part is a hard requirement though but luckily on AWS the curriculum is somewhat standardized through developer and solutions architect certifications. That helps if you need to do handovers to maintenance or similar.
That said, even as a serverless fan, I immediately thought of containers when the performance requirements came up in the article. Same with the earlier trending "serverless sucks" about video processing on AWS. Most of the time serverless is great but it's definitely not a silver bullet.
Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.
The problem here is that pretty much all services have dependencies - if they didn't, you could have already moved that logic client-side.
This bites edge-compute architectures really hard. You move compute closer to the client, but far from the DB, and in doing so your overall latency is almost always worse.
The most latency-friendly architecture is almost always going to be an in-memory DB on a monolithic server (albeit this comes with its own challenges around scaling and redundancy)
That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.
That’s not a technology issue.
Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)