>the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered.
Lesson learned: ensure your design is sane before turning your PoC into a full blown production system.
It is almost ridiculously common anti-pattern these days to see teams stringing lambda functions together for no good reason other then “cuz serverless”, or “microservices”, for what really should be just function calls in a single process.
Developers are losing touch on what process boundaries mean, and the result is brittle and poor performing systems.
I regularly review design from teams that turn reading a file from S3 and putting into into a database into, well, something that requires a design review. Some horrific monstrosity of step functions, lambdas, and glue. All because they've been told that serverless is the correct way to build "scalable" software.
If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing.
Cloud providers are surely pleased as punch that there is now an entire generation of developers that will argue vehemently that infrastructure that only grows over time is not only healthy, but the best way to build scalable software.
I have no idea what to call it other than "Marketing Material Driven Development".
And on the flipside - One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
I sometimes worry for the future of our industry because of exactly this. The culture of buzzword driven development has become enshrined in the recruitment process - often from both sides - because of this kind of vicious circle effect. Now anyone - again from either side - who tries to escape it and instead advocates using sensible, robust tech to build useful, good quality products risks becoming a pariah and seriously damaging their career as a candidate or their ability to recruit strong candidates as an employer. Of course this assumes we are judging career progress by level of seniority reached or compensation paid instead of more indirect achievements like producing something that provided real value to those who used it. But if we're being honest most people obviously will.
I was job hunting about a year ago and got rejected several times for this reason. The worst part? Each one was a small startup just trying to get a new product off the ground.
In one interview the engineer interviewing me explained how they didn’t have any paying customers, and then went on to describe an elaborate over-engineered micro-service based architecture he was working on.
The real kicker was that instead of asking me about my 20 years building products, or our infrastructure at my last gig that handled millions of customers across the globe, he instead wanted to watch me code some goofy coding quiz he had prepared.
That's because hiring managers and team leads genuinely care about this stuff. Your last job had you working in a giant monstrous horrible Java EE thing from 2005, and you refactored 500k lines and saved the company $10 million? Sorry, we are only looking to hire somebody who is good enough to know they need to always be learning and keeping up to date with modern standards".
It's horseshit but at least half of the people in charge of hiring think exactly that way and so I end up leaving the work I'm the most proud of off my resume a lot of the time.
> Some horrific monstrosity of step functions, lambdas, and glue
I could see that. Lambda has a 15 minute timeout, and if you’re processing something large you want an orchestrator that can manage retries. I don’t like the SFN developer experience, but it’s only like 15% worse than boto3.
Why is spinning up new infrastructure wrong when that infrastructure is consumption based and easy to create?
Cloud providers don’t care about your serverless function utilization for 95% of people. A huge chunk of people will never pay more than a single digit serverless bill, regardless of if they have 450 lambdas or 40. If creating a new function matches your mental model of how things works, simplifies ops and log reading, or gives you access to a library in a different language, I’d say you made a great call.
Sure, creating is easy, what about monitoring, testing, rate limiting, updating (it can't be atomically updated with another resource, good luck testing that), security patching? What if you create a condition which causes an infinite loop of invocations? This is really rare in monoliths because we all agree on patterns like MVC that determine who calls what. When Conway s Law determines your infrastructure this is no longer the case.
Yes cloud services make it trivial to set up and run software offerings. They are designed so it is easy to enter and hard to leave. And they make things unnecessarily complicated and expensive over if it grows to any non trivial degree. By then you're stuck rationalizing your previous choice because it would be very costly and complicated to move away. I'm glad you haven't had these problems, but it doesn't mean they don't exist. The issue is, when you have one of these 'good to have' scaling problems after going all in on cloud vendor hype, it's too late. So random internet strangers who have seen the issues that lie ahead try to warn other people. Some will arrogantly say they see no problem, full steam ahead. That'll just mean they will have only themselves to blame if they ever find themselves in this quagmire.
> If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing.
I have to call out your take because it's either outright wrong or is overgeneralizing, to put it politely.
A lambda is just someone else's computer running your function, which you call through a fancy rpc interface. There is no infrastructure.
Just because you can and should deploy a lambda with infrastructure-as-code tools such as CloudFormation or CDK, it still does not make a lambda "infrastructure". It's just your function running on space CPU cycles of someone else's computer, which you call through RPC.
Running your lambda as a background task in someone else's computer does not make it infrastructure, just like running code in your own server is not adding infrastructure.
Also, it seems you're missing the whole point of offloading background processes out of your servers. If they are computationally demanding and your servers are well provisioned them a background task can lead your system to scale up. You don't change the way your system is deployed or is operating if you run your background task as a lambda. You make a request and things run just the same. Zero potential for alarms and being paged in the middle of the night.
And more importantly, for some usecases lambdas are entirely free. Does this have no value to you?
Some people feel compelled to criticize everyone around them just to feel self important. A little humility would serve them well. Just because you fail to understand something's traits and properties that does not mean everyone else around you is wrong and you are the only enlightened soul who is right.
The term "infrastructure" has kind of a loose and fuzzy definition these days. But basically I think of a serverless function as like a one-shot container that is spun up when a webhook or some other invocation method is called, handles some kind of request, puts out some kind of response, and shuts down. A container is infrastructure, ECS is infrastructure. Therefore a Lambda is some kind of ephemeral invocation-based infrastructure. It's infrastructure the whole way down.
My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't. We use Cloudflare Workers fairly heavily, for example, to perform web redirects for some statically generated sites. It's a fairly sensical use-case for "serverless" in my opinion.
> But basically I think of a serverless function as like a one-shot container (...)
Implementation details don't really matter. The CloudFormation/CDK deployment of a lambda can include only the source code of the task you want to run. We don't know if AWS uses containers, an intern pressing enter, or purple unicorns.
> Therefore a Lambda is some kind of ephemeral invocation-based infrastructure.
Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
> My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't.
There is no magic or arcane wizardry involved. If you need to peel a background task out of a server that doesn't run that often and requires no internal data other than input parameters and takes far less than 15min to run, and you're already on AWS, then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
There's also the usecase for plugging together AWS events, such as processing a file that's uploaded to a S3 bucket.
If you are already neck deep in AWS, lambdas are not an arcane trick that only fit weird corner cases. You need to know what's the free tier, lambda pricing scheme, and do back of the napkin math to figure out where it makes sense to you to switch from lambdas to a dedicated service.
> Implementation details don't really matter. The CloudFormation/CDK deployment of a lambda can include only the source code of the task you want to run. We don't know if AWS uses containers, an intern pressing enter, or purple unicorns.
I'd argue the intern pressing enter at AWS is my infrastructure.
> Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
I feel like you took my pragmatic explanation and obfuscated it.
> There is no magic or arcane wizardry involved.
Never said there was, I think you think that I'm more confused than I am. I run my own serverless platform, and make use of serverless platforms from other companies. I know how they work.
> then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
You're trying to sell me on lambdas again. I said I already make use of serverless. I just am saying there is a time and place for them, and a time where they don't make any sense. You seem to be agreeing with me, where you placed a few qualifiers on when you should use a Lambda, so I don't know why you're framing this like we're in disagreement. Maybe you're confused as to what I actually said?
>But basically I think of a serverless function as like a one-shot container that is spun up when a webhook or some other invocation method is called, handles some kind of request, puts out some kind of response, and shuts down.
You’re right on the money but I think what you’re missing is people are using lambda functions not just to run some image resizing function, but as their entire backend.
> but I think what you’re missing is people are using lambda functions not just to run some image resizing function, but as their entire backend.
Sometimes it makes sense. I see people criticizing stuff they know nothing about.
I worked on a project where I had to put together a REST API for an internal project. After considering the traffic projections, we noticed a API Gateway + lambdas fitted quite neatly in the AWS free tier, and we did everything in lambdas. We also did a proof of concept in ECS and everything was far more complex and convoluted. With lambdas, the whole project consisted of a single CDK package which included also the nodejs code for the handlers.
In another project, we even skipped the API gateway and instead had a NLB invoke a couple of lambdas. It worked flawlessly, and in the free tier.
I'm seeing comments about the need to do engineering, but these same comments are outright clueless and fail to do the faintest cursory research. It's just people commenting on stuff they know nothing about with a demeaning tone. Not cool, not smart.
I guess a lot depends on your developer costs, a docker in e.g. GCPs cloud run costs peanuts compared to a devs salary per month.
IMO the less you need to code on top of your actual business logic functions - the better off you are for small projects.
Docker is still one of the best options, as if your tiny app is somehow business critical you can easily migrate it to other infra. With lambdas you’re stuck on AWS.
You're not getting stuck on AWS because you added a lambda to offload a background task, or because you implemented a low-traffic REST API with API Gateway. If you're rethinking where you deploy services, your decision won't factor in these irrelevant details. Odds are you are more concerned with S3 and CloudWatch than REST APIs.
> A lambda is just someone else's computer running your function, which you call through a fancy rpc interface. There is no infrastructure.
“someone else’s computer” _is_ infrastructure. _everything_ you run in cloud runs on someone else’s computer. Besides, you need to specify the amount of cpu and ram you need for your lambda, the kind of scaling you want, whether you can live with a cold start or not, optimise that cold start, potentially configure networking and so on. You can claim all you want, in the end it is an extra piece of infrastructure that you have to be aware of and configure.
> You don't change the way your system is deployed or is operating if you run your background task as a lambda.
Seriously? You do not need some kind of authentication to call your lambda? The lambda does not need access to some kind of private data to store its results? You won’t introduce a queue or something to offload background tasks?
I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it. And it makes me honestly wonder how much experience you have with lambdas or functions on production systems.
> "someone else’s computer” _is_ infrastructure. _everything_ you run in cloud runs on someone else’s computer.
That's meaningless pedantry. You don't talk nor care about infrastructure when you're adding a controller to one of your services, or when a container scales up.
The only thing that matters are the relevant attributes like performance and operational costs. You don't care if running a task in the background ends up scaling up your service, just like you don't care about that when running the same task in a function-as-a-service service like AWS Lambda. You only care if your system is stable and the invoice you get at the end of the month.
If your mental model cares about meaningless low-level details that do not factor in your operations, your mental model is brokenand you have no business lecturing others on how they are doing things wrong.
> Seriously? You do not need some kind of authentication to call your lambda?
You don't, if you configured them right.
> The lambda does not need access to some kind of private data to store its results?
What? It does not have any extra requirements than calling them from a service running on EC2, ECS, Fargate, EKS, whatever. It's an internal AWS service that, just like any AWS service, is covered by AWS' access management. In fact, arguably it has less requirements, specially considering least-privilege policies.
> I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it.
I argue that talking about infrastructure when talking about function-as-a-service is ignorant and naive, and reflects lack of insight and first-hand experience which materializes in a mental model that's fundamentally broken.
And people suffering from this problem should not be making bold statements about something they know nothing about.
> Some horrific monstrosity of step functions, lambdas, and glue. All because they've been told that serverless is the correct way to build "scalable" software.
If it’s a horrific monstrosity it’s either because you don’t understand it or it was poorly done.
Seen several .NET step function implementations which were clean and easy to understand. And we’re cheaper than what they replaced. And new devs could easily pick it up and understand it.
> I regularly review design from teams that turn reading a file from S3 and putting into into a database into, well, something that requires a design review. Some horrific monstrosity of step functions, lambdas, and glue. All because they've been told that serverless is the correct way to build "scalable" software.
This is typically true with a lot of ClickOps teams.
I use Lambda for personal projects because at that scale it's free. OTOH I can't find a job and when I'm lucky enough to get detailed feedback it's always "you don't have a significant cloud experience"
Once had a coworker advocate for microservices because "you split everything into separate services and they all just work" quote, unquote. I've wondered ever since how typical that mindset might be whenever I hear anyone else pushing for similar such solutions.
The biggest benefit to microservices to many people is the fact that everyone else is also doing microservices.
Externally mediated taste is how a lot of people operate. And they may have a vague idea that they should not defer as much to it, but they don’t have the experience to take the risk. So they double down on consensus.
"Developers are losing touch on what process boundaries mean,"
I feel like a lot of the problems are from misused Agile. Devs are building systems that haven't fully been thought out. I don't know when I've seen a fully built out process map in the past 5 years.
MVP1 is easily built as severless, but then what happens when the full implementation gets more complicated? Do we rewrite all the server less architecture as a monolith? Nah, just tack on some more Lambdas. No way can we slow down for a refactor, and all the original devs that wrote it have job hopped.
No way can we slow down for a refactor, and all the original devs that wrote it have job hopped.
The most likely way to escape the current madness might be for a few companies with more enlightened management to actively look for people who have stayed in past roles for long enough to have real experience not just building but also maintaining software and pay them more. Same with loyalty incentives to retain existing staff. And then make sure those more experienced developers are choosing tools built to do a good job and not just tools built to make naive resume scanners happy.
If frequent job-hopping and leaving any junk you've just built to become someone else's problem stops being the fastest way to boost your career and compensation then good people and good employers will start to prioritise building stuff properly and looking after it for the long term and almost everyone will be better off.
In senior roles I’ll only hire people that have had some longer stints for exactly these reasons.
If you’ve never worked anywhere long enough to have to live with the fallout and consequences of your decisions, you’re not senior in my mind. You’ve never received useful real world feedback to be given a chance to refine your approach.
Exactly. I've encountered a small number of people who apparently reached very senior technical positions impressively early in their career but when you looked at their job history it was mostly a string of 1-2 year stays with a title bump on each jump. SE, Senior, Lead, random nine months as a cofounder of something that went nowhere but says CTO on the resume, Staff, and now after less than a decade they're running the show somewhere with a silly name but a few million in funding. In all that time they might never have worked on a single product with sustainable revenue or been responsible for the consequences of any technical decision they made as much as one year after they made it. Maybe two of those people that I can think of weren't bad anyway - sometimes their inexperience showed but at least they weren't crazy and had open minds - but others were among the most over-rated and damaging people I've ever seen in the business.
I wouldn’t put this as an absolute truth, over the last decade or so I’ve spent most of my time un-fucking projects that haven fucked by people making bad decision. Overengineering - nonexistent senior coders - poor infrastructure - I’ve seen the lot.
My experience is that once the system is built, they outsource the maintenance, or they stack another project on your team (over time it adds up, I've been on a team supporting about 15 apps).
Have you even read the article? Leave it to HN to spam the exact same message in every thread where the word severless is mentioned or hinted. This article is nuanced as was original prime video team claim.
While I do think lambda is sometimes overused, but no good engineer says "cuz serverless" for using lambda function as is implied in all these conversation. And there is no best choice in the absolute sense, and I wouldn't want to hire someone who claims that monolith is the better architecture for everything or everything should be serverless.
IMO that's the difference in calling yourself an engineer. Weighing resources and tradeoffs is engineering 101. Maybe the scope of the problem doesn't require due diligence for budget constraints, and that's fine for those problems. But if you don't have a rough idea of this then you're just not practicing engineering.
It's not even just Lambda, I have mentioned it here before but I've inherited a CRUD application where the read is a separate "application" and because of this I have two databases with minimal differences with multiple applications that are basically the same.
On the other hand, I've decreased costs significantly in previous positions by moving the cart functions to a lambda because it made it highly responsive, scaled up to thousands of transactions a second without concern during initial sales of certain fruit based devices which generate a rush, but basically sit idle 90% of the rest of the time.
I am not concerned with "the death of serverless", there is a time for lambda, and a time for long running processes. You can stick multiple rest endpoints in a single lambda, and that's the recommended thing to do. Moving from micro to monolith isn't the death of serverless, people not knowing when the appropriate time to use different techniques will be.
> Moving from micro to monolith isn't the death of serverless, people not knowing when the appropriate time to use different techniques will be.
Exactly, people should take the time to study different techniques, learn about their advantages and disadvantages, and then use some basic common sense when choosing a specific technique in a specific context.
Yep, you're essentially writing a module that's going to be imported by a server someone else wrote. The server will call the appropriate functions in your module based on the requests it receives, and Amazon gets a bigger check at the end of the month.
I think there are probably reasonable use cases for this, like automation that's going to run infrequently, but it seems like a terrible way to implement a typical application that's going to receive frequent requests.
> I think there are probably reasonable use cases for this, like automation that's going to run infrequently
Exactly our usecase. Then the network admins wanted to click button and select their template from lists rather than having to write json, so website it was, and now we run a MVC on lambda.
It isn't used that much (at most, you'll get 20 simulaneous users, and its always during a short time period, most of the time nobody use it). Don't know if it's the best choice. Honestly it seems fine.
Functions-as-a-Service definitely makes more sense. However, now that we're on the subject, how the hell did we end up with Cloud to mean Infrastructure-as-a-Service?
I'd say it's more like "framework as a service", because that's what they're doing for you: they provide the framework and you just write the request handlers.
Thanks for re-sharing. I was quite disappointed to see the negative community reaction to the original Amazon article. I felt that those knocking it didn't analyze the circumstances that lead to the Serverless Architecture's gaps and just saw some problems with it and assumed "serverless bad".
In reality, the series of decisions made a lot of sense. Take something that exists, try using it, learn from it, and make better decisions as a result of it. It just so happened that in this case, serverless architecture was the wrong tool for the job and dedicated machines were a better fit.
Anyways, happy to see my take being shared. If you enjoy this kind of content, I have a newsletter and Youtube channel where I discuss / make videos about more AWS & cloud concepts:
I think serverless could be the right fit, something like detectors seems ideal. Theyre somewhat isolated individually, etc etc.
I kind of want to take a Crack at what Prime Video did except serverless from a green field perspective, and I'm willing to bet I wind up provisioning a RedPanda cluster or similar and having them feed detector lambdas. The bottleneck and cost here appeared to be storing intermediate results and the orchestration. Orchestration is solved by using something like SQS or what have you. Intermediate results are somewhat harder depending on size - but I think you could get there and it'd be an interesting exercise.
But isn't this an example of even AWS developers using cloud services wrong?
They used a serverless approach (which should have excellent scalability -- it's one of the most common marketing lines of it!) then had to switch architecture because they realised it wasn't scalable.
> Secondly, the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered.
> Finally, I think the PV team made all the right moves by switching to provisioned infratructure. They identified an existing tool that could have been a potential solution. They attempted to leverage it before realizing it had some real scaling / cost bottlenecks. After assessing the situation, they pivoted and re-designed for a more robust solution that happened to use provisioned infrastructure. This is what the software development process is all about.
I was downvoted this morning for suggestion that developers need to focus on objectively measurable results and stop insisting on using their favorite language or framework or paradigm. I guess things are much worse than I thought, because the headline here simply assumes as a given that "serverless lovers" are a thing that exists and are worth taking seriously.
When it comes to complex flows with many branches serverless really shines. It makes it easier to manage such complexity. For simpler flows serverless might just be an overkill.
It makes it a lot clearer where logs, alerts, etc are coming from and it makes them easier to test - because the process flow of a given instruction is in the architecture.
Verifying that the call patterns between the functions are correct is challenging sometimes - but it helps there to have the architecture mapped out visually.
During my 3 years as a backend developer at my current job I've seen a lot of application of "serverless" stuff from AWS, and I've also used quite a lot of those services myself.
Personally I would never use something like Lambda for compute, unless it's something that needs to integrate between another AWS service and something else (likely also a service.)
Also it seems like the "serverless" hype started (atleast at my employer), because some higher ups didn't trust the developers in my department, when they first had an AWS account provisioned.
My general feeling about picking the right tool to solve a problem is to, firstly, pick the problem apart. Then you can evaluate technologies that can solve your problem. This includes code that you already own and it should most likely be this one that you pick. After that you can start to look at other technologies to bring in.
Arguably it's nice that with AWS you can spin something up relatively quickly, however sometimes I'm also stumped by their limitations and what workarounds that seems to pop up as a result of that.
The big place it shines for compute tasks is workloads with high variability.
As long as you can select a p99 for timeouts that's sane, lambda does a really good job handling highly heterogeneous workflows in general.
So if a request could take 10ms or multiple minutes depending on Reasons(TM) - lambda makes a lot of sense because you don't have to overprovision for theoretical peak demand, and a swamp of expensive requests is isolated from your cheap requests.
Yes, serverless lovers should be worried. They should have always been worried.
About just how much money their ideas have thrown at $CLOUD that could've been used for things other than helping build Bezos' Freudian Me-Too Ego Trip What Go Zoom.
Or how much fragility and painfully slow development cycles have been inflicted on people in their org by the inevitable tight coupling to $SPECIFIC_PRODUCT_IN_SPECIFIC_REGION_OF_SPECIFIC_PROVIDER.
And, they should be worried that they've drunk so deeply of the Kool Aid provided at $VENDOR_CONFERENCE_PROBABLY_IN_VEGAS that they considered the cost and the friction a worthwhile trade-off.
I feel like the hype caught up a lot of folks off guard and swayed them to do something that doesn't necessarily make sense. For example, building a simple web app (single page app, or whatever) using the hype around microservices, serverless, nosql, etc. is bound to fail.
Fortunately, I see a trend in more "advanced" companies to stay away from microservices/lambdas in the first iterations of their products and embrace good old modular monolith with a simple architecture.
Serverless is great for event handling and bursty workloads. It is not efficient for a long running or constantly running processes. That was never what it was designed for.
ECS on Fargate is still serverless. It seems that people think only Lambda is serverless, when serverless just means "product category where you pay for the usage directly, instead of for the resources provisioned to serve it".
Lambda is a burstable, short-lived compute service; it's only natural that it's not well suited to situations where you have predictably sustained high throughput and homogeneous task requirements. To not even talk about how dumb the original architecture was for shuffling frames in and out of S3.
99 comments
[ 2.4 ms ] story [ 189 ms ] thread>the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered.
Lesson learned: ensure your design is sane before turning your PoC into a full blown production system.
Developers are losing touch on what process boundaries mean, and the result is brittle and poor performing systems.
If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing.
Cloud providers are surely pleased as punch that there is now an entire generation of developers that will argue vehemently that infrastructure that only grows over time is not only healthy, but the best way to build scalable software.
I have no idea what to call it other than "Marketing Material Driven Development".
And on the flipside - One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
I sometimes worry for the future of our industry because of exactly this. The culture of buzzword driven development has become enshrined in the recruitment process - often from both sides - because of this kind of vicious circle effect. Now anyone - again from either side - who tries to escape it and instead advocates using sensible, robust tech to build useful, good quality products risks becoming a pariah and seriously damaging their career as a candidate or their ability to recruit strong candidates as an employer. Of course this assumes we are judging career progress by level of seniority reached or compensation paid instead of more indirect achievements like producing something that provided real value to those who used it. But if we're being honest most people obviously will.
In one interview the engineer interviewing me explained how they didn’t have any paying customers, and then went on to describe an elaborate over-engineered micro-service based architecture he was working on.
The real kicker was that instead of asking me about my 20 years building products, or our infrastructure at my last gig that handled millions of customers across the globe, he instead wanted to watch me code some goofy coding quiz he had prepared.
It's horseshit but at least half of the people in charge of hiring think exactly that way and so I end up leaving the work I'm the most proud of off my resume a lot of the time.
I could see that. Lambda has a 15 minute timeout, and if you’re processing something large you want an orchestrator that can manage retries. I don’t like the SFN developer experience, but it’s only like 15% worse than boto3.
Cloud providers don’t care about your serverless function utilization for 95% of people. A huge chunk of people will never pay more than a single digit serverless bill, regardless of if they have 450 lambdas or 40. If creating a new function matches your mental model of how things works, simplifies ops and log reading, or gives you access to a library in a different language, I’d say you made a great call.
Like many of the problems (security patching, monitoring, logging) are part of the serverless pitch.
I have to call out your take because it's either outright wrong or is overgeneralizing, to put it politely.
A lambda is just someone else's computer running your function, which you call through a fancy rpc interface. There is no infrastructure.
Just because you can and should deploy a lambda with infrastructure-as-code tools such as CloudFormation or CDK, it still does not make a lambda "infrastructure". It's just your function running on space CPU cycles of someone else's computer, which you call through RPC.
Running your lambda as a background task in someone else's computer does not make it infrastructure, just like running code in your own server is not adding infrastructure.
Also, it seems you're missing the whole point of offloading background processes out of your servers. If they are computationally demanding and your servers are well provisioned them a background task can lead your system to scale up. You don't change the way your system is deployed or is operating if you run your background task as a lambda. You make a request and things run just the same. Zero potential for alarms and being paged in the middle of the night.
And more importantly, for some usecases lambdas are entirely free. Does this have no value to you?
Some people feel compelled to criticize everyone around them just to feel self important. A little humility would serve them well. Just because you fail to understand something's traits and properties that does not mean everyone else around you is wrong and you are the only enlightened soul who is right.
My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't. We use Cloudflare Workers fairly heavily, for example, to perform web redirects for some statically generated sites. It's a fairly sensical use-case for "serverless" in my opinion.
Implementation details don't really matter. The CloudFormation/CDK deployment of a lambda can include only the source code of the task you want to run. We don't know if AWS uses containers, an intern pressing enter, or purple unicorns.
> Therefore a Lambda is some kind of ephemeral invocation-based infrastructure.
Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
> My personal take here is just that serverless functions have a time and a place in one's infrastructure, and it requires good engineering to determine where that time or place is or isn't.
There is no magic or arcane wizardry involved. If you need to peel a background task out of a server that doesn't run that often and requires no internal data other than input parameters and takes far less than 15min to run, and you're already on AWS, then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
There's also the usecase for plugging together AWS events, such as processing a file that's uploaded to a S3 bucket.
If you are already neck deep in AWS, lambdas are not an arcane trick that only fit weird corner cases. You need to know what's the free tier, lambda pricing scheme, and do back of the napkin math to figure out where it makes sense to you to switch from lambdas to a dedicated service.
I'd argue the intern pressing enter at AWS is my infrastructure.
> Not really. A lambda is your function running somewhere. Unless you explicitly create your lambdas as runnable containers instead of simply passing your handler's code, you don't know or care how are things under the hood.
I feel like you took my pragmatic explanation and obfuscated it.
> There is no magic or arcane wizardry involved.
Never said there was, I think you think that I'm more confused than I am. I run my own serverless platform, and make use of serverless platforms from other companies. I know how they work.
> then lambdas are far easier to implement and to run and to operate than modifying some server deployment to include support for a background task. In some cases, they can also be free to run.
You're trying to sell me on lambdas again. I said I already make use of serverless. I just am saying there is a time and place for them, and a time where they don't make any sense. You seem to be agreeing with me, where you placed a few qualifiers on when you should use a Lambda, so I don't know why you're framing this like we're in disagreement. Maybe you're confused as to what I actually said?
we used to call those "cgi scripts"
That’s the part that’s being critiqued.
Sometimes it makes sense. I see people criticizing stuff they know nothing about.
I worked on a project where I had to put together a REST API for an internal project. After considering the traffic projections, we noticed a API Gateway + lambdas fitted quite neatly in the AWS free tier, and we did everything in lambdas. We also did a proof of concept in ECS and everything was far more complex and convoluted. With lambdas, the whole project consisted of a single CDK package which included also the nodejs code for the handlers.
In another project, we even skipped the API gateway and instead had a NLB invoke a couple of lambdas. It worked flawlessly, and in the free tier.
I'm seeing comments about the need to do engineering, but these same comments are outright clueless and fail to do the faintest cursory research. It's just people commenting on stuff they know nothing about with a demeaning tone. Not cool, not smart.
IMO the less you need to code on top of your actual business logic functions - the better off you are for small projects.
Docker is still one of the best options, as if your tiny app is somehow business critical you can easily migrate it to other infra. With lambdas you’re stuck on AWS.
You're not getting stuck on AWS because you added a lambda to offload a background task, or because you implemented a low-traffic REST API with API Gateway. If you're rethinking where you deploy services, your decision won't factor in these irrelevant details. Odds are you are more concerned with S3 and CloudWatch than REST APIs.
“someone else’s computer” _is_ infrastructure. _everything_ you run in cloud runs on someone else’s computer. Besides, you need to specify the amount of cpu and ram you need for your lambda, the kind of scaling you want, whether you can live with a cold start or not, optimise that cold start, potentially configure networking and so on. You can claim all you want, in the end it is an extra piece of infrastructure that you have to be aware of and configure.
> You don't change the way your system is deployed or is operating if you run your background task as a lambda.
Seriously? You do not need some kind of authentication to call your lambda? The lambda does not need access to some kind of private data to store its results? You won’t introduce a queue or something to offload background tasks?
I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it. And it makes me honestly wonder how much experience you have with lambdas or functions on production systems.
That's meaningless pedantry. You don't talk nor care about infrastructure when you're adding a controller to one of your services, or when a container scales up.
The only thing that matters are the relevant attributes like performance and operational costs. You don't care if running a task in the background ends up scaling up your service, just like you don't care about that when running the same task in a function-as-a-service service like AWS Lambda. You only care if your system is stable and the invoice you get at the end of the month.
If your mental model cares about meaningless low-level details that do not factor in your operations, your mental model is brokenand you have no business lecturing others on how they are doing things wrong.
> Seriously? You do not need some kind of authentication to call your lambda?
You don't, if you configured them right.
> The lambda does not need access to some kind of private data to store its results?
What? It does not have any extra requirements than calling them from a service running on EC2, ECS, Fargate, EKS, whatever. It's an internal AWS service that, just like any AWS service, is covered by AWS' access management. In fact, arguably it has less requirements, specially considering least-privilege policies.
> I think it is naive to state that introducing lambdas into the architecture of an application does not have an impact on the infrastructure you need for it.
I argue that talking about infrastructure when talking about function-as-a-service is ignorant and naive, and reflects lack of insight and first-hand experience which materializes in a mental model that's fundamentally broken.
And people suffering from this problem should not be making bold statements about something they know nothing about.
If it’s a horrific monstrosity it’s either because you don’t understand it or it was poorly done.
Seen several .NET step function implementations which were clean and easy to understand. And we’re cheaper than what they replaced. And new devs could easily pick it up and understand it.
This is typically true with a lot of ClickOps teams.
Externally mediated taste is how a lot of people operate. And they may have a vague idea that they should not defer as much to it, but they don’t have the experience to take the risk. So they double down on consensus.
I feel like a lot of the problems are from misused Agile. Devs are building systems that haven't fully been thought out. I don't know when I've seen a fully built out process map in the past 5 years.
MVP1 is easily built as severless, but then what happens when the full implementation gets more complicated? Do we rewrite all the server less architecture as a monolith? Nah, just tack on some more Lambdas. No way can we slow down for a refactor, and all the original devs that wrote it have job hopped.
The most likely way to escape the current madness might be for a few companies with more enlightened management to actively look for people who have stayed in past roles for long enough to have real experience not just building but also maintaining software and pay them more. Same with loyalty incentives to retain existing staff. And then make sure those more experienced developers are choosing tools built to do a good job and not just tools built to make naive resume scanners happy.
If frequent job-hopping and leaving any junk you've just built to become someone else's problem stops being the fastest way to boost your career and compensation then good people and good employers will start to prioritise building stuff properly and looking after it for the long term and almost everyone will be better off.
If you’ve never worked anywhere long enough to have to live with the fallout and consequences of your decisions, you’re not senior in my mind. You’ve never received useful real world feedback to be given a chance to refine your approach.
While I do think lambda is sometimes overused, but no good engineer says "cuz serverless" for using lambda function as is implied in all these conversation. And there is no best choice in the absolute sense, and I wouldn't want to hire someone who claims that monolith is the better architecture for everything or everything should be serverless.
The issue is using them poorly, and in particular stringing them together into a sort of pipeline.
On the other hand, I've decreased costs significantly in previous positions by moving the cart functions to a lambda because it made it highly responsive, scaled up to thousands of transactions a second without concern during initial sales of certain fruit based devices which generate a rush, but basically sit idle 90% of the rest of the time.
I am not concerned with "the death of serverless", there is a time for lambda, and a time for long running processes. You can stick multiple rest endpoints in a single lambda, and that's the recommended thing to do. Moving from micro to monolith isn't the death of serverless, people not knowing when the appropriate time to use different techniques will be.
Exactly, people should take the time to study different techniques, learn about their advantages and disadvantages, and then use some basic common sense when choosing a specific technique in a specific context.
https://news.ycombinator.com/item?id=35853148
I was always vaguely irritated by the term and never understood what it meant bc I was like „there must be some server??“
(not working in web dev)
I think there are probably reasonable use cases for this, like automation that's going to run infrequently, but it seems like a terrible way to implement a typical application that's going to receive frequent requests.
Or no check at all since it scales to zero and the free tier is ridiculously high for infrequent / low usage.
PoC works, costs nothing, code delivered much faster. CTO impressed.
PoC tidied up and rolled out to production. CTO very not impressed when quarterly expenses are reviewed, but customers are relying on this now.
Exactly our usecase. Then the network admins wanted to click button and select their template from lists rather than having to write json, so website it was, and now we run a MVC on lambda.
It isn't used that much (at most, you'll get 20 simulaneous users, and its always during a short time period, most of the time nobody use it). Don't know if it's the best choice. Honestly it seems fine.
But we obviously haven’t.
Which, despite not sharing your opinion, probably indicates the name was poorly chosen. Since it still has be defended.
Or perhaps it has just turned into one of those gnome v kde / vim v emacs things?
Thanks for re-sharing. I was quite disappointed to see the negative community reaction to the original Amazon article. I felt that those knocking it didn't analyze the circumstances that lead to the Serverless Architecture's gaps and just saw some problems with it and assumed "serverless bad".
In reality, the series of decisions made a lot of sense. Take something that exists, try using it, learn from it, and make better decisions as a result of it. It just so happened that in this case, serverless architecture was the wrong tool for the job and dedicated machines were a better fit.
Anyways, happy to see my take being shared. If you enjoy this kind of content, I have a newsletter and Youtube channel where I discuss / make videos about more AWS & cloud concepts:
Newsletter: https://mailinglist.beabetterdev.com Youtube: https://www.youtube.com/c/BeABetterDev
Cheers,
Daniel
I kind of want to take a Crack at what Prime Video did except serverless from a green field perspective, and I'm willing to bet I wind up provisioning a RedPanda cluster or similar and having them feed detector lambdas. The bottleneck and cost here appeared to be storing intermediate results and the orchestration. Orchestration is solved by using something like SQS or what have you. Intermediate results are somewhat harder depending on size - but I think you could get there and it'd be an interesting exercise.
They used a serverless approach (which should have excellent scalability -- it's one of the most common marketing lines of it!) then had to switch architecture because they realised it wasn't scalable.
Did you read the old article or this one?
> Secondly, the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered.
> Finally, I think the PV team made all the right moves by switching to provisioned infratructure. They identified an existing tool that could have been a potential solution. They attempted to leverage it before realizing it had some real scaling / cost bottlenecks. After assessing the situation, they pivoted and re-designed for a more robust solution that happened to use provisioned infrastructure. This is what the software development process is all about.
In short, there is a place for both.
Verifying that the call patterns between the functions are correct is challenging sometimes - but it helps there to have the architecture mapped out visually.
Personally I would never use something like Lambda for compute, unless it's something that needs to integrate between another AWS service and something else (likely also a service.)
Also it seems like the "serverless" hype started (atleast at my employer), because some higher ups didn't trust the developers in my department, when they first had an AWS account provisioned.
My general feeling about picking the right tool to solve a problem is to, firstly, pick the problem apart. Then you can evaluate technologies that can solve your problem. This includes code that you already own and it should most likely be this one that you pick. After that you can start to look at other technologies to bring in.
Arguably it's nice that with AWS you can spin something up relatively quickly, however sometimes I'm also stumped by their limitations and what workarounds that seems to pop up as a result of that.
As long as you can select a p99 for timeouts that's sane, lambda does a really good job handling highly heterogeneous workflows in general.
So if a request could take 10ms or multiple minutes depending on Reasons(TM) - lambda makes a lot of sense because you don't have to overprovision for theoretical peak demand, and a swamp of expensive requests is isolated from your cheap requests.
I cant figure out what that refers to. From a user perspective there is only 1 channel (maybe one per region, but that still doesn’t hit 100s)
And for streams I can’t think of any kind of definition of stream that wouldn’t have orders of magnitude larger number associated with it on prime.
About just how much money their ideas have thrown at $CLOUD that could've been used for things other than helping build Bezos' Freudian Me-Too Ego Trip What Go Zoom.
Or how much fragility and painfully slow development cycles have been inflicted on people in their org by the inevitable tight coupling to $SPECIFIC_PRODUCT_IN_SPECIFIC_REGION_OF_SPECIFIC_PROVIDER.
And, they should be worried that they've drunk so deeply of the Kool Aid provided at $VENDOR_CONFERENCE_PROBABLY_IN_VEGAS that they considered the cost and the friction a worthwhile trade-off.
A fast modern computer will give a whole lotta vertical scaling.
Even more so if you use a compiled language.
I wrote a web application with rust using actix that did about 7 million requests per second.
Fortunately, I see a trend in more "advanced" companies to stay away from microservices/lambdas in the first iterations of their products and embrace good old modular monolith with a simple architecture.
Serverless is great for event handling and bursty workloads. It is not efficient for a long running or constantly running processes. That was never what it was designed for.
There is a time and a place for serverless.