AWS Lambda can run all programming languages via layers.
Cloudflare Workers can run all programming languages that compile to WebAssembly.
"Vendor Lock"
True, but being locked into Kubetnetes isn't a cakewalk either.
"Performance"
Cold-starts aren't a thing for Cloudflare Workers and can be mitigated for AWS Lambda. AppSync and API Gateway don't even have them if you directly integrate with AWS services.
"You Can't Run Entire Applications"
You can and it has been done multiple times. Sure, you shouldn't blindly build everything with Lambda and API-Gateway without measuring. But many services, especially Cloudflare Workers are quite cheap.
K8s can go almost anywhere any
time with fairly minimal ramp. With the frequency I am hearing about AWS billing disputes these days it's probably best to not seal the keys to your kingdom in their service.
But how many hours/days/years did it take you to learn how all the parts move before you became capable of setting something like that up within a few hours?
Anecdotally, stories of businesses abandoning entire cloud automation projects because they wasted weeks and never had anything to show from it don't seem unusual, so evidently a considerable amount of knowledge and skill is required to get value out of these tools.
My worry is less about getting it set up to run a container and more about securing it, monitoring it and being able to respond should things go wrong.
> AppSync and API Gateway don't even have them if you directly integrate with AWS services.
I was quite impressed when I exposed Kinesis via API Gateway, it just worked, although the Kinesis requirement that data be b64 encoded did require some fiddling about with mapping templates.
People always think serverless mean FaaS, while in fact you should use FaaS only to glue together other serverless services when it's not otherwise possible.
Sure, a container is better than a serverless function in many ways, but the function isn't what the container is competing with.
> I'm not against serverless solutions per se. I promise. It's just that developers should realize – especially if they are exploring serverless models for the first time – that this technology is not a straight replacement for servers.
Ha. So the name didn't give that away then?
In all seriousness though, I think another avenue for serverless to still explore seriously is declarative application definitions rather than just functions/runtime as a service. Hasura[0] are doing this with GraphQL but I think we'll soon seen (are already seeing?) a surge in these kinds of cloud abstractions that do much more for you than a simple on-demand runtime ever could.
Serverless is the new microservice. Fortunately, for the most part it’s fallen flat on its face and for good reason - most companies don’t need it: it adds unnecessary complexity for minimal gains.
They are helpful in probably 10% of cases where they are indeed the right tool for the job and the benefits outweigh the downsides.
For the majority of companies out there it's just a tool to overcomplicate your stack and turn it into an engineering playground so you can justify 2-3x the headcount despite no significant productivity increase. But hey, at least your company can now be giving talks about how they solve their (self-inflicted) problems managing all the microservices and throw that buzzword on the careers page.
People on HN seems to believe that outside of Google it is impossible to have problems that are complex and high scale, therefor if you use microservices and/or serverless and/or nosql etc etc it's just that you're following the hype, not that you legitimately benefit from them.
I think people are jaded because a lot of these use cases stem from ignorance and/or incompetence of the underlying tech. The majority of people arguing the benefits of NoSQL aren’t professional DBA’s, they’re people that don’t know how to use EXPLAIN. The majority of people touting the “infinite scale” of serverless haven’t even benchmarked their server, never mind optimised it.
Prove to me that what you’re trying to do can’t be solved with a Python script, Postgres and a beefy machine. I don’t care about the theoretical benefits of whatever you’re pushing, I want hard numbers. Unfortunately the community has spent quite a lot of its time and effort these last few years making it easy to unnecessarily scale horizontally. I want no part of that.
I think people simultaneously underestimate how many jobs out there are extremely low volume, and also underestimate how many are high volume. I'm in a data analytics space, every product in my space is talking about 10's or 100's of terabytes a day per customer, and running many different forms of analytics on that data as quickly as possible.
It's always so strange, being in this space where everyone is dealing with tons of data, and hearing constantly on HN about how "you don't need this level of scale unless you're Google".
Complexity always exists. Sometimes it just exists in the interactions between what you think of as your application pieces (most often the case with microservices and lambda architectures).
That being said, it's always possible to add a bit more complexity if you're not careful.
It‘s not a new microservice, it’s a new name of grid computing with more than 20 years of history. I have been working with a grid in ~2001, running Java-based GUI apps on thin clients from a distributed cluster of several sparc machines. Was an interesting concept with unclear economics (likely useless with the internet connection speeds of that time). It’s not difficult to imagine it as an alternative to GSuite with much less resources required on the client.
From an evolutionary point of view, I find serverless appealing and exciting - a romantic idea.
However, the frequent posts along the lines of "my Lambda function was buggy and now I owe Amazon three billion dollars" definitely kills the romanticism.
Isn't there some sort of security built-in, like "if suddenly the bill becomes 1500% the norm, kill the instance and start serving 404s"? (I know nothing about cloud, honestly curious)
It's in cloud vendor's interests to charge more money regardless of mistakes. They do offer charge limits, but they don't want you to use them.
There are lots of posts online that document "talk to amazon and they'll take out the extra charges" but there are even more cases where the people just pay the bill and thus amazon makes money and profit.
No. Amazon very much prefers uptime over cost cutting; they rather have your website up than read a story about a bill limiter taking you offline. They also tend to be quite lenient on unintended bills.
Additionally, bill cutting is not that easy. Should they shut down current services? Stop sending in the middle of a newsletter? Delete your S3 Storage? Even then, a large company might accumulate hundreds or thousands of dollars within a second ; even if they decide to cut at all cost, minimal delays might break the limit.
I fully agree that it sucks for experimenting; it is very much the reason I don't have an AWS account. But I can see why they don't have it.
Also, Amazon bill’s monthly. They might be able to charge by the second, but they don’t actually have an accurate to the second running count of your total bill.
Nope. As the GP mentioned, billing is delayed, but you could set alarms on your lambda usage. You could even use something like X-Ray to get more insight into what your usage looks like.
The problem is there's not a clear way to tell the difference between recursive calls and heavy traffic.
AWS components are fundamentally a network of nodes that are sending traffic to other nodes. But there's no unifying language that can model this, so you can't describe what this graph ought to look like. (e.g. you want to be able to say "lambda Foo should be called twice for each event to Bar".)
And that'd be hard to do since most nodes are general purpose computers, like Lambda and EC2, that can send traffic anywhere for arbitrarily complex reasons.
Since you can't describe what it ought to look like, trying to alarm on it becomes a problem of detecting anomalies. You can sort of do it, but AWS doesn't want to do it because they'd be making a promise they couldn't generally keep.
I never quite understood what problem serverless platforms were solving for companies that already deployed their own SAAS solution.
As the article says, serverless is one of many (many) ways to wrap a quantum of functionality inside an internet-accessible environment. You could have a chunk of python in a serverless setup, a small flask server in a container in k8s, as an endpoint in a monolith, etc. Each of these environments have difference performance, cost and maintenance characteristics, but I think of the star feature of serverless as the light deploy. For companies that don't have a deploy chain, that makes serverless very attractive, but if you've already invested in one it seems less compelling.
I think serverless products have a ton of potential for side projects and hobby projects. It's great that you can just type out python and access it on the internet immediately! That quality just never seemed to be what was holding back SAAS companies.
I’d be curious to see the data behind that claim that the hobbyist market is overwhelmingly PHP. I must not have any exposure to that ecosystem because PHP does not come to mind when I think of hobbyist development.
Not natively, but fairly easily. We’re using Laravel Vapor - it handles all the setup and deployment of Lambda bundles that include the app code and a PHP runtime. Smooth and fast so far.
Yeah; I'd imagine anyone technically savvy enough to use Lambda + API Gateway, in any form, is savvy enough to bundle PHP if they really want to use it. And if they're not savvy enough to do that, they're probably using hosted Wordpress somewhere and it's a moot point.
Another niche where it shines is the small, low-volume tool. You don't want to give it an entire VM, because that's a major waste. You don't want to have it share one "tools box," because it is awkward sharing like that, security becomes more of an issue, and it generally devolves into a mess. So if you've got other things running on AWS, then AWS Lambda is great for this. Do you need a random web-facing snippet to receive a webhook a few times a day and send a Slack message to your team? Perfect.
Not the core of your app, but a tool of great utility. On the other hand, this isn't exactly the Serverless Revolution either.
I'd add on top of that, that it provides even more value when volume is usually low and sparse, but unpredictable.
If you have a server that can handle up to 100 requests at a time, but you're only getting one or two a day, you could probably save money by switching to serverless. On the flip side though, you're also a bit screwed if 1000 requests all come in at once, since, even if you have some autoscaling solution, it probably won't be able to bring new servers up in time. Serverless provides a solution for that case as well, since you have almost unlimited resources.
But yeah, serverless is a tool for solving a certain set of problems. This idea of the "Serverless Revolution" was kind of silly from the start
All the platforms (GCP, Lambda, Azure Functions) have fairly long cold start times, but some platforms let you essentially pay for one or more cold start instances that are always on (I saw Azure had this, others are sure to follow). According to the article this "undermines" the model, but I disagree, as instance start time measured in the seconds is crazy difficult to achieve when doing VM automation. Typically it'll take 5m to get a fresh VM into the load balancer during scaling, could be as low as 1-2m if you do it just right. Buying one or two instances and then having the rest of the requests billed per invocation with a fast scale up time is very appealing.
We happily connection pool using sequelize by leveraging the fact that lambdas run in reusable containers.
If the next lambda invocation happens within minutes of the previous one ending, you can carry forward a db connection from the older lambda, no magic needed. Just place your db connection object in global scope (nodejs).
Yeah, they can use the same connection for a while. The problem is that 1000 lambdas cannot use 50 connections while waiting their turn. Each of them wants a connection to the DB server.
Now if you somehow get a spike of requests to some endpoint that goes over your DB connection limit, suddenly all your newly scaled functions fail because they cannot get a connection to the database.
AWS added a service for RDS to deal with this, but it all just feels like a big kludge to deal with a problem that shouldn’t have to exist in the first place.
If you are expecting one thousand simultaneous requests, serverless or not, you will need a db and an appserver which support one thousand simultaneous connections. There is no getting around that.
You will have to analyze your burst rate and the lifetime of each request to figure out the size of your connection pool.
I also am not clear how AWS introduced the problem of not being able to connect to the DB. We've known about the need for connection pools and connection reuse from before AWS was a thing, no?
If your function spends 80% of its time on logic and 20% on database time, with a server app, you can get by with 200 DB connections to handle that throughout, but will need 1000 connections in a serverless setup.
I don't think connection pooling works this way. Happy to be corrected.
A process is given a connection from a pool when it requests the connection and holds it until it relinquishes the connection regardless of whether the process was in the logic or data access portions of their lifecycle. It is up to you as the developer to ask for the connection and to return it on an as-needed basis...Which is exactly what you should be doing in a lambda as well.
Just because a lambda container is persistent does not mean the connection given to it is stuck to that lambda even after that lambda returns the connection.
Lambda does hold onto it in order improve response time. It connects to the database in a handler outside the context of a request, so you do hold onto N connections persistently (at least until the lambda scales down).
With a connection pool, you don’t need to hold onto a connection longer than necessary, but you are right that you could implement it in such a way that you hold it for life of the request.
I've never understood why people who would never dream of using a single-instance application server are ok with using a single-instance datastore. Of course your datastore needs to autoscale just like your application does. Duh.
That only works if you’re relatively latency tolerant. Serverless systems currently suffer badly from cold start issues, particularly when large spikes arise. This can be solved, but it involves enough silliness that paying for a VM is probably cheaper once you factor in engineering time.
> Do you need a random web-facing snippet to receive a webhook a few times a day and send a Slack message to your team? Perfect.
How many people have that use-case but not in multiples? How many of those people have such a use-case but would not be better served by platforms such as IFTTT or webhooks on other services?
Exactly our use case, esp. for things that are outside of k8s, for example, standard tasks that every cloud account of ours needs to perform (they are heavily segmented due to regulations) and developers or k8s ops do not get to touch. Almost all are on events or timers. Cronjobs without infrastructure to worry about are nice. Basically I wrote some of this stuff 5 years ago and didn't have to look at it since; it just runs.
To be honest, the serverless environments I dealt with had worse deploy experience than plain servers. There's a bit of a ritual around deploying a lambda for example and it requires a specific packaging process. I can't easily use poetry for python projects there.
I guess if we count fargate as serverless, it's much better for this use case.
While I am not that big fan of serverless projects (I am strongly convinced lambda should be used as a glue to automate AWS infrastructure and any other uses is just abusing it) you can use poetry with lambda.
I'm sure there are other ways of doing it, but the way I do it is using serverless framework (is a nodejs app) together with serverless-python-requirements plugin. The plugin understand poetry (make sure you don't leave requirements.txt because it will use that, you don't need that file if you use poetry).
Anyway it has warts, one big thing is that nodejs developers don't care about norms and standards and constantly reinvent the wheel so for starters you're pretty much forced to create project.json project-lock.json files, which then will create node_modules directory in root of your project. Then you need to start project through npx command (I believe you could still install serverless globally but looks like that's being depreciated).
The node_modules directory will contain over 200MB of javascript code after you install it.
But after all of those things, when you get it to work it is not terrible.
>(I am strongly convinced lambda should be used as a glue to automate AWS infrastructure and any other uses is just abusing it)
I've created a whole photo sharing website in AWS Lambda, including a complete user accounts system (register, login, forgot password, email verification, profile photos, etc), social aspects, as well as other things like youtube video search and transcoding in the same system.
I also had to create my own Lambda build system because what was out there wasn't what I wanted. I wanted to be able to hit save on a file and have it repackage my Lambda on AWS (using a Lambda to create the lambda) including shared code in 'Layers'. Kind of a "live-reload" for Lambda. It's all working very efficiently.
This was a lot of effort though, but now that I've got a basic system I can extend it to any kind of website.
My photo sharing site for friends costs me about $0.25 every few months, and that is mostly/all the cost of storing gigabytes of photos on S3. No, it doesn't see a lot of traffic, and that's exactly why I chose Lambda to build this site on, because I don't have to run an EC2 instance for my photo sharing site 24/7 if nobody is using it. It's worked out exactly how I wanted, costs me practically nothing to run every year.
And, if I did build a system with a ton of active users, Lambda handles the scaling of that. Another reason I spent the time to create this build system and write all the code to handle user accounts, etc.
> And, if I did build a system with a ton of active users, Lambda handles the scaling of that.
This is a bold claim to make without verifying it.
> Another reason I spent the time to create this build system and write all the code to handle user accounts, etc.
But did you also spend the additional time to scale test it and verify it won’t break in confusing and annoying ways when scaling / spiking up? (see the other discussions re: connection pooling above).
The reason a lot of folks are skeptical is that in practice there don’t seem to be a lot of great examples of stateful services using a “pure” serverless lambda pattern but with arbitrary dependencies that actually do scale smoothly and don’t break in annoying ways such that you end up building something more complicated than just using App Engine or Heroku...
Serveless platforms like AWS Lambda are great for "glue" between services, or for workloads that are used infrequently. It can also be a great replacement for scripts that need to be run via a cron job (its surprisingly hard to create a highly available idempotent cron system using servers) provided all of the dependencies fit in the small space allotted.
I'm not sure I understand your point about not needing a deployment chain. I developed a framework (open sourcing it soon) using Jenkins Pipeline DSL for deploying Lambda functions and its very useful and helpful.
Regardless of the target environment the reasons for using a deployment pipeline are still there even if you use a server less platform.
Fargate is considered a serverless service. Serverless generally encompasses FaaS + BaaS. Think anything whose infrastructure you can use without touching the underlying VM(s): SQS, SNS, EventBridge, RDS, Lambda, Fargate, Athena, Kinesis and so on.
I'm not really interested in having the 1,000th discussion about the definition of "serverless". The parent was wondering what the value of the paradigm is, so I provided a list of specific features that many companies consider valuable with serverless offerings.
I hope this "risk aversion" mindset starts to apply to code as well.
Can't wait for codeless, zero code, nocode, whatever name it gets called... something where normal people can build their own ideas without having meaningless discussion about tdd, frameworks and etc, nor the need to maintain codebases, ci/cd and all the jazz
"Low-code" is the buzzword to search for. It's been around a while now.
Of course like all the things the idea has existed a long time, but "low-code" development services seem to be increasingly popular and visible in the last couple of years, and they do seem to be getting more advanced and ergonomic.
One serverless scenario I find useful is for processing messages from low-volume queues. It avoids having to stand up "real" infrastructure that would be under-utilised.
Of course, if you already have servers with something like k8s or Swarm running, you might deploy a container there instead.
The things that put me off were hard resource limitations (e.g. runtime/memory), idiosyncratic vendor set up and vendor lock in. The former ends up making the latter so much worse.
The wins also didn't seem great enough to overcome these twin beasts.
On Azure, I haven't really seen this problem. But you can also pay for dedicated serverless via Premium (but you pay per core/mem allocated vs. number of executions). It seems pretty interesting since multiple function apps can share the same infrastructure (allowing for more dense deployments).
User/application density is the real potential win for serverless IMO, that no one ever really talked about (to my knowledge) and noisy neighbor was the problem to be solved. The real world indicator would have been hosting costs going to zero and 10-100x apps multiplexed onto the same "real" server substrate. Instead everyone is perfectly happy paying for cloud vm instances. In hindsight I guess "function as a service" wasn't the complete solution I thought it was.
To implement serverless, you'll need to re-architecture your entire infra. For that itself, I don't think it's worth it; because serverless is pretty much all event driven (which many things might be, but might not be able to fit on serverless; even it does, it might just be easier to plop it on a simple instance).
I do see a big potential for small and simple projects though. No need to setup instances and worry about networking: just deploy it and go.
I think it's more a case of it coming full circle.
People realising that it's just another tool in the toolbox rather than 1 tool that can replace their entire toolbox.
Much of the cynicism seems to come from that.
I do agree with jason though - open ended serverless things charged per use are no fun. Now you can bankrupt yourself at scale with that bug & the 1000 instances.
People place such high expectations for new innovations solving all of their problems. No, it isn't a panacea: apply it only when it makes sense to do so.
I'm yet to meet such people in real life. At places I work people always treated serverless as a tool. Usually solutions would have a little lambda icing on a large monolith cake, so to speak.
>Pay per request needs to be within 2x magnitude of an always on server.
Are any of them? Stuff like cloud run is very competitively priced when assuming base case...but by its very nature it can scale up near infinitely. And with it bugs & bills.
The ability to cap things by 2 order of magnitudes would make me sleep much better. (1 would be better).
They don't, they are all crazy priced which means the cloud providers don't really know what they are doing, forcing people to deploy their own FaaS layers inside their cloud infra, which is a totally valid solution.
Currently, FaaS platforms should be viewed like a cron. The majority of the benefit is if the fn is called in the low thousands per day.
I was baffled, bemused even, by the article's premise that anyone rational ever thought serverless was going to take over the software world. It's a tool for solving a certain niche problem, not a general replacement for the vast majority of software we run. It's quite a good example of both the potential benefits and the potential difficulties of working in the cloud.
Whenever I read these articles, I always wonder what proportion of tech buzzwords from the past decade are just obscure ways of saying "someone else's servers", "automation" and "unnecessarily complicated architecture".
A lot of this just seems to be that Kubernetes works really well. Serverless seemed more compelling to me 5+ years ago.
But nowadays, I just setup my K8 templates with auto-scaling groups on GKE, and it pretty much works just as well as serverless was promised. Compute demand gets seamlessly transformed into compute supply.
I almost never have to think about managing individual hosts. And the cost is cheaper and performance is more predictable than serverless. Plus it's mostly all platform agnostic.
Because serverless is not serverless. Fundamentally this doesn't work because if you've gotta have a server in the end, why not either admin yourself or use one of the cloud vendor's other products for more control?
Serverless just means that your infrastructure is programmatically defined through configuration files rather than manually stiched together, so (in theory) the underlying silicon is abstracted away in a similar fashion to the x86 ASM that your code gets compiled to. In practice that isn't always the case (horror stories of rogue $xx,xxx/day lambda functions) but from my understanding that's the serverless end game.
Serverless is an economic description, not a technical one. A "serverless" service puts ownership and the immediate costs of ownership on the other side of an economic boundary.
It's a silly term, but successful. If anything, arguing against it made it more successful through mere repetition of the word.
The issue is that they represent in some manner an 'alternative architecture' that's less mature and hard to piecemeal break out of.
You need to move a few pieces around.
I don't agree that 'once you've gone K8 why bother, just use that' - I don't think K8 is as elegant as the promise of serverless, and it has it's own 'lock in'. Just so happens you may need to have K8s anyhow ... so the pragmatic question then is 'We already have K8s because we have to have it ... so in that context, we can just use it'
If we had a 'severless' version of Docker, i.e. some de-facto standard for it, I think it would obliterate a lot of architectures, just because the promise is powerful.
Developers don't actually want K8s or even true DevOps complexity, we just want a giant computer we can run stuff on and not have to worry.
K8s lockin isn't the same. You can run it on any cloud, or run it locally. None of that is true for serverless functions (unless you go for knative, which is, yep, built on K8s).
Kind of surprised the article didn't mention lack of reasonable development environment.
At least on AWS, the "SAM" experience has been probably the worst development experience I've ever had in ~20 years of web development.
It's so slow (iteration speed) and you need to jump through a billion hoops of complexity all over the place. Even dealing with something as simple as loading environment variables for both local and "real" function invokes required way too much effort.
Note: I'm not working with this tech by choice. It's for a bit of client work. I think their use case for Serverless makes sense (calling something very infrequently that glues together a few AWS resources).
Yeah, I took a look at using a serverless framework for a hobby project, and it was just a real pain to get started at all, let alone develop a whole application in.
I tried AWS, and then IBM's offering which is based on an open source (Apache OpenWhisk) project, thinking that it might be easier to work with, but that was also a pain.
I just lost interest as I was only checking it out. For something constantly marketed on the ease of not having to manage servers, it fell a long way short of "easy".
> Yeah, I took a look at using a serverless framework for a hobby project, and it was just a real pain to get started at all, let alone develop a whole application in.
Look into Firebase functions. Drop some JS in a folder, export them from an index.js file and you have yourself some endpoints.
The amount of work AWS has put in front of Lambdas confuses me. Firebase does it right. You can go from "never having written a REST endpoint" to "real code" in less than 20 minutes. New endpoints can be created as fast as you can export functions from an index.js file.
And if you need a dependency that has a sub dependency with a subdependency that uses a native module prepare for poorly defined -fun- hell getting it to work. A surprising amount of standard is libs do.
Being able to throw up a new REST endpoint in under 10 minutes with 0 config is really cool though.
And Firebase Functions are priced to work as daily drivers, they can front an entire application and not cost an insane amount of $, per single ms pricing. Lambda's are a lot more complicated.
GCP has https://cloud.google.com/functions/docs/functions-framework but I will not use it. I have found the best solution is to abstract away the serverless interface and create a test harness that can test the business logic. This adds some extra complexity in the code, but iterations are fast and do not rely on the overly complex and bug prone "platforms" like SAM and Functions Framework.
This is precisely what I do when I write code destined to be an AWS Lambda Function. It really feels like the only sane way to do it. It also makes it easy to mock the incoming request/event for integration tests.
Keep in mind Firebase has a big caveat. Firebase is great... for what it does. However, there's no way to easily migrate the Firebase resources to the larger GCP ecosystem. Firebase does what it does, and if you need anything else, you're out of luck.
Firebase is magic... but I never recommend it for anyone, until there's some sort of migration path.
[Firebaser here] that’s not quite accurate. For cloud functions they’re literally the same. Your Firebase function is actually a GCP function that you can manage behind the scenes.
With Cloud Firestore (our serverless DB) that’s the case as well. And Firebase Auth can be seamlessly upgraded to Google Cloud Identity Platform with a click.
However you’re right that for many Firebase products (Real-time Database, Hosting) there’s no relation to Cloud Resources.
Deploying is super slow. Usually it takes a minute or two, which is already quite long, but sometimes something goes wrong and then you can't redeploy immediately. You have to wait a couple of minutes before being able to redeploy.
To be fair, Firebase recently released a local development tool which alleviates the need to deploy on every change, but I haven't used it yet.
I'm a big Firebase user with Firestore and has been great...no not perfect and the "cold start" is probably the worst issue. However, deployments are easy, the GUI tool keeps getting better, like the extension packages, and the authentication system is quick to implement.
Developer experience for serverless is such a pain point, spot on. AWS SAM has tackled some of the IaC modeling problem (on top of CloudFormation which is a mature choice) and they've had a crack at the local iteration (invoke Lambda function or API Gateway locally with connectivity to cloud services).
It's a little incomplete, missing some of the AWS IAM automation that makes local development smooth, environment management for testing and promoting changes, and some sort of visualization to make architecture easier to design as a team.
I work for a platform company called Stackery which aims to provide an end-to-end workflow and DX for serverless & CloudFormation. Thanks for comments like these that help identify pain points that need attention.
I felt your pain immediately and decided to write my own mini-framework to accomplish this.
What I have now is a loosely coupled, serverless, frontend+backend monorepo that wraps AWS SAM and CloudFormation. At the end of the day it is just a handful of scripts and some foundational conventions.
I just (this morning!) started to put together notes and docs for myself on how I can generalize and open source this to make it available for others.
stack is vue/python/s3/lambda/dynamodb/stripe but the tooling I developed is generic enough to directly support any lambda runtime for any sub-namespace of your project so it would also support a react/rails application just as well.
A lot of these DX (developer experience) concerns are, imo, rooted in what the article describes as "Vendor Lock".
Sure, you can write a bunch of tools to work around the crufty, terrible development environment's shortcomings. But ultimately, you are just locking yourself further & further & further in to the hostile, hard to work with environment, bending yourself into the bizarre abnormal geometry the serverless environment has demanded of you.
To me, as a developer who values being able to understand & comprehend & try, I would prefer staying far far far away from any serverless environment that is vendor locked. I would be willing & interested to try serverless environments that give me, the developer, the traditional great & vast powers of running as root locally that I expect. Short of a local dev environment, one meets both vendor lock in, & faces ongoing difficulties trying to understand what is happening, with what performance profiles/costs. I'd rather not invest my creativity & effort in trying to eek more & more signals out of the vendor's black box. Especially if trouble is knocking, then I would very much like to be able to fall back on the amazing toolkits I know & love.
aws's whole pitch has been cutting out server huggers & engineers, relying on aws. since day 1. often to wonderful effect. with far far better software programmability than our old crufty ways.
but lambda gets to the point where there is no local parity, where it's detached, is no longer an easier managed (remotely operated) parallel to what we know & do, but is a system entirely into itself, playing by different rules. one must trust the cloud-native experience it brings entirely, versus the historical past where the cloud offered native local parallels.
I never got the hand of cloud formation. I suppose it is nice from a visual (drag and drop) point of view, but I couldn't use it in production and moved on to manage my architecture with terraform.
I manage a handful of projects with Terraform and it works well in many situations. It has improved a lot recently but for a long time I really hated the syntax. I still do to some extent but have learned to cope with it most of the time.
If you are working on a project where all of your infrastructure will live on AWS I would definitely urge you to give it a second look. The amount of infrastructure I manage right now with a single .yaml file is really killer.
This is a really funny thing, since for the last ~10 years I've been hearing how we're deliberately doing IaC/CM tools "not a programming languages because reasons" (and thus have to do horrible hacks to support trivial things like loops and conditions), and now suddenly we're building libraries in programming languages that convert the code into non-programming-language description, which is then interpreted by a program into several other intermediate representations and finally emits trivial API commands. I guess the next step would be write a declarative language on top of CDK or Pulumi that will compile it into python which will generate CF/TF files.
It sounds like you're describing the Cloudformation template visualiser/editor in the AWS Console, which I have never heard of anyone using as the primary interface for their Cloudformation templates.
Personally for simple projects I've had pretty good experiences writing a Yaml-based template directly, and for more complex projects I use Troposphere to generate Cloudformation template Yaml in Python.
Yes, it (Python) was chosen because we could leverage existing internal code that was written in Python and it happens to be my strongest language.
If I could do it all over, I would still choose Python. That being said, I have been working professionally (building apps like this) for almost 14 years so my willingness to bite off a homebrew Python framework endeavor as I did here is a lot different than someone just getting into the field.
Django: avoid unless you have a highly compelling (read: $$$$) reason to learn and use this tool. I cannot think of one, honestly.
Flask: fantastic, but be conscientious about your project structure early on and try to keep businees-logic out of your handler functions (the ones that you decorate with @app...)
Sophisticated or more sugary Node.js backends are not something I have ever explored, aside from the tried-n-true express.js. I tend to leverage Python for all of my backend tasks because I haven't found a compelling reason not to.
Django gets you a lot if you have a traditional app with a traditional RDBMS and a traditional set of web servers. It’s too opinionated to easily map into AWS serverless.
On the flip side, not being able to use Django is one of the reasons against serverless for me. There's immense value in having a library for anything you might think of, installable and integratable in minutes.
You have to roll your own way too often in Flask et al, so much so that I don't see any reason to use Flask for anything other than ad-hoc servers with only a few endpoints.
That is such a tough question to answer carte blanche.
All-in-all, Django is not bad software. I have a bad taste in my mouth though because as I learned and developed new approaches to solving problems in my career I feel like Django got in the way of that.
For instance, there are some really killer ways you can model certain problems in a database by using things like single table inheritance or polymorphism. These are sorta possible in Django's ORM, but you are usually going against the grain and bending it to do things it wasn't really supposed to. Some might look at me and go: ok dude well don't do that! But there are plenty of times where it makes sense to deviate from convention.
That is just one example, but I feel like I hit those road blocks all the time with Django. The benefit of Django is it is pre-assembled and you can basically hit the ground running immediately. The alternative is to use a microframework like Flask which is very lightweight and requires you to make conscious choices about integrating your data layer and other components.
For some this is a real burden - because you are overwhelmed by choice as far as how you lay out your codebase as well as the specific libraries and tools you use.
After your 20th API or website backend you will start to have some strong preferences about how you want to build things and that is why I tend to go for the compose-tiny-pieces approach versus the ready-to-run Django appraoch.
It's really a trade-off. If you are content with the Django ORM and everything else that is presented, it is not so bad. If you know better, you know better. Only time and experience will get you there.
That's great, cheers for that. It's helpful to know that your concerns are mainly to do with taking an opinionated vs non-opinionated approach - that's a framework for thinking about the choice between Django and (e.g.) Flask that many people (including myself) can hang their hat on.
Django is decent for POCs that need some level of security since you get authentication out of the box with no external database configuration necessary due to sqlite. Sometimes you have an endpoint that needs that due to resource usage, but the number of users is so low that setting up a complicated auth system isn’t worth it.
Minimalist frameworks are great for either very small (since they don’t need much of anything) or very large projects (since they will need a bunch of customization regardless).
In that regard, I think Django is kind of like the Wordpress of Python.
As a systems developer, comments like yours make me amazed at the state of web development. From the outside looking in, it seems like 10% code and 90% monkeying around with tooling and frameworks and stacks.
I believe it's the moment when there's a solution that just makes sense and works well for most people. A gold standard that other solutions will try to develop more and spice up, instead of reinventing it.
> Kind of surprised the article didn't mention lack of reasonable development environment.
I've been pretty happy with Cloudflare Workers.
You can easily define environments with variables via a toml file. The DX is great and iteration speed is very fast. When using `wrangler dev` your new version is ready in a second or two after saving.
I used to be complain about the same thing and even asked someone who was head of BD for Serverless at AWS what they recommended, and didn't get an answer to my satisfaction. After working with more and more serverless applications (despite the development pains, the business value was still justified) I realized that local development was difficult because I was coupling my code to the delivery. This is similar to the way you shouldn't couple your code to your database implementation. Instead, you can write a function that takes parameters from elsewhere and call your business logic there. It definitely adds a bit more work, but it alleviates quite a bit of pain that comes with Lambda local development.
Disclaimer: I work at AWS, however not for any service or marketing team. Opinions are my own.
> I realized that local development was difficult because I was coupling my code to the delivery.
Of interest, I've spent some free time crunching on CNCF survey data over the past few months. Some of the strongest correlations are between particular serverless offerings and particular delivery offerings. If you use Azure Functions then I know you are more likely to use Azure Devops than anything else. Same for Lambda + CodePipeline and Google Cloud Functions + Cloud Build.
I think his point was that you should be able to run and test the Lambda code independently of Lambda. After all the entry point is just a method with some parameters, you can replicate that locally.
Yes, this is a great way of doing things - I have no problems TDD'ing business logic hosted in Lambda, because the business logic is fully decoupled from the execution environment. SAM should be for high-fidelity E2E integration testing.
> Instead, you can write a function that takes parameters from elsewhere and call your business logic there.
This is what I tried to do initially after experiencing the dev pain for only a few minutes.
But unfortunately this doesn't work very well in anything but the most trivial case because as soon as your lambda has a 3rd party package dependency you need to install that dependency somehow.
For example, let's say you have a Python lambda that does some stuff, writes the result to postgres and then sends a webhook out using the requests library.
That means your code needs access to a postgres database library and the requests library to send a webhook response.
Suddenly you need to pollute your dev environment with these dependencies to even run the thing outside of lambda and every dev needs to follow a 100 step README file to get these dependencies installed and now we're back to pre-Docker days.
Or you spin up your own Docker container with a volume mount and manage all of the complexity on your own. It seems criminal to create your own Dockerfile just to develop the business logic of a lambda where you only use that Dockerfile for development.
Then there's the whole problem of running your split out business logic without it being triggered from a lambda. Do you just write boiler plate scripts that read the same JSON files, and set up command line parsing code in the same way as sam local invoke does to pass in params?
Then there's also the problem of wanting one of your non-Serverless services to invoke a lambda in development so you can actually test what happens when you call it in your main web app but instead of calling sam local invoke, you really want that service's code to be more like how it would run in production where it's triggered by an SNS publish message. Now you need to somehow figure out how to mock out SNS in development.
Unless I’be misunderstood, every knock against serverless above has actually been a knock against the complexity of having tiny, de-coupled cloud native services and how difficult it can be to mock... to which the answer is often “don’t mock, start by using real services” and then when that is less reliable or you need unit tests, then mock the data you expect. In the case of SNS, mock a message with the correct SNS signature, or go one layer deeper, stub out SNS validation logic and just unit test the function assuming the response is valid or invalid? In the case of Postgres, you could use an ORM that supports SQLite for dependency-free development but at a compatibility cost... worst case you might need to have your local machine talk to AWS and host it’s own LetsEncrypt certificate and open NAT port... but one can hope it doesn’t come to that...? Even so... that’s not exactly a knock against serverless itself, is it?
Well sure, but then your development (and test) environment is tied to live services and you now you have two problems. It really, really sucks when an unrelated service goes down during an existing incident and you never mocked.
> In the case of SNS, mock a message with the correct SNS signature, or go one layer deeper, stub out SNS validation logic.
SAM already provides a way to mock out what SNS would send to your function so that the function can use the same code path in both cases. Basically mocking the signature. This is good to make sure your function is running the same code in both dev and prod and lets you trigger a function in development without needing SNS.
But the problem is locally invoking the function with the SAM CLI tool is the trigger mechanism where you pass in that mocked out SNS event, but in reality that only works for running that function in complete isolation in development.
In practice, what you'd really likely want to do is call it from another local service so you can test how your web app works (the thing really calling your lambda at the end of the day). This involves calling SNS publish in your service's code base to trigger the lambda. That means really setting up an SNS topic and deploying your lambda to AWS or calling some API compatible mock of SNS because if you execute a different code path then you have no means to test the most important part of your code in dev.
> In the case of Postgres, you could use an ORM that supports SQLite for dependency-free development but at a compatibility cos
The DB is mostly easy. You can throw it into a docker-compose.yml file and use the same version as you run on RDS with like 5 lines of yaml and little system requirements. Then use the same code in both dev and prod while changing the connection string with an environment variable.
> That’s not exactly a knock against serverless itself, is it?
It is for everything surrounding how lambdas are triggered and run. But yes, you'd run into the DB, S3, etc. issues with any tech choice.
So there’s an argument that the future deployment model is actually Kubernetes Operators, which means you could have test code that deploys and sets up AWS APIs... thus if your code responds to the trigger, it’s up to another bit of code to make sure the trigger is installed and works as expected against AWS APIs?
And yes, I think the problem here are APIs you use in multiple places but can’t easily run yourself in a production-friendly way. Until AWS builds and supports Docker containers to run their APIs locally, I don’t see how this improves... end to end testing of AWS requires AWS? ;-)
This principle works with front end development too. Crazy build times on large applications can be alleviated if your approach is to build logic in isolation, then do final testing and fitting in the destination.
It’s hard to do this when surrounding code doesn’t accommodate the approach, but it’s great way to design an application if you have the choice. I really love sandboxing features before moving them into the application. Everything from design to testing can be so much faster and fun without the distractions of the build system and the rest of the application.
Take a look at the [CDK](https://aws.amazon.com/cdk/) if you haven't already. It lets you define your infrastructure using TypeScript, which then compiles to CloudFormation. You can easily mix infrastructure and Lambda code in the same project if all you're doing is writing some NodeJS glue Lambdas which sounds like what you're looking for.
There's a couple of sharp edges still but in general it just 'makes sense'. If you don't like TypeScript there are also bindings for Python and Java, among others, although TypeScript is really the preferred language.
CDK made IaC accessible to me. I hated raw CloudFormation and never bothered with it because of that reason. I had a crack at Terraform, but never got passed the learning curve before my enthusiasm died.
Currently using some CDK in a production app and finally I found a way of doing IaC I actually enjoy.
You might really like pulumi. I'm kind of on the opposite end, ops>swe so tons of IAC and i'm using pulumi now as I'm more swe focused https://www.pulumi.com/ (ive no relation to them)
Basically exact same as CDK. I really prefer this style over CloudFormation and Terraform. I think Pulumi emerging as another player in the space legitimizes the approach.
CDK is moving quite fast and not all parts are out of the experimental phase, so there are breaking changes shipped often. I think in a couple of years it will stabilize and mature and become a very productive way of working with infrastructure.
I found Amplify excellent to get up and running quickly. I’d highly recommend it for anyone without a well-oiled CICD setup who wants to quickly get a website up to test out an idea.
Unfortunately, I quickly hit the limits of its configurability (particularly with Cloudfront) and had to move off it within a few months.
Couldn’t agree more, the dev experience was awful. You basically have to develop against public AWS services, my dev machine became a glorified terminal. They do seem to be iterating on the tooling quickly, but I wouldn’t use it again if I had a choice.
Edit: CloudFormation was also painful for me, the docs were sparse and there were very few examples that helped me out.
SAM templates are a subset of CloudFormation templates; that PDF could be three times as long and still not have the content I needed.
Yes there are examples, but there wasn’t one at the time that mapped to what I was trying to accomplish. Because, again, SAM templates are not one-for-one CloudFormation templates.
I found the community around SAM to be very limited. One of the many reasons I’ve moved to the Kubernetes ecosystem.
> It's so slow (iteration speed) and you need to jump through a billion hoops of complexity all over the place. Even dealing with something as simple as loading environment variables for both local and "real" function invokes required way too much effort.
Honestly, it reminds me of PHP development years ago: running it locally sucked, so you need to upload it to the server and test your work. It. Sucked.
It was actually pretty good if you had an IDE with sftp/scp support because you could save a file, refresh your browser, and have immediate new results.
Yeah this wasn't too bad and it was what I used to do back in the day with Notepad++. By the time you hit save in your editor, your changes were ready to be reloaded in the browser.
With SAM we're talking ~6-7 seconds with an SSD to build + LOCALLY invoke a new copy of a fairly simple function where you're changing 1 line of code, and you need to do this every time you change your code.
That's even with creating a custom Makefile command to roll up the build + invoke into 1 human action. The wait time is purely waiting for SAM to do what it needs to do.
With a more traditional non-Serverless set up, with or without Docker (using volumes) the turn around time is effectively instant. You can make your code change and by the time you reload your browser it's all good to go. This is speaking from a Python, Ruby, Elixir and Node POV.
The workaround my team uses is to make two entries to start the webapp: one for SAM, one for local. For fast iteration we just `npm start`, and when we're ready to do more elaborated testing we run with SAM. This works pretty well so far.
Out of genuine interest... is there a modern solution to this problem with PHP/MySQL?
(I'm still doing the "upload to server to test" thing.... I've tried MAMP and Vagrant/VirtualBox for local dev but both of them seem horribly complex compared to what we can do with local dev with node.js/mongo and so on.)
"docker-compose up" and your OS, codebase, dependecies and data is up and running locally in the exposed local port of your preference. You can even split parts in different layers to mimic a services/cross-region logic.
Of course this won't fix the fact that you have a lambda behind api gateway that does some heic->jpg conversion and can't be hit outside the DMZ, or some esoteric SQS queue that you can't mimmic locally - but it should get you almost there.
FWIW I haven't hit any scenario out of the basic services that localstack couldn't run locally. I even have it executing Terraform on localstack as if it was AWS (without IAM which is indeed a problem when I forget to create/update policies)!
this doesnt solve the OPs problems though, if Vagrant is complex, so would be a docker image. the problem is the user doesn't know how to manage/configure the underlying complexity of the OS and needed services, which would still be a problem if using docker. unless you find that perfect docker image with every dep you need... but that would also be true with vagrant.
Just run PHP and MySQL locally? Native PHP on Windows is horrible to set up, but with WSL/WSL2 you suddenly can get a normal Linux environment without much hassle.
sudo apt install nginx php mysql, point the www directory of nginx to something on your computer (/mnt/c/projects/xyz) and you've got a running setup. Or run Linux in general, that's what most people I've seen work on backends seem to do. You can run the same flavour of software that your deployment server runs so it'll save you time testing and comparing API changes or version incompatibilities as well.
I don't know any solution for macOS but you can probably get the necessary software via Brew if you're so inclined. Then run the built-in PHP web server (php -s) and you get the same effect.
What's horrible about it? I just download it, unzip to Program Files, add the folder to my %PATH% and that's about it. I didn't find myself in a situation where I would need an Apache or other webserver, the built-in one is good enough. It also makes using different versions easy, no need to deal with Apache and CGI/FPM. You just use other PHP executable.
I find it easier to handle multiple PHP versions on Windows than on Linux. As you say just download zip,unpack somewhere copy php.ini-development to php.ini, and you can do this for every minor PHP-version.
Apache is almost as easy, download zip, unpack & configure apache conf to use your php.
MySQL is somewhat more complicated because you need to run an setup script after unpacking the zip.
I'm not sure why that's PHP's fault? I never had problems running it locally... and to "get my code to the servers" was as easy as a git pull from the server which is probably the 2nd laziest way of accomplishing that.
I can report that Azure Function App development is at least pretty decent, as long as you have the paid Visual Studio IDE and Azure Function Tools (haven't tried the free version yet).
I tried AWS Lambdas a few years back and it felt way more primitive.
Azure Function App development experience is indeed pretty nice at least when using .NET Core. There are some issues, like loading secrets to the local dev environment from Key Vault has to be done manually and easy auth (App Service Authentication) does not work locally.
It definitely doesn’t have to be that way. I work on Firebase and I’ve spent most of the last year specifically working on our local development experience for our serverless backend products:
https://firebase.google.com/docs/emulator-suite
Still very actively working on this but our goal is that nobody ever has to test via deploy again.
Love firebase, thanks for your work! the local emulator suite is so important a feature and keenly following your progress.
Slightly OT... perhaps cheeky... any idea why firebase doesn’t provide automated backups for firestore and storage out of the box? Seems like a no brainer and a valuable service people would pay for.
I'm currently working on a little project backed by Firebase. Really interesting. Good to hear you're doing this - at my day job one of our key factors in choosing a technology is whether we can spin it up in docker-compose in CI and do real-as-possible ephemeral integration tests.
My experience is with .NET Core and the development experience is awesome... Dropped a $250/m cost down to ~$9/m moving it from ec2 to lambda's. Environment variables are loaded no differently between development and prod. Nothing is all over the place as it's almost 0 difference between building a service to run in Linux/Windows vs a Lambda.
I've used Azure's serverless offering "Functions" quite a bit. The dev experience is pretty good, actually - it "just works" - start it and point your browser at the URL. And certainly no problems setting up env vars or anything basic like that.
My only nitpick, and only specifically relating to dotnet, is that config files and env vars differ between Functions and regular ASP.NET Core web apps. I think there is some work going on to fix that, but it's taking forever.
- performance: first rule of architecture is that you don't build the entire thing on the needs of high performance...you only address performance as-needed
- vendor lock in: this is the worst reason. Most companies choose cloud vendors and stick with them over many years. I'd love any survey that showed some massive migration between vendors on a regular basis, but it does not exist.
- the monolith: granted, serverless is really best when building something from scratch. Trying to "migrate" to serverless is just a bad idea. Trying to "upgrade" portions is okay, but do it well using the autonomous bubble pattern, making sure you have a solid translation layer
I believe serverless is our best modern architecture, but I also understand that not every company is ready to do the work and/or pay the price of the change in development processes. I'd agree that it's stalled, but moreso out of fear and ignorance than anything else.
> vendor lock in: this is the worst reason. Most companies choose cloud vendors and stick with them over many years. I'd love any survey that showed some massive migration between vendors on a regular basis, but it does not exist.
Why do you think companies stick with vendors for so long instead of switching?
>"I'd agree that it's stalled, but moreso out of fear and ignorance than anything else."
How about simple common sense instead of "fear and ignorance". I have a C++ monolith that serves me just fine already for years. The performance is great and I am able to rent dedicated servers that can serve at least 10 times more requests that I have at a price that is a tiny fraction of AWS for the same work.
Why would I switch and incure more expenses? What's the ROI?
What change? I am not going to worry about some abstraction. You have to be more particular. In my case I change features on on need basis with no problems. Hence I have 0 issues with the monolith. Micro- services actually do not really make life any easier. All that immense cloudy push is driven by handful of very big corps who'd build architecture for scale that most of the businesses would never even dream to approach. So I'll let them worry about their problems and I'll worry about mine. I am not in business of transferring my money to cloud paying 10x and more comparatively to my current solutions.
We’re not buying into IBM mainframes anymore and although there are still customers tied to Oracle databases, vendor lock-in just isn’t that big of a deal anymore.
I see serverless working really well for a few niche tasks. Finite, stateless interactions that don't happen often. A good example will be a site say example.org that has a donate page. On form submit, a lambda can take the payload and create a subscription at a payment service like Stripe or Razorpay. The donate page at https://internetfreedom.in works the exact same way.
The broader idea is a service (or static site even) sending messages to a different service. Given how popular no-code platforms are getting, tiny serverless pieces can help automate operations between these platforms.
Caching is another big one. Beyond the simplest of services you typically need to cache some info in memory, create connection pools, etc at startup to improve latencies. With serverless that is not possible without killing your cold start time. So it's hard to go serverless except for the simplest of applications.
I'd point out two other aspect not mentioned here - first is that the complexity doesn't go away. Application architectures running on a 'serverless stack' have a lot more moving parts than a traditional server based equivalent. Try showing the you from 10 years ago a modern serverless architecture and they'll think you're mad. More moving parts is more maintenance/problems, but on the plus side, does allow you to throw more resources at a specific part of the whole.
Second - with each cloud provider's serverless service, I've noticed that the underlying complexity is shifted into your application and provisioning. They're a great example of leaky abstractions. The name 'serverless' may lure you into thinking that you don't need to think about servers any more, but actually you do have to adjust to the way they've implemented a service that you're using. Think of Lambda cold starts, or have a look at the DynamoDB pricing page. There's still a server running somewhere.
Personal experience: I've found that ECS (Fargate) is a decent step into the serverless realm, if you've been running normal containers on normal EC2s. It does take away the EC2 management aspects, and it's especially useful if you have some applications that are resource hungry, and some are not. It's a cheap way to run ETLs (we use the ECS Operator from Apache Airflow, which itself is on EC2). It doesn't take long to learn, and it's a good place to start. It does come with its own baggage... the 'stack' that you had on a single EC2 is now scattered about; you may need AWS Cloud Map for service discovery, ALBs for networking, SSM/KMS for secrets, Cloudwatch for logging.
That's when you realize, those sticker infested laptops at Devops conferences are actually modern architecture diagrams.
I agree. I only generally use Fargate with containers, and control with cron or with always on APIs, with Cloudwatch for monitoring, and it’s seems like a good midpoint between ec2 and serverless.
The APIs cost a bit more a month (they’re on quite low cpu/memory) because they’re always on, but it minimises complexity and it’s nice to know that you can move those containers over to another platform if needed.
Infrastructure is always much harder to maintain (and test) than application code. Serverless architectures can risk moving complexity from (easy-to-test) application code to (hard-to-test) infra.
Hmmm maybe if you're talking about developing from scratch, but most developers have a lot of knowledge and assumptions about existing implementations of infrastructure; also, in many organizations developers don't have to think too much about implementing the infrastructure, someone else takes care of that. And at the very least, there's a separation of concerns: you don't have to think about both at the same time.
Actually, re-reading your comment, I'm not sure what you were trying to say. I may be agreeing with you.
---
I think it is possible for serverless to supplant PaaS et al, but I don't know if it's quite mature enough yet.
For stateful systems such as databases, that typically require complications like replication and backup, configuration, testing and maintenance is hard to get right.
But for stateless systems, IMO infrastructure is pretty straightforward.
Seconded. Serverless has the benefits and drawbacks of Microservices: you build and deploy smaller units, at the consequence of having to debug the interconnects between everything, with poor tools.
I'm a fan of OpenFaas: you get the benefits (but not all the drawbacks) of containers and serverless. It's easy to mix and match running 100% locally, or mix-in baked 3rd party components, or running in Kuberenetes or other systems in production. Also, no vendor lock-in.
+1 for openfaas. I haven’t used it in production, but the option to run an entirely custom container underneath the function opens up the possibility of running a “proper” application underneath (the use case I’ve been meaning to test is a fully-featured rails app)
Most of these seem like fairly weak reasons at this point.
Some degree of vendor lock-in is pretty much inevitable no matter what solution you go with.
Limited programming languages may have been an issue a few years ago, but with the support of Java, Go, Python, Javascript to name a few - most of the core user bases are covered.
Additionally, after the introduction of API gateway, most CRUD applications, including those that require async tasks, can be served by the lambda and serverless.
After having used serverless for a while, the biggest turn off for using it (still) seems to be the performance issues. Provisioned concurrency, i.e leaving function containers running, is really just a bandaid and runs counter-intuitive to the original motivation of serverless to begin with.
Serverless just moves complexity around and makes some things a bit more awkward, like debugging and logging. You have to be all in on the platform. I’ve just taken over a serverless app after the lead left and there are so many moving parts to make it all work.
I am still on the serverless high, and I don't see it going away any time soon. For the large corporation that I work for, things like soc2, patching, server maintenance, etc. are a giant pain every single month. The quicker we switch as many things as possible to serverless the better. One of the services I converted over went a year and a half with absolutely no attention from our engineers. It was great seeing how old the latest deploy was. Coupling serverless with terraform and git makes everything even easier. It means doing things like adding another lambda to an API as simple as copy and paste terraform, replace phrases in terraform, write code. terraform apply and done.
How do you use Terraform with Lambda? That's been one of the huge painpoints in my limited Lambda experience. All the CI tools (SAM, Serverless Framework, e.g.) seems to want to handle the IaaS part themselves and step on Terraform's toes. I just want to set everything up with Terraform and then have a nice simple tool to build and deploy the application.
We use terraform and terragrunt by themselves to deploy lambdas. No frameworks. It's slightly more work, I guess. But deploying and understanding how deployments happen is easier this way. Plus, getting any kind of framework approved in a corporation is annoying as heck, whereas we already use terraform and terragrunt.
The biggest annoyance is unit testing, although it forces certain good habits on us. So for example, since we don't use SAM, we have to do unit testing separately (our code is in Python). But because we can't exactly run a lambda via Python, we have to put most of our code in the layers that our lambdas import.
So we run unit tests on the layers, and the positive is that we can essentially copy and paste the code that our lambdas would run into the unit tests. And then we can keep it DRY because almost all the code is in layers, and that incentivizes us to create generic functions that can be used across all of our lambdas.
For things like unit testing DynamoDB, we use Moto which works exceptionally well.
We have build our apps and create a lambda zip file and deploy it out of band of our Terraform. We just upload the zip file to S3 after deploy and if Terraform runs and makes a change to the Lambda, it deploys the most recently deployed version automatically.
We did this so we could totally separate our applications and the infrastructure as code stuff since infrastructure changes at a drastically slower rate than applications.
How does moving to 'serverless' change the need for patching and other security maintenance? You've just moved it to a server owned by someone else. Sounds more like passing the buck rather than ensuring security.
So for example, if we have a service deployed in an EC2 instance, we have to update the instance(s) every month by a certain date. Sometimes, we have to update early if there is a severe security issue. We also have to manage things like security software that has to run on every instance, logging for soc2 compliance, access controls, manage ssh keys, etc. etc.
That's a lot for our engineers to handle every single month. A lot of context switching. It's not so bad if we only have one service to manage. But the actual issue is that we have many services to manage. So every month, our board had about 12 stories related to patching. Patch service X development. Patch service x staging. Patch service x production. Patch service Y development.... and so on.
For serverless, the team responsible for SOC2 compliance gets the reports from AWS without us ever having to do anything. No patching stories, no security software that breaks on occasion, no access management, etc.
All we have to worry about in terms of SOC2 compliance are the basics like principle of least privilege (i.e. lambdas do not have full access to DynamoDB), no public endpoints, we use VPCs blessed by the security team, and that's about it. Things we're used to doing everywhere else and things that we only worry about when writing the terraform.
Now when I add a Lambda, I only worry about writing the Python code for the lambda and copying the terraform code. That's it. There is no patching story for this service.
And yes, of course we're passing the buck. That's kinda what we pay for in the premiums that AWS gets from our massive bill every month. That's kind of the entire point for cloud-based systems, anyway. We're passing the buck on a ton of other things, like not having to manage hard disks, not having to manage network cables, not having to manage a building, not having to manage HVAC, not having to manage backup batteries, etc.
I guess in places where I've worked we've had Ops folks that handled things like access controls, managing secrets, servers, network, etc. Usually in space rented in a commercial datacenter, where the datacenter handles power, cooling, physical security, and where the building has a diversity of internet providers. Its a team effort. In my opinion, software developers shouldn't be responsible for OS level patches, OS access controls, etc. It makes sense in a startup situation but it doesn't scale and it only takes one misstep.
It's a valid opinion piece, but I don't see any numbers or facts in it that point to the stall of serverless. From what I can see in the industry services like Lambda are still very highly used and growing exponentially.
I like the term "nano services" is it probably the wrong term.
But I like it.
We went monolith -> micro services -> micro services + nano services
Everything that adds complexity in a micro service system, adds far more complexity with nano services.
Debugging and tracing is a nightmare.
And yes, logging, a lot of logging and instrumentation helps.
But if you add too much of it then the whole point of the nano services is lost.
Versioning can get complex, depending on your deploy model.
Presumably if you push 100% of your nano services with every deploy you can get around it.
We have about 600 nanoservices running on AWS now
across 4 different systems.
My advice, that nobody would ever ask for, keep AWS Lambda for certain special tasks where they are a very good fit.
As always run the numbers, performance, cost to run, cost to maintain. Will the immense ability to scale really help your
use case?
In most of our cases, no, they will not.
For some companies it will be a great solution and solve
pain-points, save money
669 comments
[ 2.7 ms ] story [ 339 ms ] threadAWS Lambda can run all programming languages via layers.
Cloudflare Workers can run all programming languages that compile to WebAssembly.
"Vendor Lock"
True, but being locked into Kubetnetes isn't a cakewalk either.
"Performance"
Cold-starts aren't a thing for Cloudflare Workers and can be mitigated for AWS Lambda. AppSync and API Gateway don't even have them if you directly integrate with AWS services.
"You Can't Run Entire Applications"
You can and it has been done multiple times. Sure, you shouldn't blindly build everything with Lambda and API-Gateway without measuring. But many services, especially Cloudflare Workers are quite cheap.
Anecdotally, stories of businesses abandoning entire cloud automation projects because they wasted weeks and never had anything to show from it don't seem unusual, so evidently a considerable amount of knowledge and skill is required to get value out of these tools.
I used K8s and serverless tech and while serverless wasn't as simple as some evangelists try to sell it, it was at least a magnitude simpler than K8s.
Even managed K8s, which removed most of the admin plane work with nodes, was still significantly more work to get up and running.
Serverless makes somethings easier and other things harder, so it's advantages over good old fashioned servers are clear cut.
I was quite impressed when I exposed Kinesis via API Gateway, it just worked, although the Kinesis requirement that data be b64 encoded did require some fiddling about with mapping templates.
Sure, a container is better than a serverless function in many ways, but the function isn't what the container is competing with.
Ha. So the name didn't give that away then?
In all seriousness though, I think another avenue for serverless to still explore seriously is declarative application definitions rather than just functions/runtime as a service. Hasura[0] are doing this with GraphQL but I think we'll soon seen (are already seeing?) a surge in these kinds of cloud abstractions that do much more for you than a simple on-demand runtime ever could.
[0] - https://hasura.io/
For the majority of companies out there it's just a tool to overcomplicate your stack and turn it into an engineering playground so you can justify 2-3x the headcount despite no significant productivity increase. But hey, at least your company can now be giving talks about how they solve their (self-inflicted) problems managing all the microservices and throw that buzzword on the careers page.
As a worker, and not a company owner, this sounds awesome.
Prove to me that what you’re trying to do can’t be solved with a Python script, Postgres and a beefy machine. I don’t care about the theoretical benefits of whatever you’re pushing, I want hard numbers. Unfortunately the community has spent quite a lot of its time and effort these last few years making it easy to unnecessarily scale horizontally. I want no part of that.
It's always so strange, being in this space where everyone is dealing with tons of data, and hearing constantly on HN about how "you don't need this level of scale unless you're Google".
That being said, it's always possible to add a bit more complexity if you're not careful.
However, the frequent posts along the lines of "my Lambda function was buggy and now I owe Amazon three billion dollars" definitely kills the romanticism.
You'd be surprised how great it is to pass a parametized query and have it turned into a function quickly.
Doing stupid things on AWS is really easy and sadly billing can be up to 24 hours delayed.
How about we call it auto-snailing. "I wanted to grow fast, but not THAT fast! Slow it down there buddy."
There are lots of posts online that document "talk to amazon and they'll take out the extra charges" but there are even more cases where the people just pay the bill and thus amazon makes money and profit.
Additionally, bill cutting is not that easy. Should they shut down current services? Stop sending in the middle of a newsletter? Delete your S3 Storage? Even then, a large company might accumulate hundreds or thousands of dollars within a second ; even if they decide to cut at all cost, minimal delays might break the limit.
I fully agree that it sucks for experimenting; it is very much the reason I don't have an AWS account. But I can see why they don't have it.
The problem is there's not a clear way to tell the difference between recursive calls and heavy traffic.
AWS components are fundamentally a network of nodes that are sending traffic to other nodes. But there's no unifying language that can model this, so you can't describe what this graph ought to look like. (e.g. you want to be able to say "lambda Foo should be called twice for each event to Bar".)
And that'd be hard to do since most nodes are general purpose computers, like Lambda and EC2, that can send traffic anywhere for arbitrarily complex reasons.
Since you can't describe what it ought to look like, trying to alarm on it becomes a problem of detecting anomalies. You can sort of do it, but AWS doesn't want to do it because they'd be making a promise they couldn't generally keep.
As the article says, serverless is one of many (many) ways to wrap a quantum of functionality inside an internet-accessible environment. You could have a chunk of python in a serverless setup, a small flask server in a container in k8s, as an endpoint in a monolith, etc. Each of these environments have difference performance, cost and maintenance characteristics, but I think of the star feature of serverless as the light deploy. For companies that don't have a deploy chain, that makes serverless very attractive, but if you've already invested in one it seems less compelling.
I think serverless products have a ton of potential for side projects and hobby projects. It's great that you can just type out python and access it on the internet immediately! That quality just never seemed to be what was holding back SAAS companies.
Not the core of your app, but a tool of great utility. On the other hand, this isn't exactly the Serverless Revolution either.
If you have a server that can handle up to 100 requests at a time, but you're only getting one or two a day, you could probably save money by switching to serverless. On the flip side though, you're also a bit screwed if 1000 requests all come in at once, since, even if you have some autoscaling solution, it probably won't be able to bring new servers up in time. Serverless provides a solution for that case as well, since you have almost unlimited resources.
But yeah, serverless is a tool for solving a certain set of problems. This idea of the "Serverless Revolution" was kind of silly from the start
Not really, since it takes a few seconds to spin up all the serverless instances, so your app response becomes really erratic.
Then you need some magic to deal with database connections from 1000 lambda functions. All of them use their own since they cannot pool.
what platform?
If the next lambda invocation happens within minutes of the previous one ending, you can carry forward a db connection from the older lambda, no magic needed. Just place your db connection object in global scope (nodejs).
Now if you somehow get a spike of requests to some endpoint that goes over your DB connection limit, suddenly all your newly scaled functions fail because they cannot get a connection to the database.
AWS added a service for RDS to deal with this, but it all just feels like a big kludge to deal with a problem that shouldn’t have to exist in the first place.
You will have to analyze your burst rate and the lifetime of each request to figure out the size of your connection pool.
I also am not clear how AWS introduced the problem of not being able to connect to the DB. We've known about the need for connection pools and connection reuse from before AWS was a thing, no?
A process is given a connection from a pool when it requests the connection and holds it until it relinquishes the connection regardless of whether the process was in the logic or data access portions of their lifecycle. It is up to you as the developer to ask for the connection and to return it on an as-needed basis...Which is exactly what you should be doing in a lambda as well.
Just because a lambda container is persistent does not mean the connection given to it is stuck to that lambda even after that lambda returns the connection.
With a connection pool, you don’t need to hold onto a connection longer than necessary, but you are right that you could implement it in such a way that you hold it for life of the request.
How many people have that use-case but not in multiples? How many of those people have such a use-case but would not be better served by platforms such as IFTTT or webhooks on other services?
I guess if we count fargate as serverless, it's much better for this use case.
I'm sure there are other ways of doing it, but the way I do it is using serverless framework (is a nodejs app) together with serverless-python-requirements plugin. The plugin understand poetry (make sure you don't leave requirements.txt because it will use that, you don't need that file if you use poetry).
Anyway it has warts, one big thing is that nodejs developers don't care about norms and standards and constantly reinvent the wheel so for starters you're pretty much forced to create project.json project-lock.json files, which then will create node_modules directory in root of your project. Then you need to start project through npx command (I believe you could still install serverless globally but looks like that's being depreciated).
The node_modules directory will contain over 200MB of javascript code after you install it.
But after all of those things, when you get it to work it is not terrible.
I've created a whole photo sharing website in AWS Lambda, including a complete user accounts system (register, login, forgot password, email verification, profile photos, etc), social aspects, as well as other things like youtube video search and transcoding in the same system.
I also had to create my own Lambda build system because what was out there wasn't what I wanted. I wanted to be able to hit save on a file and have it repackage my Lambda on AWS (using a Lambda to create the lambda) including shared code in 'Layers'. Kind of a "live-reload" for Lambda. It's all working very efficiently.
This was a lot of effort though, but now that I've got a basic system I can extend it to any kind of website.
My photo sharing site for friends costs me about $0.25 every few months, and that is mostly/all the cost of storing gigabytes of photos on S3. No, it doesn't see a lot of traffic, and that's exactly why I chose Lambda to build this site on, because I don't have to run an EC2 instance for my photo sharing site 24/7 if nobody is using it. It's worked out exactly how I wanted, costs me practically nothing to run every year.
And, if I did build a system with a ton of active users, Lambda handles the scaling of that. Another reason I spent the time to create this build system and write all the code to handle user accounts, etc.
This is a bold claim to make without verifying it.
> Another reason I spent the time to create this build system and write all the code to handle user accounts, etc.
But did you also spend the additional time to scale test it and verify it won’t break in confusing and annoying ways when scaling / spiking up? (see the other discussions re: connection pooling above).
The reason a lot of folks are skeptical is that in practice there don’t seem to be a lot of great examples of stateful services using a “pure” serverless lambda pattern but with arbitrary dependencies that actually do scale smoothly and don’t break in annoying ways such that you end up building something more complicated than just using App Engine or Heroku...
I mean, Lambda's ability to scale is a well documented thing. It will happily do more if you ask it to.
The real scaling problem is your wallet, because Lambda is damn expensive once you have any somewhat serious compute requirements.
I'm not sure I understand your point about not needing a deployment chain. I developed a framework (open sourcing it soon) using Jenkins Pipeline DSL for deploying Lambda functions and its very useful and helpful.
Regardless of the target environment the reasons for using a deployment pipeline are still there even if you use a server less platform.
Can't wait for codeless, zero code, nocode, whatever name it gets called... something where normal people can build their own ideas without having meaningless discussion about tdd, frameworks and etc, nor the need to maintain codebases, ci/cd and all the jazz
"Low-code" is the buzzword to search for. It's been around a while now.
Of course like all the things the idea has existed a long time, but "low-code" development services seem to be increasingly popular and visible in the last couple of years, and they do seem to be getting more advanced and ergonomic.
Of course, if you already have servers with something like k8s or Swarm running, you might deploy a container there instead.
The wins also didn't seem great enough to overcome these twin beasts.
We've come full circle.
I do see a big potential for small and simple projects though. No need to setup instances and worry about networking: just deploy it and go.
People realising that it's just another tool in the toolbox rather than 1 tool that can replace their entire toolbox.
Much of the cynicism seems to come from that.
I do agree with jason though - open ended serverless things charged per use are no fun. Now you can bankrupt yourself at scale with that bug & the 1000 instances.
Are any of them? Stuff like cloud run is very competitively priced when assuming base case...but by its very nature it can scale up near infinitely. And with it bugs & bills.
The ability to cap things by 2 order of magnitudes would make me sleep much better. (1 would be better).
Currently, FaaS platforms should be viewed like a cron. The majority of the benefit is if the fn is called in the low thousands per day.
Whenever I read these articles, I always wonder what proportion of tech buzzwords from the past decade are just obscure ways of saying "someone else's servers", "automation" and "unnecessarily complicated architecture".
The way people talk about it, I certainly get the impression that a lot of people think that way.
But I agree with you in full otherwise - there is no way that Lambda is going to completely usurp all forms of computing in the cloud.
But nowadays, I just setup my K8 templates with auto-scaling groups on GKE, and it pretty much works just as well as serverless was promised. Compute demand gets seamlessly transformed into compute supply.
I almost never have to think about managing individual hosts. And the cost is cheaper and performance is more predictable than serverless. Plus it's mostly all platform agnostic.
It's a silly term, but successful. If anything, arguing against it made it more successful through mere repetition of the word.
You need to move a few pieces around.
I don't agree that 'once you've gone K8 why bother, just use that' - I don't think K8 is as elegant as the promise of serverless, and it has it's own 'lock in'. Just so happens you may need to have K8s anyhow ... so the pragmatic question then is 'We already have K8s because we have to have it ... so in that context, we can just use it'
If we had a 'severless' version of Docker, i.e. some de-facto standard for it, I think it would obliterate a lot of architectures, just because the promise is powerful.
Developers don't actually want K8s or even true DevOps complexity, we just want a giant computer we can run stuff on and not have to worry.
At least on AWS, the "SAM" experience has been probably the worst development experience I've ever had in ~20 years of web development.
It's so slow (iteration speed) and you need to jump through a billion hoops of complexity all over the place. Even dealing with something as simple as loading environment variables for both local and "real" function invokes required way too much effort.
Note: I'm not working with this tech by choice. It's for a bit of client work. I think their use case for Serverless makes sense (calling something very infrequently that glues together a few AWS resources).
Is the experience better on other platforms?
I tried AWS, and then IBM's offering which is based on an open source (Apache OpenWhisk) project, thinking that it might be easier to work with, but that was also a pain.
I just lost interest as I was only checking it out. For something constantly marketed on the ease of not having to manage servers, it fell a long way short of "easy".
Look into Firebase functions. Drop some JS in a folder, export them from an index.js file and you have yourself some endpoints.
> exports.helloWorld = functions.https.onRequest((request, response) => { response.send("Hello from Firebase!"); });
The amount of work AWS has put in front of Lambdas confuses me. Firebase does it right. You can go from "never having written a REST endpoint" to "real code" in less than 20 minutes. New endpoints can be created as fast as you can export functions from an index.js file.
Being able to throw up a new REST endpoint in under 10 minutes with 0 config is really cool though.
And Firebase Functions are priced to work as daily drivers, they can front an entire application and not cost an insane amount of $, per single ms pricing. Lambda's are a lot more complicated.
Firebase is magic... but I never recommend it for anyone, until there's some sort of migration path.
With Cloud Firestore (our serverless DB) that’s the case as well. And Firebase Auth can be seamlessly upgraded to Google Cloud Identity Platform with a click.
However you’re right that for many Firebase products (Real-time Database, Hosting) there’s no relation to Cloud Resources.
To be fair, Firebase recently released a local development tool which alleviates the need to deploy on every change, but I haven't used it yet.
https://firebase.google.com/docs/emulator-suite
Super useful though!
It's a little incomplete, missing some of the AWS IAM automation that makes local development smooth, environment management for testing and promoting changes, and some sort of visualization to make architecture easier to design as a team.
I work for a platform company called Stackery which aims to provide an end-to-end workflow and DX for serverless & CloudFormation. Thanks for comments like these that help identify pain points that need attention.
What I have now is a loosely coupled, serverless, frontend+backend monorepo that wraps AWS SAM and CloudFormation. At the end of the day it is just a handful of scripts and some foundational conventions.
I just (this morning!) started to put together notes and docs for myself on how I can generalize and open source this to make it available for others.
stack is vue/python/s3/lambda/dynamodb/stripe but the tooling I developed is generic enough to directly support any lambda runtime for any sub-namespace of your project so it would also support a react/rails application just as well.
Sure, you can write a bunch of tools to work around the crufty, terrible development environment's shortcomings. But ultimately, you are just locking yourself further & further & further in to the hostile, hard to work with environment, bending yourself into the bizarre abnormal geometry the serverless environment has demanded of you.
To me, as a developer who values being able to understand & comprehend & try, I would prefer staying far far far away from any serverless environment that is vendor locked. I would be willing & interested to try serverless environments that give me, the developer, the traditional great & vast powers of running as root locally that I expect. Short of a local dev environment, one meets both vendor lock in, & faces ongoing difficulties trying to understand what is happening, with what performance profiles/costs. I'd rather not invest my creativity & effort in trying to eek more & more signals out of the vendor's black box. Especially if trouble is knocking, then I would very much like to be able to fall back on the amazing toolkits I know & love.
That isn't a compliment.
Time will tell.
but lambda gets to the point where there is no local parity, where it's detached, is no longer an easier managed (remotely operated) parallel to what we know & do, but is a system entirely into itself, playing by different rules. one must trust the cloud-native experience it brings entirely, versus the historical past where the cloud offered native local parallels.
[1] https://github.com/openfaas/faas
If you are working on a project where all of your infrastructure will live on AWS I would definitely urge you to give it a second look. The amount of infrastructure I manage right now with a single .yaml file is really killer.
Personally for simple projects I've had pretty good experiences writing a Yaml-based template directly, and for more complex projects I use Troposphere to generate Cloudformation template Yaml in Python.
Did you chose python for backend dev? Any framework like flask or django?
Also no preference of single-language for both backend and front-end, by using node.js backend?
Just trying to get into web-dev.
If I could do it all over, I would still choose Python. That being said, I have been working professionally (building apps like this) for almost 14 years so my willingness to bite off a homebrew Python framework endeavor as I did here is a lot different than someone just getting into the field.
Django: avoid unless you have a highly compelling (read: $$$$) reason to learn and use this tool. I cannot think of one, honestly.
Flask: fantastic, but be conscientious about your project structure early on and try to keep businees-logic out of your handler functions (the ones that you decorate with @app...)
Sophisticated or more sugary Node.js backends are not something I have ever explored, aside from the tried-n-true express.js. I tend to leverage Python for all of my backend tasks because I haven't found a compelling reason not to.
You have to roll your own way too often in Flask et al, so much so that I don't see any reason to use Flask for anything other than ad-hoc servers with only a few endpoints.
All-in-all, Django is not bad software. I have a bad taste in my mouth though because as I learned and developed new approaches to solving problems in my career I feel like Django got in the way of that.
For instance, there are some really killer ways you can model certain problems in a database by using things like single table inheritance or polymorphism. These are sorta possible in Django's ORM, but you are usually going against the grain and bending it to do things it wasn't really supposed to. Some might look at me and go: ok dude well don't do that! But there are plenty of times where it makes sense to deviate from convention.
That is just one example, but I feel like I hit those road blocks all the time with Django. The benefit of Django is it is pre-assembled and you can basically hit the ground running immediately. The alternative is to use a microframework like Flask which is very lightweight and requires you to make conscious choices about integrating your data layer and other components.
For some this is a real burden - because you are overwhelmed by choice as far as how you lay out your codebase as well as the specific libraries and tools you use.
After your 20th API or website backend you will start to have some strong preferences about how you want to build things and that is why I tend to go for the compose-tiny-pieces approach versus the ready-to-run Django appraoch.
It's really a trade-off. If you are content with the Django ORM and everything else that is presented, it is not so bad. If you know better, you know better. Only time and experience will get you there.
Minimalist frameworks are great for either very small (since they don’t need much of anything) or very large projects (since they will need a bunch of customization regardless).
In that regard, I think Django is kind of like the Wordpress of Python.
I don’t think it’s as bad as people make it out to be.
I've been pretty happy with Cloudflare Workers.
You can easily define environments with variables via a toml file. The DX is great and iteration speed is very fast. When using `wrangler dev` your new version is ready in a second or two after saving.
I was planning on trying out FastAPI + Zappa, but have not gotten round to it yet.
Disclaimer: I work at AWS, however not for any service or marketing team. Opinions are my own.
Of interest, I've spent some free time crunching on CNCF survey data over the past few months. Some of the strongest correlations are between particular serverless offerings and particular delivery offerings. If you use Azure Functions then I know you are more likely to use Azure Devops than anything else. Same for Lambda + CodePipeline and Google Cloud Functions + Cloud Build.
This is what I tried to do initially after experiencing the dev pain for only a few minutes.
But unfortunately this doesn't work very well in anything but the most trivial case because as soon as your lambda has a 3rd party package dependency you need to install that dependency somehow.
For example, let's say you have a Python lambda that does some stuff, writes the result to postgres and then sends a webhook out using the requests library.
That means your code needs access to a postgres database library and the requests library to send a webhook response.
Suddenly you need to pollute your dev environment with these dependencies to even run the thing outside of lambda and every dev needs to follow a 100 step README file to get these dependencies installed and now we're back to pre-Docker days.
Or you spin up your own Docker container with a volume mount and manage all of the complexity on your own. It seems criminal to create your own Dockerfile just to develop the business logic of a lambda where you only use that Dockerfile for development.
Then there's the whole problem of running your split out business logic without it being triggered from a lambda. Do you just write boiler plate scripts that read the same JSON files, and set up command line parsing code in the same way as sam local invoke does to pass in params?
Then there's also the problem of wanting one of your non-Serverless services to invoke a lambda in development so you can actually test what happens when you call it in your main web app but instead of calling sam local invoke, you really want that service's code to be more like how it would run in production where it's triggered by an SNS publish message. Now you need to somehow figure out how to mock out SNS in development.
Serverless is madness from start to finish.
SAM already provides a way to mock out what SNS would send to your function so that the function can use the same code path in both cases. Basically mocking the signature. This is good to make sure your function is running the same code in both dev and prod and lets you trigger a function in development without needing SNS.
But the problem is locally invoking the function with the SAM CLI tool is the trigger mechanism where you pass in that mocked out SNS event, but in reality that only works for running that function in complete isolation in development.
In practice, what you'd really likely want to do is call it from another local service so you can test how your web app works (the thing really calling your lambda at the end of the day). This involves calling SNS publish in your service's code base to trigger the lambda. That means really setting up an SNS topic and deploying your lambda to AWS or calling some API compatible mock of SNS because if you execute a different code path then you have no means to test the most important part of your code in dev.
> In the case of Postgres, you could use an ORM that supports SQLite for dependency-free development but at a compatibility cos
The DB is mostly easy. You can throw it into a docker-compose.yml file and use the same version as you run on RDS with like 5 lines of yaml and little system requirements. Then use the same code in both dev and prod while changing the connection string with an environment variable.
> That’s not exactly a knock against serverless itself, is it?
It is for everything surrounding how lambdas are triggered and run. But yes, you'd run into the DB, S3, etc. issues with any tech choice.
And yes, I think the problem here are APIs you use in multiple places but can’t easily run yourself in a production-friendly way. Until AWS builds and supports Docker containers to run their APIs locally, I don’t see how this improves... end to end testing of AWS requires AWS? ;-)
It’s hard to do this when surrounding code doesn’t accommodate the approach, but it’s great way to design an application if you have the choice. I really love sandboxing features before moving them into the application. Everything from design to testing can be so much faster and fun without the distractions of the build system and the rest of the application.
There's a couple of sharp edges still but in general it just 'makes sense'. If you don't like TypeScript there are also bindings for Python and Java, among others, although TypeScript is really the preferred language.
Currently using some CDK in a production app and finally I found a way of doing IaC I actually enjoy.
CDK is moving quite fast and not all parts are out of the experimental phase, so there are breaking changes shipped often. I think in a couple of years it will stabilize and mature and become a very productive way of working with infrastructure.
Have you tried Amplify?
Unfortunately, I quickly hit the limits of its configurability (particularly with Cloudfront) and had to move off it within a few months.
> serverless invoke local
All the time for doing testing. You can pass in a json file that contains a simulated event to test different scenarios. Works very well for us.
I believe they decided to re-create their own credentials chain and a lot of things are not working with MFA (like support for credential_process).
Edit: CloudFormation was also painful for me, the docs were sparse and there were very few examples that helped me out.
Over 5000 pages of documentation
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
> very few examples that helped me out
AWS provide an example for most common services, plus there are thousands of other community supplied examples out there.
https://github.com/awslabs/aws-cloudformation-templates
Yes there are examples, but there wasn’t one at the time that mapped to what I was trying to accomplish. Because, again, SAM templates are not one-for-one CloudFormation templates.
I found the community around SAM to be very limited. One of the many reasons I’ve moved to the Kubernetes ecosystem.
Honestly, it reminds me of PHP development years ago: running it locally sucked, so you need to upload it to the server and test your work. It. Sucked.
With SAM we're talking ~6-7 seconds with an SSD to build + LOCALLY invoke a new copy of a fairly simple function where you're changing 1 line of code, and you need to do this every time you change your code.
That's even with creating a custom Makefile command to roll up the build + invoke into 1 human action. The wait time is purely waiting for SAM to do what it needs to do.
With a more traditional non-Serverless set up, with or without Docker (using volumes) the turn around time is effectively instant. You can make your code change and by the time you reload your browser it's all good to go. This is speaking from a Python, Ruby, Elixir and Node POV.
(I'm still doing the "upload to server to test" thing.... I've tried MAMP and Vagrant/VirtualBox for local dev but both of them seem horribly complex compared to what we can do with local dev with node.js/mongo and so on.)
Of course this won't fix the fact that you have a lambda behind api gateway that does some heic->jpg conversion and can't be hit outside the DMZ, or some esoteric SQS queue that you can't mimmic locally - but it should get you almost there.
sudo apt install nginx php mysql, point the www directory of nginx to something on your computer (/mnt/c/projects/xyz) and you've got a running setup. Or run Linux in general, that's what most people I've seen work on backends seem to do. You can run the same flavour of software that your deployment server runs so it'll save you time testing and comparing API changes or version incompatibilities as well.
I don't know any solution for macOS but you can probably get the necessary software via Brew if you're so inclined. Then run the built-in PHP web server (php -s) and you get the same effect.
Apache is almost as easy, download zip, unpack & configure apache conf to use your php.
MySQL is somewhat more complicated because you need to run an setup script after unpacking the zip.
I tried AWS Lambdas a few years back and it felt way more primitive.
Yes, SAM sucks. Boto3 sucks. I mean, yes, the AWS APIs are not the most obvious or ergonomics thing.
[0] https://www.serverless.com/
TJ abstracts the Lambda nonsense so you just get to use a standard HTTP res/resp interface. https://github.com/apex/up-examples/
Deployment iterations for me in Singapore can be as low as 2s, but tbh I use https://github.com/codegangsta/gin locally for development iterations.
What does it mention?
Still very actively working on this but our goal is that nobody ever has to test via deploy again.
Slightly OT... perhaps cheeky... any idea why firebase doesn’t provide automated backups for firestore and storage out of the box? Seems like a no brainer and a valuable service people would pay for.
DISCLAIMER: I used to be a developer of this product (it's from the previous company I've worked)
My only nitpick, and only specifically relating to dotnet, is that config files and env vars differ between Functions and regular ASP.NET Core web apps. I think there is some work going on to fix that, but it's taking forever.
The biggest issue we run into is the more servers you have, the harder it is on the database.
- performance: first rule of architecture is that you don't build the entire thing on the needs of high performance...you only address performance as-needed
- vendor lock in: this is the worst reason. Most companies choose cloud vendors and stick with them over many years. I'd love any survey that showed some massive migration between vendors on a regular basis, but it does not exist.
- the monolith: granted, serverless is really best when building something from scratch. Trying to "migrate" to serverless is just a bad idea. Trying to "upgrade" portions is okay, but do it well using the autonomous bubble pattern, making sure you have a solid translation layer
I believe serverless is our best modern architecture, but I also understand that not every company is ready to do the work and/or pay the price of the change in development processes. I'd agree that it's stalled, but moreso out of fear and ignorance than anything else.
Why do you think companies stick with vendors for so long instead of switching?
Isn't this a sign of existing lock-in?
Also, I think lock-in comes in degrees. Some are worse than others. Even if we're slightly locked in right now, we can easily make it worse if we try.
How about simple common sense instead of "fear and ignorance". I have a C++ monolith that serves me just fine already for years. The performance is great and I am able to rent dedicated servers that can serve at least 10 times more requests that I have at a price that is a tiny fraction of AWS for the same work.
Why would I switch and incure more expenses? What's the ROI?
Vendor lock-in is magically valuable, until it is not.
The problem isn’t lock-in per se: the problem is choosing lock-in that will bring surplus extra value for an extended period of time.
https://www.cloudtp.com/doppler/is-vendor-lock-in-keeping-yo...
We’re not buying into IBM mainframes anymore and although there are still customers tied to Oracle databases, vendor lock-in just isn’t that big of a deal anymore.
The broader idea is a service (or static site even) sending messages to a different service. Given how popular no-code platforms are getting, tiny serverless pieces can help automate operations between these platforms.
Second - with each cloud provider's serverless service, I've noticed that the underlying complexity is shifted into your application and provisioning. They're a great example of leaky abstractions. The name 'serverless' may lure you into thinking that you don't need to think about servers any more, but actually you do have to adjust to the way they've implemented a service that you're using. Think of Lambda cold starts, or have a look at the DynamoDB pricing page. There's still a server running somewhere.
Personal experience: I've found that ECS (Fargate) is a decent step into the serverless realm, if you've been running normal containers on normal EC2s. It does take away the EC2 management aspects, and it's especially useful if you have some applications that are resource hungry, and some are not. It's a cheap way to run ETLs (we use the ECS Operator from Apache Airflow, which itself is on EC2). It doesn't take long to learn, and it's a good place to start. It does come with its own baggage... the 'stack' that you had on a single EC2 is now scattered about; you may need AWS Cloud Map for service discovery, ALBs for networking, SSM/KMS for secrets, Cloudwatch for logging.
That's when you realize, those sticker infested laptops at Devops conferences are actually modern architecture diagrams.
The APIs cost a bit more a month (they’re on quite low cpu/memory) because they’re always on, but it minimises complexity and it’s nice to know that you can move those containers over to another platform if needed.
Actually, re-reading your comment, I'm not sure what you were trying to say. I may be agreeing with you.
---
I think it is possible for serverless to supplant PaaS et al, but I don't know if it's quite mature enough yet.
For stateful systems such as databases, that typically require complications like replication and backup, configuration, testing and maintenance is hard to get right.
But for stateless systems, IMO infrastructure is pretty straightforward.
I'm a fan of OpenFaas: you get the benefits (but not all the drawbacks) of containers and serverless. It's easy to mix and match running 100% locally, or mix-in baked 3rd party components, or running in Kuberenetes or other systems in production. Also, no vendor lock-in.
Some degree of vendor lock-in is pretty much inevitable no matter what solution you go with.
Limited programming languages may have been an issue a few years ago, but with the support of Java, Go, Python, Javascript to name a few - most of the core user bases are covered.
Additionally, after the introduction of API gateway, most CRUD applications, including those that require async tasks, can be served by the lambda and serverless.
After having used serverless for a while, the biggest turn off for using it (still) seems to be the performance issues. Provisioned concurrency, i.e leaving function containers running, is really just a bandaid and runs counter-intuitive to the original motivation of serverless to begin with.
Not yet a fan.
https://registry.terraform.io/providers/hashicorp/aws/latest...
The biggest annoyance is unit testing, although it forces certain good habits on us. So for example, since we don't use SAM, we have to do unit testing separately (our code is in Python). But because we can't exactly run a lambda via Python, we have to put most of our code in the layers that our lambdas import.
So we run unit tests on the layers, and the positive is that we can essentially copy and paste the code that our lambdas would run into the unit tests. And then we can keep it DRY because almost all the code is in layers, and that incentivizes us to create generic functions that can be used across all of our lambdas.
For things like unit testing DynamoDB, we use Moto which works exceptionally well.
https://pypi.org/project/moto/
More detail:
Our terraform has infrastructure code for:
API Gateways
Hard-coded API keys we use for testing
DynamoDB tables
IAM roles used by the Lambdas and Step Functions
Lambdas themselves
s3 buckets used for the Lambda zip files and state
Step functions
We did this so we could totally separate our applications and the infrastructure as code stuff since infrastructure changes at a drastically slower rate than applications.
So for example, if we have a service deployed in an EC2 instance, we have to update the instance(s) every month by a certain date. Sometimes, we have to update early if there is a severe security issue. We also have to manage things like security software that has to run on every instance, logging for soc2 compliance, access controls, manage ssh keys, etc. etc.
That's a lot for our engineers to handle every single month. A lot of context switching. It's not so bad if we only have one service to manage. But the actual issue is that we have many services to manage. So every month, our board had about 12 stories related to patching. Patch service X development. Patch service x staging. Patch service x production. Patch service Y development.... and so on.
For serverless, the team responsible for SOC2 compliance gets the reports from AWS without us ever having to do anything. No patching stories, no security software that breaks on occasion, no access management, etc.
All we have to worry about in terms of SOC2 compliance are the basics like principle of least privilege (i.e. lambdas do not have full access to DynamoDB), no public endpoints, we use VPCs blessed by the security team, and that's about it. Things we're used to doing everywhere else and things that we only worry about when writing the terraform.
Now when I add a Lambda, I only worry about writing the Python code for the lambda and copying the terraform code. That's it. There is no patching story for this service.
And yes, of course we're passing the buck. That's kinda what we pay for in the premiums that AWS gets from our massive bill every month. That's kind of the entire point for cloud-based systems, anyway. We're passing the buck on a ton of other things, like not having to manage hard disks, not having to manage network cables, not having to manage a building, not having to manage HVAC, not having to manage backup batteries, etc.
I like the term "nano services" is it probably the wrong term. But I like it.
We went monolith -> micro services -> micro services + nano services
Everything that adds complexity in a micro service system, adds far more complexity with nano services.
Debugging and tracing is a nightmare. And yes, logging, a lot of logging and instrumentation helps. But if you add too much of it then the whole point of the nano services is lost.
Versioning can get complex, depending on your deploy model. Presumably if you push 100% of your nano services with every deploy you can get around it.
We have about 600 nanoservices running on AWS now across 4 different systems.
My advice, that nobody would ever ask for, keep AWS Lambda for certain special tasks where they are a very good fit.
As always run the numbers, performance, cost to run, cost to maintain. Will the immense ability to scale really help your use case?
In most of our cases, no, they will not.
For some companies it will be a great solution and solve pain-points, save money