Ask HN: Does anyone else find the AWS Lambda developer experience frustrating?
I've been using AWS lambda a bit recently, mostly as a way to glue together various bits and pieces.
Maybe I'm doing this wrong but does anyone else find the experience to be really frustrating?
I can unit test bits of the code just fine, but at some point I always end up stuck in a slow feedback loop where I deploy the code, do some manual invoking, go and dig through the logs in CloudWatch, add another print statement in my lambda... and so on.
What I want is to run the lambdas locally, ideally more than one, and then exercise them with streams of test events (perhaps captured from a real environment). It would be quite cool if I could define BDD style tests around them too.
Anyone have any suggestions or share my frustrations?
I have heard localstack is quite good although I haven't given it a go yet. Would that work for me? I did try SAM but I was a bit underwhelmed and I don't want to use a separate IaC tool for these.
Alternatively, do other FaaS providers solve this problem?
Thanks for any help.
285 comments
[ 4.4 ms ] story [ 261 ms ] thread[1] https://github.com/lambci/docker-lambda
As you say, it only does the 'host lambdas locally' bit though. If you wanted to do stuff like test the interactions between lambdas, or test them over HTTP (for lambdas that act as components in REST APIs), you'd need to build something yourself.
You’re not testing that HTTP works, you’re testing that the two functions work together. If you can abstract a calling interface between them with different backends you’ll have a better time developing lambdas, while also not being tightly coupled to them.
Create an RPC container object that has method stubs for the other lambdas you want to call. When you deploy it on AWS configure it to make an HTTP call; when running it locally, configure it to just call the code as normal.
Just think of the HTTP call and JSON blob as the calling convention. The API is really simple.
I think one of the biggest mistakes people make when developing Lambadas is designing them differently to normal software.
That's probably very true (and I'm probably quite guilty of it!).
Thanks for your suggestions, that does sound like a good approach.
But if your function more or less resembles a little web app that maybe talks to S3 and DynamoDB, this works really well.
I'm using python mostly atm. I've played around with go/node too (both of which I'm familiar with in and of themselves) and I was thinking about switching over to one of those already.
The dependency management is easier for both of them than with python I found.
AWS has Step Functions, which is a vaguely functional and declarative language for linking your lambdas together. It’s a bit verbose and annoying to write, but at least a it makes your lambada setup repeatable.
The state machine which actually runs your step functions is publicly available.
https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...
What people need to understand about testing against AWS is that the SDKs provides the typing guarantee. So it’s nearly impossible to make an invalid call with any AWS SDK.
Moreover, doing a test along the lines of ‘post to S3, then get the object to verify it has been posted’ is pointless too. Because that’s not how AWS works. In a real application, you don’t have any guarantees the object exists in S3 immediately afterwards, just a high probability that it will become consistent relatively quickly. If you do such a thing in a real world environment, you’re liable to have intermittent failures.
The eventually consistent behaviour is more obvious with other APIs like the EC2 interface, where no one expects an instance to be created immediately.
There are a few exceptions to this rule, but they’re not accurately modelled by localstack.
What people need to test when developing AWS applications is that there _was_ a call to an AWS API, not that the AWS API behaved with all of its documented side effects. You can’t unit test the black box of AWS, you have to QA it as part of an application release.
Cosmotic's comment kind of confirms my suspicions to be honest.
It's not a first-class experience, but you can get into a groove with it and for 90% of projects (which need sparing use of Lambda functions for specific purposes).
As an example, I use Hasura for some of my backends, and it requires you to set up authentication somewhere else (e.g. Auth0). I usually set up a JWT based authentication module on AWS lambda that works specifically for the given project, and I use serverless to get that done pretty quickly.
I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live without having to mock AWS services locally. Here’s a short demo (https://youtu.be/hnTSTm5n11g)
Cloudflare's Workers have a good dev experience. It takes a second to build and locally preview a function. It live-reloads when you make a change. Later, it takes a second to build and deploy. On the flip side, Workers can't run arbitrary Docker containers. Your code has to be JS/TS or WebAssembly.
I'm not affiliated with Cloudflare, just a satisfied customer.
Containers are coming to the Cloudflare edge: https://blog.cloudflare.com/containers-on-the-edge/ (inevitable really)
Nit: Workers should really be compared to Lambda@Edge instead.
Am I right in saying that they serve a somewhat narrower use case than lambdas though? My understanding is that they are mostly used for things like modifying requests/responses rather than for building a whole backend.
My experience has been the opposite.
Wrangler doesn't give you any meaningful errors (since it converts JS errors to Rust) and it crashes way too often.
Recently I worked on a Workers project and I had an undefined missing variable. The only error Wrangler would output is "error".
IMO Workers are only viable for really small and simple projects.
Literally today I had a billing problem with AWS and it was one of the worst, broken UI experiences I've ever seen. Involving money to boot (usually those things have higher standards).
They are really bad at quite a lot of things, it is what it is - there might be some rhyme or reason to it, for example, they might not care about documentation, or may be much more focused on bread and butter ec2, bit it's totally plausible that there are a lot of frayed edges.
You know that difficult-to-attribute quote: 'Don't attribute to malice that which can just as easily be attributed to incompetence'.
Here’s a short demo (https://youtu.be/hnTSTm5n11g)
I'd love for you to give it a try.
If I'm honest, I do find AWS in general fairly high friction at times but it's way better than renting servers directly I think.
It's the only cloud provider I've really worked with to be honest though (other than a few experiments), maybe some others are much better in this regard?
I don't really get where you're coming from on renting servers directly, using AWS is far more expensive if you actually need a server for a while.
Though I do admit being able to fire up a server for a couple of hours and then delete it is pretty awesome.
The operational overhead of provisioning them, configuring them (and so on) is really difficult for a lot of teams though.
I find it more simplistic that Digital Ocean , tbh
I’ve been thinking a lot about AWS lately and it definitely seems to me like a modern take on what IBM did in the 80’s.
The purpose of the AWS Console, AFAICT, is:
(1) To be used by semi-technical ops and management types, and people doing tutorials, and
(2) To drive everyone else as quickly as possible to the APIs/CLIs/SDKs.
I worked on Netlify Dev, which does offer a simulated lambda environment for testing locally: https://news.ycombinator.com/item?id=19615546
I was also quite disappointed to see that AWS Amplify did not offer a running dev server for lambdas, best you could do was a local onetime manual invoke and even that often failed (mismatched to the live lambda environment where it succeeds)
We had to detect if running locally or deployed in our code because Cognito auth was giving us trouble when running locally, and Step Functions can only be tested deployed.
Cloud Watch logs take several minutes to appear, funcs take some time to package and deploy (even with caching) - making feedback loop very slow.
I personally think the best way is to develop serverless funcs on platform - using the built in editor. You get logs instantly. But can't get devs to give up their IDEs...
It uses AWS Lambda underneath but it makes the dev experience just awesome.
AFAIK even though it's running AWS Lambda, the cloud functions can't be integrated directly with other AWS services.
In my own work with serverless I think about this a lot. In my day job, we built a new service and we pay AWS a few dollars per month to run. It would have cost us around $100 a month in AWS costs. However, the operational complexity is extremely high and we are also coupled to another service via Kinesis. For a billion dollar business, the trade off doesn't seem worth it.
I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors. It certainly hasn't made my life as a developer easier. It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).
Appropriate for a small amount of apps, but used inappropriately by a lot more for a while because it's the hot new thing.
And if one doesn't need an API first design then SSR with a robust framework might be cheaper to stand up and maintain.
It depends on what you mean by "serverless". Serverless webapps? Sure, maybe a fair bit. But the real killer feature of the Lambda service is that it's a universal extension mechanism for other AWS services. Want to run some shoddy virus scanning whenever an object is added to a bucket for compliance reasons? That's a lambda. Want to manipulate Firehose records before they are sent to the sink? That's a lambda. Want to execute some crazy Python function from an Athena SQL query? You guessed it. That's a lambda.
So when you say "It certainly hasn't made my life as a developer easier" it means you haven't needed to do these bespoke gluing together of stuff, because "being able to do this somehow" is infinitely easier than "not being able to do it at all because AWS hasn't exposed an API for this use case".
It’s a hammer, and if it’s a good fit for your particular nail is very dependent on it’s shape.
That, and also offload some somewhat computationally expensive background tasks. Instead if screwing up your deployments by ramping up CPU and memory utilization to run a one-off task, just let he service fire a lambda and, if needed, just act when the lambda replies back.
I think the true geniuses are the ones who spent so much time trying to sell people on "Server Admin is hard, lets hire it out!" as well as building the billing and metering systems.
In a gold rush, screw panning, sell shovels (and pans).
This statement is a red herring. People don't flock to AWS because they want to use lambdas. Lambdas are adopted by AWS customers who already have their data in S3 or DynamoDB and/or already use one of AWS message brokers, and happen to stumble on a requirement for a background that runs asynchronously in response to an event, such as someone uploading a file to a bucket.
When that usecases pops up, the decision process consists of either adding a brand new service to be ran on ECS, add a background task on a service already being developed, or simply write the event handler and deploy it somewhere for AWS to run without us bothering about scaling and maintaining stuff. Writing and deploying the handler is an order of magnitude simpler to deploy and maintain, which happens to win over some technical discussions.
I started off with GCP, Cloud Storage and Cloud Firestore, but it was such a hassle and in the end, I couldn't even manage to run my Cloud Functions. The 10s cold start time was another non starter factor that led me to switch.
Honestly, I don't see myself staying with DynamoDB or AWS lambda. I definitely prefer local-hosted data stores such as on a DO droplet, and I would have started off with them were it not for their shoddy service. AWS service has been extremely strong so far.
IAM (consistent identity and access control across all of your estate) and KMS (at rest encryption of all data, partitioned by service) are two examples of two things that would be very hard to replicate on your own hardware.
For example: Setting up different SSH keys when deploying a machine is not hard; getting granular ACL is more complicated.
Getting encryption at rest is very service specific, for example Postgres has TDE (free, with paid support if my memory serves correctly); I'm sure there are some services where EaR has been implemented by aws and there is no OSS alternative.
Base functionality is relative. The base functionality of IAM is that every server, pod, user or AWS service has an centrally managed identity controlled via RBAC that is used to access any other AWS resource, requires no long-lived credentials, powered via a flexible policy engine that can rely on principal and resource attributes, with updates to policies applied in seconds across the globe. The actual advanced functionality is more complex and useful in certain situations.
If your benchmark for base identity functionality is “different SSH keys for different servers” then I’m not sure what to say.
The same for encryption - the point is that it’s not very service specific with AWS. That’s the base functionality: encrypt this data (queue, disk, object, database, backup, whatever) with this key. Done. Consistent, transparent encryption at rest for every service, access controlled through IAM roles and policies.
Instead I suggested writing a Lambda service in Go and to run it incrementally (I think it was hourly and monthly). It worked out to tens of dollars per month, basically free. [At first it cost much more than this because the original implementation wasn't specifying known sizes for initializing collections that was constantly being resized in several places in its pipeline resulting in copying/double consumption plus severe fragmentation.]
The development, in particular versioning and deployment sucked back then. It's surprising to still hear mention of challenges so much later.
The complexity bogeyman is a red herring. What is software development if not a constant fight between increasing complexity and adding features, including making systems more robust and resilient and reliable?
Sure, you are free to claim that function-as-a-service paradigm is very complex because the code runs somewhere else, and you have to use a separate workflow than the one you're used to, and that you actually have to write unit tests to have some assurances that stuff does work as expected.
But what would be the alternative approach?
Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.
AWS Lambda is a fancy way to let developers develop event-driven systems by just writing tiny event handlers and plug in events. It's no rocket science. In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt. I'm terms of web services, they are a service where you just have to write the request controller bit after that runs after all auth things passed and request messages were parsed and validated. Considering this, isn't it unreasonable to talk about increasing complexity, when everything was made.far simpler than what it would otherwise be?
Static site? Yes. Basic CRUD? Not sure about cost efficiency, but otherwise that’s right in the APIGateway+Lambda wheelhouse.
Static sites do not involve any form of tasks being executed: you just serve dumb files and let everything happen client side. I fail to see how a usecases that involves zero computational needs, and arguably zero servers, is used as an example in a discussion about how to implement services.
It's not really a red herring when Lambda mostly forces communication between different application layers to be over the network. You suddenly have to deal with rate limiting, retries, bulkheading, etc. in a lot more places than you would even with even a SOA.
> you actually have to write unit tests
I'm not really sure unit tests are a differing factor in testing functions as a service and other software. The real issue with testing is that it's very difficult if not impossible to do more than unit testing locally. You're often using docker containers or mock services to try and simulate what is going on in production which is insufficient. Things like CloudFormation can't be done locally (and this can be where a lot of gross complexity lies), so ultimately you have to set up an identical-to-prod testing environment and do the bulk of your testing there (which you probably should have anyway, but that's besides the point). Things like localstack HAVING to exist but not being supplied by AWS is a symptom of something gone wrong.
> Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.
Why is this obviously no? It isn't to me. You likely need a lot less these type of things if you're running a monolith.
> In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt.
It seems to me you're ignoring all the extra stuff you HAVE to have to deploy, secure, and maintain serverless. There is a lot more than the code for the function themselves.
Well now you're deep into Terraform or CloudFormation, setting up API Gateway, building an authorizer, coming up with an API Schema, wiring up everything with IAM, setting up Route 53 and Certificate Manager, building CloudWatch Dashboards, etc etc. I hope you got all the permissions right because there are now 25 different places someone can maliciously gain access to your AWS account. And in order to change anything you should be prepared to update your IaaC schema once every couple months or it will all stop working.
Back in days past you'd rent a hosted server or shell account (It was hideous, but bear with me), write 15 lines of PHP, connect to the included database, and you're done. It wasn't "web scale" but the time to reach a MVP was probably 1/100th that of the current era of gross complexity. And any one who built software in that time period had a very clear runway to bring it in house and run on bare-metal. No such thing exists in today's era of subscription-model infrastructure.
Terraform is a magical amount of arcane.
No, not really. You only use any of that if that's what you want to do. You can use a EC2 instance and do everything yourself, if that's your thing.
I mean, let's go through your list. CloudFormation is just infrastructure as code. It helps treat your deployments as cattle, but you aren't forced to use it, right? And you only need an API gateway if you want to put together an API using Amazon's proprietary stuff. Nothing forces you to do that, right? And you only need Route53 if you feel the need to manage a DNS and do stuff as DNS-based routing. And dashboards is just to show metrics in a way that fits your need.
From these services, which ones are a must-have? None.
> Back in days past you'd rent a hosted server (...)
Your "back in the day" is "right now" if you want to. You can fire up an EC2 instance, open a port, and done. Why are you leaving this out, and instead opt to misrepresent extras as required?
Hell, AWS also offers lightsail, and you can use S3 to run static sites directly. Both are far simpler than managing your own VM or baremetal service. Why are you leaving that out?
And why do you leave out the fact that with API Gateway and lambdas you can setup a fully versioned API with multiple endpoints in a few minutes? You don't even need to deploy anything as you can write the controllers for each route directly in the dashboard. If reaching an MVP instantly is what you really want, it's weird you leave that out.
> No such thing exists in today's era of subscription-model infrastructure.
I regret to tell you that you don't know what you're talking about. Even in AWS, where they try to upsell you everything, you can easily ramp up a VM, open a port, and do everything you need by ssh-ing into a linux box. And AWS is not alone on it. You can do the same on pretty much any cloud vendor. Say Hetzner: you can rent a baremetal box for around 30€ a month and leave it at that. If you prefer VMs you can get one for 3€ month. What's stopping you?
The same thing happened on frontend: all the juniors bootcampers' only experience of doing frontend development is running create-react-app and hacking away at React. Before that it was angular.
Everybody would frown at deploying a pure frontend js app or a jQuery app, in the same way most people would frown at deploying a PHP script on Apache.
Despite this, a huge portion of the internet runs on PHP and jQuery (including shiny new things, like levels.io's million making 1-2 men saas).
I feel as though there are similar amounts of friction across AWS offerings that make them unpleasant to work with in general, too. I'm numb to it now, but imagine my surprise when I actually went to use it for the first time after being hyped up about AWS.
For these reasons, and because you pay a significant premium for network traffic on AWS, I never use AWS for personal projects and I'm very happy with that choice.
It was surprisingly hard - the local development of lambdas is still very raw, documentation is scarce, and there are various small issues appearing here and there. I should probably write a blogpost about how to setup decent developer environment for AWS CDK and Lambdas, because there's not much on the Internet about it.
I set up the whole AWS infrastructure via AWS CDK. I have one TypeScript file, that creates a stack with lambdas, dynamodb tables, API gateways, S3 buckets, wires up the secrets manager, etc - all of that in 2 environments - dev and prod.
AWS CLI also can generate a Cloudformation YAML file from the CDK file (via `cdk synth`), which could be fed into SAM. So, I generate a `template.yaml` Cloudformation file this way, then run SAM like `sam local start-api`, and it runs exactly the same lambda locally, as in AWS, using that Cloudformation YAML file. SAM also supports live reload, so if you change any source files, it will automatically get those changes.
So, okay-ish developer experience for lambdas is possible. There're caveats though:
* I couldn't figure out how to use "nameless" Dynamodb tables (i.e. when CDK assigns the name to them automatically), because if I omit a table name in CDK template, then the local lambda and lambda in AWS assume different names for some reason.
* Locally, the binary outputs don't work. It ignores `isBase64Encoded` by some reason, and lambda just returns Base64 output, instead of binary.
* And the main problem - local lambdas are SLOW. It seems like it restarts some container or something under the hood on each API call, which adds 2-3 seconds to each API call. So, the calls that should be like 30ms, are actually 2 seconds now. This is super frustrating.
But they're too limiting. Cloudflare Worker should be a pure JS script (that looks like a service worker script in a browser). You cannot use any Node libraries. In my app I need to generate images on the fly, so I couldn't figure out how to do that in a Cloudflare worker, so I ended up creating lambdas for those, and proxying the calls from Cloudflare worker to those lambdas.
KV (their key-value store) is way too simple. No secondary indexes, no sorting, no querying (you can only sorta query list of keys, but then you have to fetch a record by each key individually). My app hit the limits of KV very quickly.
Also, there's no monitoring or any good way to do backups for KV. So, it's almost unusable in a real-world production system, except as maybe a cache layer.
Monitoring was a big pain for Cloudflare Workers. Like, even accessing production logs is weird - you have to "tail" them from your CLI. Even Cloudwatch is 100x better here.
Development environment is okay, I guess, but the local server crashes constantly. Also, it kinda forces you to use Webpack and bundle the sources. It also doesn't have anything like layers in AWS lambdas, and they have a limit of 1MB for a bundle, which I almost hit (I had like a 900kb JS file), so that part got me worried too.
Basically all of that made me worry I will hit a wall soon with Cloudflare Workers, so I moved to AWS. It's not piece of cake too, but at least I get a way to run proper Node scripts, a decent document database, continuous backups and pretty good monitoring out of the box.
Which seems to fail for access-protected sites since `tail` uses the same domain as the real endpoint. Also no error reporting - the exception may be in the logs, but it's missing the stack trace unless you log it explicitly.
The monitoring side of workers is extremely poor currently.
Also unit testing workers is hard to get right - there are no good mock libraries for request and kv which behave exactly like the production versions, so you end up with surprises after deployment.
- local dev server (“wrangler dev”) crashes often
- local dev server executes code always on Cloudflare -> seeing changes takes a few seconds, you need an Internet connection
- forces Webpack on you
- managing KV store is difficult (How to back up? How to update schema of millions of entries?)
- KV store is slow (400 ms in Europe) unless keys are being hit every couple of seconds to keep the edge cache warm
Currently I’m trying an alternative – Deno Deploy – and love it so far. Local dev server executes your code locally, quick compile times, native TypeScript support due to Deno runtime.
https://deno.com/deploy
Each use case is so unique, that it is likely difficult for them to account for every single situation where Lambdas are being used. Instead they opt for a generalized and non-specific documentation site, which results in docs I find practically useless.
I frequently find solutions to problems for AWS on someone's obscure blog, who after many days of suffering stumbled onto the answer, rather than finding anything of use directly from those who wrote the software.
Does a site exist where programmers can submit useful findings? Similar to stackoverflow, but rather than asking for questions, you just submit answers. I would think being able to search across such a crowdsourced utility would save programmers a ton of time.
SO kind of lets you do this. You can write a question and immediately answer it yourself - there's an option in the question UI for it.
localstack
mintel/pytest-localstack
charlieparkes/boto3-fixtures
> Anyone have any suggestions or share my frustrations?
I shared my experience and asked them a question. I at least contributed to the discussion.
Your comment basically adds nothing to the discussion and is essentially cyber bullying.
[1]: https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm... [2]: https://github.com/aws/aws-lambda-runtime-interface-emulator
I didn't join the dots up for this use case though, thanks!
https://serverless-stack.com/
1. If you are building APIs and using Lambda functions as targets from an API Gateway API, look into libraries like serverless-wsgi (Python) or wai-handler-hal (Haskell) that translate between API Gateway request/response payloads and some kind of ecosystem-native representation. Then as long as you're writing code where all state gets persisted outside of the request/response cycle, you can develop locally as if you were writing for a more normal deploy environment.
2. Look into the lambda runtime interface emulator ( https://github.com/aws/aws-lambda-runtime-interface-emulator... ). This lets you send invoke requests to a fake listener and locally test the lambda more easily. While the emulator is provided in the AWS container base images, you don't need to run it inside a container if you're deploying with zip files. (AWS-provided container images automatically enable the emulator if not running in a lambda runtime environment, and using docker for port remapping, which is nice but not at all required.)
3. Get really good at capturing all requests to external services, and mocking them out for local testing. Whether this is done with free monads, effect systems, or by routing everything through gateway classes will depend on your language and library choices.
It would be quite a powerful workflow for running integration tests. I shall do some research!