HN Explorer

Serverless dev practices questions

51 points by rsh210m ↗ HN
Hi guys, So after spending some time exploring serverless practices (Lambda in particular), a few questions popped into my mind which i’d love to get some more experienced views. The development guidelines seem to be quite straightforward for some simple stuff to do and the given sample code demonstrate it nicely. Yet - what happens when the overall application isn’t as simple and requires more than 15 lines of code? Building the design and logic of the app shouldn’t be that hard, yet making sure that it actually works is what worries me more… What are the given practices to test, debug, troubleshoot and iterate with the code i write in Lambda? The best i came up with is traces and log collection (most common), wrappers for the code in Lambda to collect more metrics and SAM local which should allow you to run stuff locally (but then, how to simulate the real context and the flow of the app)... are these really enough to test code in Lambda? Are there any other options around? Are there any other outstanding issues in developing for for Lambda?

29 comments

[ 3.1 ms ] story [ 74.5 ms ] thread
Jdam
For lambda, I highly recommend integrating AWS X-Ray.
mseebach
AWS Lambda is a very small interface. I'm only getting started, but I'm planning to keep all my application code in a separate project with no Lambda dependencies, test that as I would any application, and then have a thin wrapper that just wraps each endpoint in the Lambda interface.

This would also maintain as loose as possible coupling to the Lambda ecosystem, making porting easy -- I'm in Java-land, so that probably means Dropwizard. I'm not there yet, but I'm planning to maintain a parallel Dropwizard wrapper, to make local dev easy.

parzivalm
Have you thought about the added execution time from JVM startup? A project I had dropped in my lap awhile ago was figuring out a way to speed up a java lambda function. Basically we decided the only way was to try and make the lambda container always hot. A cloudwatch event would trigger the lambda function every 5 minutes to make sure it was always hot.
mseebach
For now, my project isn't very sensitive to taking a couple-of-seconds warm-up hit. But I'm aware of the issue, and it seems that keeping your classpath lean works wonders. I'll be investigating further, but my tentative plan is to section the endpoints by their dependency-size. The majority of the enduser-latency-sensitive endpoints will need little more than a JDBC connector.
fiokoden
I have a very large application implemented as one single lambda python function.

Works great.

nvivo
Please define "very large". When I think large app, I think at least a few hundred endpoints split across tens of thousands of files. How can you have a very large app with a single endpoint? Could you elaborate?
shawabawa3
A single lambda function doesn't mean a single endpoint.

The lambda function gets the request, looks at the url and routes based on that

fiokoden
Ok we have very different definitions of very large. I mean s complete full featured SAAS workflow application.

I don't actually use the AWS HTTP API gateway. I control the front end entirely so my UI just talks direct to lambda. It's very simple - instead of futzing around implementing REST API mapping layers I just connect all my front end functions to a single lambda function and specify in the params the name of the python function I want to run. Super easy.

Even if you wanted to use API gateway you could still map all the end points to one single application and dispatch based on the route/URL.

I can make a solid recommendation to anyone use lambda .... seriously consider using Cognito for your user management.

There's no reason why you couldn't have tens of thousands of functions in the same function, in fact probably a good idea.

mseebach
> seriously consider using Cognito for your user management.

Can you please expand on that recommendation? I tried and found it profoundly opaque, and eventually gave up in favour of auth0. I specifically wanted to use the hosted login/sign-up forms.

fiokoden
Yeah it was fiddly to understand and set up but it all worked fine in the end after persisting with it.
danesparza
I found this tutorial had some very good information on incorporating Cognito into an app: https://aws.amazon.com/blogs/aws/build-your-first-serverless...

It lays out the architecture fairly well and how Cognito ties into it. Still very fiddly, and doesn't give any indication on how to use their hosted forms (which are a bit of a mystery to me as well).

Since it's still on point with the OP's question, I'll ask: What has your experience been with auth0? It's another service I have considered.

mseebach
I really liked it. Easy to follow instructions and an active community where all questions I could come up with were already answered. Took me an afternoon after getting almost nowhere with Cognito in three days.

The thing I spend the most time on was "user_metadata" and "app_metadata", two JSON blobs on each user, RW and RO respectively. In order to read those at all from your application, you have to define a custom "rule" (arbitrary Javascript that wraps responses). The reason for this is something about standards and name-spacing, but I couldn't really follow the argument, and it's seems like this is something that very confusing to a lot of users.

danpalmer
I work on a large (I think) Django application, ~1000 endpoints, ~150k sloc. I've been thinking of how something of that size could work in a serverless context, and I really struggle to imagine it working well.

Between the startup time (importing that much Python takes a few seconds, fine for a long running server, not good per-request), and all the dev-ops/infrastructure automation to get it all working correctly in dev/prod, I just think it would be more effort than it's worth, compared to running in "normally" on a cluster of VMs or dedicated machines.

JackC
This is true of any project using Zappa, right? https://github.com/Miserlou/Zappa

The python project defines a wsgi application (which is a python function meeting the wsgi spec), and Zappa runs the function on Lambda.

I wonder how large Zappa sites have gotten.

nevon
I would recommend splitting out the lambda-entrypoints from the rest of your app, so that you can test it separately.

Personally, I've found it tough to test full integrations where I'm depending on cron jobs, work queues and notifications. I felt like I was writing an awful lot of code just to be able to fake all these integration points, which meant that in reality there was a lot of behavior that wasn't really being tested.

kostarelo
We use AWS Lambda exclusively on the product I am currently working on (3 peoples team).

- We use the Serverless framework

- Monorepo with multiple (micro) services

- Each service exposes Lambda for particular functions (e.g. Articles service exposes GetArticle, CreateArticle, etc..)

- No business logic is being shared between services, only libraries

- Three environments, Dev, Stage, Production (and every developer can also deploy its own)

- Tests are invoking the Lambda handlers directly and are mocking responses from other Lambda calls. We use Localstack to stub AWS services, such as S3 and DynamoDB.

Still, a lot to learn of course, but so far everything is working great for us.

davemontorio
What do you use to store articles? DynamoDB? If so, are you ok with querying and searching data? What about security? How do you implement it?
kostarelo
DynamoDB for storing only. After an article is stored in DynamoDB and stream will index it into an ES Service. Then we are quering and searching only ES. DynamoDB can't handle searching very good.

Security wise, everything is private except one Lambda that exposes a GraphQL server and is being forwarded to the public net through API Gateway. We will implement subnets and security groups very soon.

davemontorio
With ES do you mean ElasticSearch from AWS? This is interesting, can you tell me something more please?
agentgt
> Are there any other options around? Are there any other outstanding issues in developing for for Lambda?

I'm not sure how AWS Lambda handles the charging of network latency (or failure) case of downstream external service dependencies but that is one of our biggest problems. ie pull/scrape architecture. The push portion (ie events) of our architecture is an obvious fit but that is already taken care of quite nicely.

We interact with hundreds of web services (REST APIS, SOAP, etc) that are actually external from us.

Its unclear if or how Amazon charges you for a slow external service (I would imagine its particularly bad if the code is blocking IO) or if that is even possible. I do know spawning a whole bunch of servers to connect to a single REST API endpoint is damn slow.

I have feeling Lambda for our case is probably not a good one? Obviously for data processing I can see it but that is not a challenge we have.

In an ideal world our downstream (or I guess it could be called upstream...) dependencies would contact us but... that sadly isn't the case particularly for adhoc stuff.

tmclaugh
This stuff is hard and we're all still figuring it out. Just recently we realized we built a monolith because we weren't paying enough attention to whether all the features should be a part of a single system. We're going to revisit that.

Don't look at your function as an app, it's an function and the best functions do a single thing. The definition of a single thing depends on the scale of view you're taking. But that may help you to realize your functions shouldn't be very complex and possibly small enough to test more easily.

However, that results in increased system complexity as you start adding SNS topics, SQS queues, DynamoDB tables, and all sorts of event triggers. Something I've adopted is using draw.io (others use Lucid Chart) to draw out the system logic. Maybe this helps?

https://www.draw.io/ https://www.lucidchart.com/

Maybe this makes the confusion worse?

davemontorio
I recommend to use the Serverless framework https://serverless.com/. It's pretty straight forward and very useful to configure imho. In this way the entire project (configuration + code) can be versioned (svn, git...) easily.
hkchad
We are using microrepos, we implement all logic in libraries then import those libs into another project that wraps it w/ the lambda logic. The lambda logic tends to be small and all the 'work' is done in libraries. Thought here being if we need to migrate to another serverless solution (or server solution) it's quick/easy to wrap the tested libraries in whatever the next logic should be.

Then we use Apigee a127 Node project to invoke the lambda's. This is where authentication is done (we use Auth0 JWT/Bearer tokens). The Apigee a127 project invokes the lambda and handles routing. We looked at serverless but didn't want to use API Gateway since we already had apigee. We plan to look at SAM next but so far haven't need it. We have a few grunt scripts that we can use for running the lambda locally so far not a big deal.

For logging we use an AWS ElasticSearch cluster and Kibana.

So far we haven't run into any big issues with lambda, we see better response times when there is a constant load on the system, I guess that helps w/ keeping the lambda's warm. Even cold start isn't a big issue b/c the $$$ saved is well worth it.

Automate EVERYTHING. You will quickly get a bunch of lambda's and you need a way to keep them all updated, manual SUCKS.