Ask HN: Have you shipped anything serious with a “serverless” architecture?

It has been pretty much set-and-forget. Last anyone had to even look at it was almost 3 years ago, and afaik it's still working (our ad sales team would be complaining loudly if it weren't).

For something peripheral like that, it's nice not to have to run servers for it or devote any energy to keeping it running.

In terms of both server costs and upkeep costs, the economics have been highly favorable.

I'm not sure I'd use it yet for something mission-critical or that shipped changes frequently. My recollection is that when we did have to adjust it, debugging was a bear. Tho tooling for that may have improved in the last 30 months.

I'll soon see if the back of the envelop calculations were correct. :)

I think serverless is the future... but not today.. in 5-10 years. That sounds like a long way off, but it's not.. it'll pass in no time. And maybe they'll improve it enough by then to make it viable.

I wouldn't build anything serious with it, unless you're ok rewriting it a few years from now.

All of https://ayvri.com is built using serverless.

(we'll be migrating from doarama to ayvri in the coming weeks)

For our processing we handled 200k uploads in one hour on ayvri when we where building scenes for the Wings for Life event (the World's largest organized run).

When just relying on serverless triggering an event from s3, the cost was high due to the volume of scaling, time in spinning up new services, etc. etc. We built a queuing system which manages load and then spins-up new instances based on the load in the queue. This resulted in a much faster response, and SIGNIFICANT reduction in cost.

For the ayvri website, some pages are slow due to the lambda's not being warm, and I'm surprised users haven't complained. The important stuff is kept warm, and we're working on scaling that out for more responsiveness across the site.

As far as visibility into the app, I'm not going to pretend this is a solved problem. At the moment, we have most of the visibility we need via cloudwatch, and we have built some of our own analytics.

We had one instance where there was an issue between db connectivity which we were not able to resolve. We have put it down to a short networking issue between services. It lasted for 5 minutes one Sunday morning and then went away. So we had enough visibility into the service not being available, but failed in deeper understanding of where the problem was.

If you have further questions I can help with, feel free to reach out.

I will say, that I bought into serverless and went whole hog. I probably don't recommend that. We jump through some hoops we probably wouldn't need to if we had run our website via an ec2 instance and cloud-formation managing the scaling.

However, we have a few of our services which can come under high load quickly, and we don't need to scale up the entire site to serve those, such as our track processing. We believe Serverless was the correct decision for those processes.

We listen to the cloud pubsub from a worker.

1) we don't manage it. We receive quite a lot of webhooks and it's nice to offload that 2) all of our webhooks are async. We just have 1 worker that handles it all, instead of provisioning a bunch of pods. 3) managing cloud functions is dope, since you can make it autodeploy from git.

10/10 would use again. Not sure about building a whole app around it tho

My biggest complaints are probably DynamoDB eventual consistency (unavoidable when using GSIs), occasional CloudFormation funkiness (though no urgent prod issues yet, thankfully), CodeDeploy CW alarm rollback jankiness (which doesn't tell you which alarm triggered a rollback!!), and lambda coldstarts. But none of these are too terribly concerning and I have faith they'll get incrementally better over time, hopefully.

The biggest cautionary tip I have is we run all our Lambdas with the max 3GB memory both for peace-of-mind and because the underlying EC2 instances have significantly faster CPU. We were seeing weird timeouts and latency initially with <1GB memory, so I'd be hesitant to run the service if the extra cost of using the biggest possible instances is a concern, which for us it is not.

Another cost concern I should also mention is that we mitigate cold starts by running multiple canaries using scheduled lambdas (in addition to the standard canary role of generating a baseline of metrics and immediately detecting/alarming on end-to-end issues). We are effectively maintaining a constant warm pool which, in theory anyway, greatly decreases the chances customer traffic will hit cold starts. I'm not intimately involved with the financial aspects but I suspect achieving the same effect with EC2 would be significantly cheaper, at least with respect to infrastructure costs. I would guess, though, that the developer time savings achieved by massively reduced ops burden and overall system simplicity are probably comparable to the increased infrastructure cost, and very possibly hugely outweighing it.

We are all in with AWS so the code build, cloudformation and code pipeline all work really well with lambdas.

I was super impressed with how they had set this all up and they were extremely well aware of all the limitations and do's and dont's of that particular cloud implementation.

Obviously there is the lock-in problem, if you ever decide to move you have a bit of work ahead, so build some abstraction layers in right from day 1 to avoid hitting all your code if that time should ever roll around.

And cultivate contacts with your cloud vendor.

So far, we love it. It handles roughly 20k images a day.

You are right that Cloudwatch logs are a hassle. So we pipe all of the log events into Scalyr (and log JSON objects, which Scalyr parses into searchable objects).

In terms of error handling, Lambda retries once on exception. So we raise exceptions in truly exceptional cases (e.g. - some weather in the cloud prevents a file from being downloaded or uploaded). We have Cloudwatch alerts that notify the team for every true exception. Happens less than once a day.

In pseudo-exceptional cases (e.g. a user emails an invalid image), we simply log to Scalyr with an attribute that identifies that the event was pseudo-exceptional, and then set up Scalyr alerts to email us if the volume of those events goes above x per hour.

tl;dr - Cloudwatch + Scalyr with good alerts and thoughtful separation of exceptions from pseudo-exceptions is my recommendation!

I have 14 or so regions so doing the same thing with EC2 would have considerable overhead, though I can still imagine many cases where Lambda would not be cost effective, but its integration with Kinesis is fantastic as well, stream processing almost cannot be easier, and while people say Kafka is more cost-effective, with a bit of batching you can get a long way with Kinesis as well.

The best part is no devops needed. We use Serverless Framework. The biggest downside are cold starts for frontend response time. But this hasn't been a terrible issue as of yet. We have considered moving these 20 API endpoints to a nodeJS server which will resolve the issue but didn't have the time to do it yet.

We'll never go back. Serverless is the future.