Or stop chasing the latest fad(s) and build a regular old boring service. The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Use boring tech, solve business problems and create value without adding needless complexity because devs want to play with the latest toys.
100% agree, however the difference between one EC2 instance that was launched manually vs. all of the IAM role mess and configuration needed for a lambda... well let's just say I'd rather have an undocumented EC2 instance.
I want a language/runtime that abstracts this away from me. I'll implement and debug functions the regular way, with the runtime in charge of deploying them whichever way fits. Bonus points for deducing infrastructure from statically analyzing code dependency paths.
I believe there is very early stage work [0] in this direction, but obviously nothing going mainstream yet. Would love to find more examples in the wild if anyone knows any!
I love how you responded to a post advocating for simpler architectures by linking to an automagic abstraction over microservices, because microservices weren't difficult enough already.
The example also mentions Stripe payments, but nowhere does it mention transactions or error recovery because they're stupidly difficult in microservices designs, so we'll just "skip right over that" and go straight to losing money any time there's a hiccup.
"Sure it's incomplete and outright dangerous to our bottom line, but I can't wait to tinker with it!" is precisely the problem with the industry.
I mean, I do agree with the parent, insofar as I don't want to deal with the mess of microservices/functions firsthand.
Now if a runtime could reap the benefits of this architecture automatically, sort of like a compiler would optimize code for a particular cpu, then that would be a best-of-both-worlds situation.
The problem with your argument is that microservices and lambdas aren’t new.
The industry has had plenty of opportunities to ditch them and go back to monolith services and yet it hasn’t happened. Is it because everyone is wrong except for you ? Or is it because they are better for certain types of use cases.
Erlang is awesome! I never had a chance to use it in anger but I've been keeping a close eye on Erlang/Elixir for years. In fact we almost considered moving to it at my project for the sake of stability, but went with adopting some of the principles in our codebase instead (embrace failure basically, expect stuff to crash and recover properly).
The language is built with this idea from the ground up, so it can move functions around between machines robustly and without running into problems with versioning. It's the only model I've seen for distributed computing that really makes it a first-class citizen of the language.
It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because you are ignorant. You have no understanding of the business requirements, NFRs etc that have led to the decision. So I prefer to give developers the benefit of the doubt.
Lambda architectures have trade-offs like every architecture and they are useful in certain situations. Building everything in one service and scaling it when the components have different scaling behaviours usually isn’t ideal.
Lambdas trade developer experience for operational simplicity.
And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding.
In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
And if it can’t scale and run reliably in production then it won’t work at all.
Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And
Lambda architectures help with this.
This whole conversation isn’t about using Lambda. It’s about using an exorbitant amount of lambda. Which means you’re using it to the exclusion of other things. Once a project is segregated to this degree, invariably discovery has gone out the window. You have a couple of people who really understood the system and little hope of replacing or duplicating them. It’s just another form of job security. Ignore Kernighan and make a system even you barely understand.
Hacker news is like that, the person you reply to could be between the person that invented and built the tech discussed or a kid replying quickly before going to do their chores.
Always fun to see either way
> If it doesn’t work for the developers, then eventually it won’t work for anybody.
Don’t conflate technical challenge with logistical challenge. Picking between two algorithms for condensing data is technically challenging. Picking between a solution that is transparent and one that is opaque are not the same kind of challenging. As an old mentor used to say, “I hate the term ‘tech debt’. This [situation] isn’t tech debt, it’s wear and tear.”
Opaque is magic. Hiring people who understand it is expensive. Building people who understand it is part luck and part patience. And the opportunity costs, which we almost never account for, let alone properly, are one of the biggest consequences. What else could you do with that time and, more importantly, energy?
Logging/Observability, and distribution of responsibility among teams. The question is then, "did my component behave correctly for the input parameters." Observability and logging gives you the parameters and the behavior, the complaint gives you the output.
I dropped a jaw at the $1,500 per month for the cluster. My mind says 15,000 Lambdas costs basically zero dollars… until they start getting run/used. I think the key thing was the request rate, or maybe that combined with the raw number of functions? I can’t imagine using something that was so expensive just to house functions rather than run them at high frequency.
As far as I know, Google does not actually use k8s for their public facing services (or anything at all?). I would love to be corrected on this.
Google was able to scale quickly back in the day by keeping things as simple as possible. "1 hard drive per server" level simple. Center everything around one massive job scheduler (Borg).
Of course a lot has changed since that time, but I can't see Google upending a established infrastructure by making k8s plugins for their network requirements, tying k8s into Borg, etc. It would give less visibility and debugability for little gain.
(But again I have zero inside knowledge here)
At scale there is no one size fits all. Kubernetes can fit a lot, but will always fall short of a tailor made solution.
For more unsolicited advice from an armchair infrastructure hyperscale expert, please subscribe to my newsletter.
> The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious.
From my own personal experience, Google Cloud Functions, and everything around it, is fantastic. Had 25k "worker" computers hitting GCF 24/7 (~50 rps) reporting their status. New status message went onto Cloud Tasks, which just hits GCF again, and stored into a Cloud SQL Postgres. Cost was about $200/mo. It ran perfectly with zero downtime for over a year.
Super easy to develop, test, and deployments were done with github actions. Very little actual code is dependent on anything GC, so vendor lockin was minimal.
Your relentless customer service is commendable. Good work. Serverless works well in many scenarios and being able to run it on any k8s cluster without proprietary platform hooks is a good thing for all of us to have an option for.
It is hard for me to understand why anyone would build so many individual functions. Just build one function, put a router in it and then call out to as many individual handlers as you want. Then you're just deploying a single function to the PaaS and everything is simple.
In that case you are deploying a whole monolithic app to the Faas platform. This needs to be initialized on every cold start which takes longer the larger your code is. The main selling point (that I have experienced) in Faas is that you can have small, individually scalable functions that are quick to initialize and can be torn down after executing. If you have the whole app under one handler you would be much better off with a different architecture where the app is persistent and always running.
If you manage to lazy load your routes it may help with cold starts but I think you effectively lose the benefit of warm handlers since every request could use new code that needs to be loaded fresh.
I've never attempted lazy loading inside a Faas platform but I think generally you would be better off creating small deployable chunks and letting the platform handle scaling them up / reusing as necessary. Less fighting against the platform and if you have to maintain strict boundaries to enable lazy loading only the needed code for each route it's a short step to deploying those as their own functions anyway.
The article is about 15k functions. You know what is a lot worse than lazy loading? Getting 15k functions to deploy accurately. I've seem functions fail to deploy, are you going to fail the whole build if one of the 15k fails? What if functions depend on other functions? How about CI/CD where you're updating multiple times a day? How about when you update a single dependency across those 15k and you need to deploy again?
I don't know what PaaS you've used, but at least with Google Cloud Functions, you can do a minimum number of instances. Set it to 1 and you never have cold start issues.
I agree 15000 functions is overkill and I would never recommend something like that. I'm trying to explain why combining all 15k functions into a single Faas deployment is not going to be the best option either.
There is a middle ground that will be much faster and cheaper all around.
To me this is just the latest iteration of things we've tried before. It's another event driven system (I think we called them Enterprise Service Buses and they only lasted a couple of years) and distributed FastCGI. We will repeat all of the same 'mistakes' (aka, consequences) of those systems and the documents produced during the Trough of Disillusionment will read like a madlib of existing documents.
It all kind of works while the fanout is low, but these systems have such long feedback loops that the team doesn't really learn how to avoid the degenerate cases and the whole thing limps along. The system isn't just allowing emergent behavior, it depends on it. Which is a special kind of insanity I've never been comfortable around.
Really, just write some code that does exactly what it says it does. It seems remedial at first but once you get going you start finding ways to add features that were difficult before and it all comes out in the wash.
Can I just say I was shocked to see a "Pricing" page on something called "openX." Then, on the pricing page, there was no price... so, now I'm just confused. I'm also trying to understand what this 15k functions post is all about, since you'd need an enterprise license to use that many. Whatever that means.
Open core and support packages are very established business models.
The OP has also hacked away on OpenFaas in the open for years, giving us a portable alternative to lambda. That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale.
Though I don’t know their motivation for hiding pricing, I suspect every enterprise level deployment (for instance into an ISP) is pretty bespoke.
> Open core and support packages are very established business models.
and “bespoke” pricing for such models is also pretty common. doesn’t mean we have to like it.
> That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale.
Portable is valuable, yes, but at what price ? Does anyone know what the price is ? If it’s customer-specific it’s impossible to say that this proposition is generally competitive with lambda.
Open core / commercially support products need to have published pricing for at least a standardised offering IMHO. If I need to get on a call with a salesperson to find that out (so I can do the back-of-the-envelope calculation that says whether or not this thing will cost in) it isn’t happening…
> The OP has also hacked away on OpenFaas in the open for years, giving us a portable alternative to lambda. That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale.
A free, MIT license is limited to 30 functions. That's enough to basically play with it, but you probably can't do anything serious. That leaves purchasing a license mandatory, but if you're pre-revenue, boot-strapping a business, that is impractical, if not impossible. Thus you pretty much have to go the vendor-locked-in lambda way; even if you already have the hardware (and expertise) to run this thing.
It is worth noting that azure functions can be run self-hosted, entirely for free.
I'm not sure if you are serious of joking but the terms of MIT license are very clear. So unless they add a clause to it that the modification of this limit is forbidden (which would automatically make it a OSI-non compatible license), anybody can do what they like with it and redistribute the result as long as you abide by the terms of the license which are actually very simple.
Personally I've never had a use case for OpenFaaS but if I did, it would probably be one of the first things after installing.
You can dual license software. And you can even revoke a license per customer (otherwise SaaS wouldn’t work). They can say “you only get this license IF X” and if you don’t have X, you get that license. So, if you can somehow qualify for the MIT license and relicense it, you can then redistribute it to someone who does not qualify for the original license. However, if that entity is you or a company you work for, they could argue that you never should have gotten the MIT license in the first place and take you to court over it. Now, not only are you probably on the hook for fraud, but also theft of copyrighted material with an “intent to distribute.”
So, yeah. It’d be an interesting case, but hopefully I can just eat popcorn on the sidelines. I don’t want to be the one fighting that battle.
59 comments
[ 3.5 ms ] story [ 105 ms ] threadAnd even then you better have the configuration defined in code because otherwise you'll spend an entire afternoon reverse engineering how it works.
I believe there is very early stage work [0] in this direction, but obviously nothing going mainstream yet. Would love to find more examples in the wild if anyone knows any!
[0]: https://egghead.io/blog/how-to-quickly-build-a-backend-with-...
The example also mentions Stripe payments, but nowhere does it mention transactions or error recovery because they're stupidly difficult in microservices designs, so we'll just "skip right over that" and go straight to losing money any time there's a hiccup.
"Sure it's incomplete and outright dangerous to our bottom line, but I can't wait to tinker with it!" is precisely the problem with the industry.
Now if a runtime could reap the benefits of this architecture automatically, sort of like a compiler would optimize code for a particular cpu, then that would be a best-of-both-worlds situation.
The industry has had plenty of opportunities to ditch them and go back to monolith services and yet it hasn’t happened. Is it because everyone is wrong except for you ? Or is it because they are better for certain types of use cases.
https://www.erlang.org/doc/design_principles/des_princ
WhatsApp famously scaled a lot with relatively few devs:
> WhatsApp scaled to 1B users with only 50 engineers https://news.ycombinator.com/item?id=28985169
And yeah, Elixir is great!
The language is built with this idea from the ground up, so it can move functions around between machines robustly and without running into problems with versioning. It's the only model I've seen for distributed computing that really makes it a first-class citizen of the language.
It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because you are ignorant. You have no understanding of the business requirements, NFRs etc that have led to the decision. So I prefer to give developers the benefit of the doubt.
Lambda architectures have trade-offs like every architecture and they are useful in certain situations. Building everything in one service and scaling it when the components have different scaling behaviours usually isn’t ideal.
Address this part, or you’re just deflecting. Comments? Suggestions?
And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding.
In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
Trading developer experience for just about anything does not scale - up or down.
It’s a false economy.
Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And Lambda architectures help with this.
Statements made confidently while also being totally untrue.
Hacker news is like that, the person you reply to could be between the person that invented and built the tech discussed or a kid replying quickly before going to do their chores. Always fun to see either way
> If it doesn’t work for the developers, then eventually it won’t work for anybody.
Don’t conflate technical challenge with logistical challenge. Picking between two algorithms for condensing data is technically challenging. Picking between a solution that is transparent and one that is opaque are not the same kind of challenging. As an old mentor used to say, “I hate the term ‘tech debt’. This [situation] isn’t tech debt, it’s wear and tear.”
Opaque is magic. Hiring people who understand it is expensive. Building people who understand it is part luck and part patience. And the opportunity costs, which we almost never account for, let alone properly, are one of the biggest consequences. What else could you do with that time and, more importantly, energy?
As far as I know, Google does not actually use k8s for their public facing services (or anything at all?). I would love to be corrected on this.
Google was able to scale quickly back in the day by keeping things as simple as possible. "1 hard drive per server" level simple. Center everything around one massive job scheduler (Borg).
Of course a lot has changed since that time, but I can't see Google upending a established infrastructure by making k8s plugins for their network requirements, tying k8s into Borg, etc. It would give less visibility and debugability for little gain.
(But again I have zero inside knowledge here)
At scale there is no one size fits all. Kubernetes can fit a lot, but will always fall short of a tailor made solution.
For more unsolicited advice from an armchair infrastructure hyperscale expert, please subscribe to my newsletter.
* 0..N Scaling
* Request/Invocation is unit of compute
* Deploy anywhere
* Don't need to bring a web server
* Instant Observability (distributed tracing, metrics, logs, alerts, & dashboards)
* Instant Security (AuthN/AuthZ/mTLS)
* Instant Cost Monitoring
How good "FaaS" is will be highly dependent on how good your ops team is.
From my own personal experience, Google Cloud Functions, and everything around it, is fantastic. Had 25k "worker" computers hitting GCF 24/7 (~50 rps) reporting their status. New status message went onto Cloud Tasks, which just hits GCF again, and stored into a Cloud SQL Postgres. Cost was about $200/mo. It ran perfectly with zero downtime for over a year.
Super easy to develop, test, and deployments were done with github actions. Very little actual code is dependent on anything GC, so vendor lockin was minimal.
You can lazy load your route implementations
I've never attempted lazy loading inside a Faas platform but I think generally you would be better off creating small deployable chunks and letting the platform handle scaling them up / reusing as necessary. Less fighting against the platform and if you have to maintain strict boundaries to enable lazy loading only the needed code for each route it's a short step to deploying those as their own functions anyway.
I don't know what PaaS you've used, but at least with Google Cloud Functions, you can do a minimum number of instances. Set it to 1 and you never have cold start issues.
There is a middle ground that will be much faster and cheaper all around.
It all kind of works while the fanout is low, but these systems have such long feedback loops that the team doesn't really learn how to avoid the degenerate cases and the whole thing limps along. The system isn't just allowing emergent behavior, it depends on it. Which is a special kind of insanity I've never been comfortable around.
Really, just write some code that does exactly what it says it does. It seems remedial at first but once you get going you start finding ways to add features that were difficult before and it all comes out in the wash.
The OP has also hacked away on OpenFaas in the open for years, giving us a portable alternative to lambda. That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale.
Though I don’t know their motivation for hiding pricing, I suspect every enterprise level deployment (for instance into an ISP) is pretty bespoke.
and “bespoke” pricing for such models is also pretty common. doesn’t mean we have to like it.
> That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale.
Portable is valuable, yes, but at what price ? Does anyone know what the price is ? If it’s customer-specific it’s impossible to say that this proposition is generally competitive with lambda.
Open core / commercially support products need to have published pricing for at least a standardised offering IMHO. If I need to get on a call with a salesperson to find that out (so I can do the back-of-the-envelope calculation that says whether or not this thing will cost in) it isn’t happening…
A free, MIT license is limited to 30 functions. That's enough to basically play with it, but you probably can't do anything serious. That leaves purchasing a license mandatory, but if you're pre-revenue, boot-strapping a business, that is impractical, if not impossible. Thus you pretty much have to go the vendor-locked-in lambda way; even if you already have the hardware (and expertise) to run this thing.
It is worth noting that azure functions can be run self-hosted, entirely for free.
I don't understand: if they used MIT license, what's to stop me from increasing this purely arbitrary limit?
Personally I've never had a use case for OpenFaaS but if I did, it would probably be one of the first things after installing.
So, yeah. It’d be an interesting case, but hopefully I can just eat popcorn on the sidelines. I don’t want to be the one fighting that battle.