Different types of concerns. Software engineers are most often concerned with writing software, not managing security settings, configuring services, and updating things. If you could declaratively write a single file which configures your server for you and then just copy paste that server state to 10,000 other servers, I'd be happy (you can, in fact, do this!)
Because they are not concerned with those things, the lack of knowledge in those areas creeps in.
If you don’t understand how the OS works at a very fundamental level, you will write sub-optimal code. This is usually fine, because people are smart, compilers are good, and hardware is absurdly fast. However, the fact remains that it’s probably giving up some bits here and there.
More germane to my point, though, the amount of libraries and tooling and abstractions being spun up to avoid having to run a server seems like more effort than its worth. Serverless sometimes has cost merits, but as a counterpoint, you can get a 1 year RI micro EC2 for something like $3/month. If your app is struggling to run on that, either you have the scale to justify more cash outlay, or your app has performance issues – and now we’re full circle back to the previous paragraph.
The cost you are trying to save is not infrastructure, or compute time, it's labor. It's easier to find a junior javascript developer and then sandbox them inside docker or some serverless service than it is to find someone qualified to lock down a Linux box
- Flame still assumes you have a server somewhere- this is purely for elastic workloads.
- Dynamically scaling horizontally is not something "linux" can do for you: It's a legitimately hard problem because of all the usual distributed systems difficulties.
- Often, there are workloads that are only needed occasionally but also have very different resource needs than the rest of the system. Being forced to not only create a new version of your application, but additionally a bespoke server setup and dedicated VM/Container image for that case takes something that could be trivial in the Flame model, and makes it take hours even for someone experienced.
Basically, you really should've read the article first.
> Dynamically scaling horizontally is not something "linux" can do for you
It can, depending on your definition of horizontal, and whether you accept that tools like HAProxy fall under the realm of “Linux administration.”
systemd can spawn N services from a single definition file, which can then register with the load balancer. A small shell script could monitor those service cgroups for basic metrics like memory or CPU utilization, and then command systemd to scale up/down. This is horizontal from the app’s perspective.
I’m not suggesting the above is a great idea, and you’d probably be better off with container orchestration of some kind – but tbf, all that’s doing is encapsulating cgroups and monitoring consumption to command scaling.
While Nix is cool and I think the model of the future, it doesn't obviate the need for elastic workloads on demand without wanting to dedicate a ton of time.
WELL, considering the time delay in provisioning on Hetzner/OVH, maybe Equinix Metal would work better? But, if you're provisioning + maybe running some configuration, and speed is a concern, probably using Fly or Hetzner Cloud, etc. still makes sense.
Author here. I’m excited to get this out and happy to answer any questions. Hopefully I sufficiently nerd sniped some folks to implement the FLAME pattern in js , go, and other langs :)
This looks great. Hopefully Microsoft are paying attention because Azure Functions are way too complicated to secure and deploy, and have weird assumptions about what kind of code you want to run.
> weird assumptions about what kind of code you want to run
Those "weird assumptions" are what makes the experience wonderful for the happy path. If you use the C#/v4 model, I can't imagine you'd have a hard time. Azure even sets up the CI/CD for you automatically if your functions are hosted in Github.
If your functions need to talk to SQL, you should be using Managed Identity authentication between these resources. We don't have any shared secrets in our connection strings today. We use Microsoft Auth to authenticate access to our HttpTrigger functions. We take a dep on IClaimsPrincipal right in the request and everything we need to know about the user's claims is trivially available.
I have zero experience using Azure Functions outside of the walled garden. If you are trying to deploy python or rust to Az Functions, I can imagine things wouldn't be as smooth. Especially, as you get into things like tracing, Application Insights, etc.
I feel like you should only use Microsoft tech if you intend to drink a large amount of their koolaid. The moment you start using their tooling with non C#/.NET stacks, things go a bit sideways. You might be better off in a different cloud if you want to use their FaaS runners in a more "open" way. If you can figure out how to dose yourself appropriately with M$ tech, I'd argue the dev experience is unbeatable.
Much of the Microsoft hate looks to me like a stick-in-bike-wheels meme. You can't dunk on the experience until you've tried the one the chef actually intended. Dissecting your burger and only eating a bit of the lettuce is not a thorough review of the cuisine on offer.
> You can't dunk on the experience until you've tried the one the chef actually intended. Dissecting your burger and only eating a bit of the lettuce is not a thorough review of the cuisine on offer.
But Microsoft isn't selling burgers that people are taking a bit of lettuce from. They're selling lettuce, and if that lettuce sucks in any context that isn't the burger that they're also selling, then complaining about the quality of their lettuce is valid.
Have you used azure functions in an enterprise setting? Because it’s a terrible experience.
I think our “best” issue is how sometimes our functions won’t be capable of connecting to their container registry. For no apparent reason. We have an Entra ID group that has Arc Pull access and we add new function slot identities to it automatically, and for the most part it works fine, but then suddenly it won’t.
We currently has a pipeline slot, that is completely the same as the four other slots, which will only deploy a new function version if you deploy it twice through the Azure pipeline. We’ve had Microsoft look at it and their suggestion was to delete the pipeline and create it again.
And that’s just part of it. Then comes the VNETs, the subnets the private endpoints and how both subscriptions and resource groups make everything a tiny battle.
I don’t really mind that much, we’re abandoning it in favour of Azure Container Apps and a fully run Bicep + Dapr pipeline and we’re never looking back.
Though, to be fair to Microsoft, the way they designed Azure Functions makes the migration away from them really, really, easy. Which is frankly a brilliant design for the more managed side of “serverless” in my opinions. It’s just a shame that the managed part just doesn’t work very well. The functions themselves work fine, well maybe not if you don’t use .net in isolation, but I can’t speak about that as we weren’t going to trust Microsoft to update our dependencies (even if they are Microsoft SDKs).
azure functions don't fit common definition of serverless, I've had a few convos with them over several years.. but there is really a mismatch owing to the original origin at azure, and real lack of understanding of the space, ie origin is as built on top of web apps.. ie.. azure functions is built on a hack for to try and enter the marketing in the serverless space at its origins. how many websites do you need to run... ie you can't run more than 50 functions, or the 16 cell table on different runtime options (ie. provision servers for your server less)... consumption is better, but the origins in web apps means its just a different product.. hey every function has a url by default :shrug: azure needs a radical rethink of what serverless is, I haven't seen any evidence they got the memo. in aws, lambda originated out of s3, re bring compute to storage.
I feel like the most natural implementation would be something like vert.x for the jvm. They already have the mechanisms to handle async execution (via reactive extensions and futures and coroutines on top of the event bus) and serialization and distribution of data across a cluster. There are eventbus clients for many popular languages as well, so you'd be able to build your application in a mix of languages.
It’s easy to dismiss these types of articles as straight sales pitches when the problem is presented as “…a fate worse than death.” and the solution so easy and painless, you’d be a fool not to dismiss any other approach! Thats the MO of snake oil sellers.
Instead, I think a more objective comparison with much less editorializing would be less of a turn off for folks curious about a better approach to solve a problem that you did well in outlining at the beginning of the article.
Hopefully you take this as constructive feedback! I think the substance of the article is interesting, I was just turned off by the presentation.
Yes and no: Maybe the rest of your workloads don't require much CPU- you only need this kind of power for one or two workloads, and you don't want them getting crowded out by other work potentially.
Or they require a GPU.
Or your core service only needs 1-2 servers, but you need to scale up to dozens/hundreds/thousands on demand, for work that only happens maybe once a day.
Thanks! I try to address this thought in the opening. The issue with this approach is you are scaling at the wrong level of operation. You're scaling your entire app, ie webserver, in order to service specific hot operations. Instead what we want (and often reach for FaaS for) is granular elastic scale. The idea here is we can do this kind of granular scale for our existing app code rather that smashing the webserver/workers scale buttons and hoping for the best. Make sense?
If you autoscale based on CPU consumption, doesn’t the macro level scaling achieve the same thing? Is the worry scaling small scale services where marginal scaling is a higher multiple, e.g. waste from unused capacity?
It's a "lie" in that of course there's a server, what else would your code run on? But we "lie" all the time by omission or shorthand or euphemism. "Cloud" computing doesn't really occur in the stratosphere, and "serverless" still runs on servers in the "cloud", and by "server" I don't mean waitstaff ;-)
I agree that it's a dumb name that doesn't really say what it means and "ephemeral" might be better, but "serverless" is better than "clouds-you-rent-by-the-second".
Very interesting concept, however it's a bit soured by the fact that Container-based FaaS is never mentioned, and it removes a decent chunk of the negatives around FaaS. Yeah you still need to deal with the communication layer (probably with managed services such as SQS or Pub/Sub), but there's no proprietary runtime needed, no rewrites needed between local/remote runtime environments.
Probably others too, those are just the ones I know off the top of my head. I see very little reason to use traditional Function-based FaaS, which forces you into a special, locked-in framework, instead of using containers that work everywhere.
Bring-your-own-container is certainly better than proprietary js runtimes, but as you said it carries every other negative I talk about in the post. You get to run your language of choice, but you're still doing all the nonsense. And you need to reach for the mound of proprietary services to actually ship features. This doesn't move the needle for me, but I would be happy to have it if forced to use FaaS.
Looks like a great integrated take on carving out serverless work. Curious to see how it handles the server parts of serverless like environment variables, db connection counts, etc.
One potential gotcha I'm curious if there is a good story for is if it can guard against code that depends on other processes in the local supervision tree. I'm assuming since it's talking about Ecto inserts it brings over and starts the whole apps supervision tree on the function executor but that may or may not be desired for various reasons.
It starts your whole app, including the whole supervision tree, but you can turn on/off services based on whatever logic you want. I talk a bit about this in the screencast. For example, no need to start the phoenix endpoint (webserver) since we aren't serving web traffic. For the DB pool, you'd set a lower pool size or single connection in your runtime configuration based on the presence of FLAME parent or not.
This is one reason I really don't like US headline casing as enforced by HN - it looks like Serverless, as in the capital-S company, serverless.com, is what's being rethought, not the small-s principle.
(Aside: I wish someone would rethink Serverless, heh.)
I think the casing is not enforced by HN but rather up to the poster?
> (Aside: I wish someone would rethink Serverless, heh.)
Not sure if you've checked out https://sst.dev/ but I think they've done precisely that. For example, they have Live Lambda Development which makes local dev a real breeze by significantly shortening feedback loops (no need to push your code up to the cloud and wait for it to deploy)
You can override it as the poster, but you have to edit it back to what you wanted after initially submitting to do so. (And I suppose I don't know if that's intentional or just the way it happens to be.) If you submit 'Foo bar' it will be made to be 'Foo Bar'.
This is a very neat approach and I agree with the premise that we need a framework that unifies some of the architecture of cloud - shuttle.rs has some thoughts here. I do take issue with this framing:
- Trigger the lambda via HTTP endpoint, S3, or API gateway ($)
* Pretending that starting a fly machine doesn't cost the same as triggering via s3 seems disingenuous.
- Write the bespoke lambda to transcode the video ($)
* In go this would be about as difficult as flame -- you'd have to build a different entrypoint that would be 1 line of code but it could be the same codebase. Node it would depend on bundling but in theory you could do the same -- it's just a promise that takes an S3 event, that doesn't seem much different.
- Place the thumbnail results into SQS ($)
* I wouldn't do this at all. There's no reason the results need to be queued. Put them in a deterministically named s3 bucket where they'll live and be served from. Period.
- Write the SQS consumer in our app (dev $)
* Again -- this is totally unnecessary. Your application *should forget* it dispatched work. That's the point of dispatching it. If you need subscribers to notice it or do some additional work I'd do it differently rather than chaining lambdas.
- Persist to DB and figure out how to get events back to active subscribers that may well be connected to other instances than the SQS consumer (dev $)
* Your lambda really should be doing the DB work not your main application. If you've got subscribers waiting to be informed the lambda can fire an SNS notification and all subscribed applications will see "job 1234 complete"
So really the issue is:
* s3 is our image database
* our app needs to deploy an s3 hook for lambda
* our codebase needs to deploy that lambda
* we might need to listen to SNS
which is still some complexity, but it's not the same and it's not using the wrong technology like some chain of SQS nonsense.
Thanks for the thoughts – hopefully I can make this more clear:
> * Pretending that starting a fly machine doesn't cost the same as triggering via s3 seems disingenuous.
You're going to be paying for resources wherever you decide to run your code. I don't think this needs to be spelled out. The point about costs is rather than paying to run "my app", I'm paying at multiple layers to run a full solution to my problem. Lambda gateway requests, S3 put, SQS insert, each have their own separate costs. You pay a toll at every step instead of a single step on Fly or wherever you host your app.
> * I wouldn't do this at all. There's no reason the results need to be queued. Put them in a deterministically named s3 bucket where they'll live and be served from. Period. This is totally unnecessary. Your application should forget it dispatched work. That's the point of dispatching it. If you need subscribers to notice it or do some additional work I'd do it differently rather than chaining lambdas.
You still need to tell your app about the generated thumbnails if you want to persist the fact they exist where you placed them in S3, how many exist, where you left off, etc.
> * Your lambda really should be doing the DB work not your main application. If you've got subscribers waiting to be informed the lambda can fire an SNS notification and all subscribed applications will see "job 1234 complete"
This is exactly my point. You bolt on ever more Serverless offerings to accomplish any actual goal of your application. SNS notifications is exactly the kind of thing I don't want to think about, code around, and pay for. I have Phoenix.PubSub.broadcast and I continue shipping features. It's already running on all my nodes and I pay nothing for it because it's already baked into the price of what I'm running – my app.
> This is exactly my point. You bolt on ever more Serverless offerings to accomplish any actual goal of your application. SNS notifications is exactly the kind of thing I don't want to think about, code around, and pay for. I have Phoenix.PubSub.broadcast and I continue shipping features. It's already running on all my nodes and I pay nothing for it because it's already baked into the price of what I'm running – my app.
I think this is fine if and only if you have an application that can subscribe to PubSub.broadcast. The problem is that not everything is Elixir/Erlang or even the same language internally to the org that runs it. The solution (unfortunately) seems to be reinventing everything that made Erlang good but for many general purpose languages at once.
I see this more as a mechanism to signal the runtime (combination of fly machines and erlang nodes running on those machines) you'd like to scale out for some scoped duration, but I'm not convinced that this needs to be initiated from inside the runtime for erlang in most cases -- why couldn't something like this be achieved externally noticing the a high watermark of usage and adding nodes, much like a kubernetes horizontal pod autoscaler?
Is there something specific about CPU bound tasks that makes this hard for erlang that I'm missing?
Also, not trying to be combative -- I love Phoenix framework and the work y'all are doing at fly, especially you Chis, just wondering if/how this abstraction leaves the walls of Elixir/Erlang which already has it significantly better than the rest of us for distributed abstractions.
You're literally describing what we've built at https://www.inngest.com/. I don't want to talk about us much in this post, but it's so relevant it's hard not to bring it up. (Huge disclaimer here, I'm the co-founder).
In this case, we give you global event streams with a durable workflow engine that any language (currently Typescript, Python, Go, Elixir) can hook into. Each step (or invocation) is backed by a lightweight queue, so queues are cheap and are basically a 1LOC wrapper around your existing code. Steps run as atomic "transactions" which must commit or be retried within a function, and are as close to exactly once as you could get.
I don't know if I agree with the argument regarding durability vs elastic execution. If I can get both (with a nice API/DX) via something like Temporal (https://github.com/temporalio/temporal), what's the drawback here?
As an alternative to Lambdas I can see this being useful.
However, the overhead concerns me. This would only make sense in a situation where the function in question takes long enough that the startup overhead doesn't matter or where the main application is running on hardware that can't handle the resource load of many instances of the function in question.
I'm still, I think, in the camp of "monoliths are best in most cases." It's nice to have this in the toolbox, though, for those edge cases.
I don't think this goes against "monoliths are best in most cases" at all. In fact it supports that by letting you code like it's all one monolith, but behind-the-scenes it spins up the instance.
Resource-wise if you had a ton of unbounded concurrency then that would be a concern as you could quickly hit instance limits in the backend, but the pooling strategy discussed lower in the post addresses that pretty well, and gives you a good monitoring point as well.
This is great! It reminds me of a (very lightweight) Elixir specific version of what we built at https://www.inngest.com/.
That is, we both make your existing code available to serverless functions by wrapping with something that, essentially, makes the code callable via remote-RPC .
Some things to consider, which are called out in the blog post:
Often code like this runs in a series of imperative steps. Each of these steps can run in series or parallel as additional lambdas. However, there's implicit state captured in variables between steps. This means that functions become workflows. In the Inngest model, Inngest captures this state and injects it back into the function so that things are durable.
On the note of durability, these processes should also be backed by a queue. The good thing about this model is that queues are cheap. When you make queues cheap (eg. one line of code) everything becomes easy: any developer can write reliable code without worrying about infra.
Monitoring and observability, as called out, is critical. Dead letter queues suck absolute major heaving amounts of nauseous air, and being able to manage and replay failing functions or steps is critical.
A couple differences wrt. FLAME and Inngest. Inngest is queue backed, event-driven, and servable via HTTP across any language. Because Inngest backs your state externally, you can write a workflow in Elixir, rewrite it in Typescript, redeploy, and running functions live migrate across backend languages, similar to CRIU.
Being event-driven allows you to manage flow control: everything from debounce to batching to throttling to fan-out, across any runtime or language (eg. one Elixir app on Fly can send an event over to run functions on TypeScript + Lambda).
I'm excited where FLAME goes. I think there are similar goals!
Ingest looks like an awesome service! I talk about job processors/durability/retries in the post. For Elixir specifically for durability, retries, and workflows we reach for Oban, which we'd continue to do here. The Oban job would call into FLAME to handle the elastic execution.
The FLAME.Pool discussed later in the post addresses this. Runners are pooled and remain configurable hot for whatever time you want before idling down. Under load you are rarely paying the cold start time because the pool is already hot. We are also adding more sophisticated pool growth techniques to the Elixir library next so you also avoid hitting an at capacity runner and cold starting one.
For hot runners, the only overhead is the latency between the parent and child, which should be the same datacenter so 1ms or sub 1ms.
Currently the per-runner concurrency is limited by a fixed number. Have you thought about approaches that instead base this on resource usage, so that runners can be used optimally?
Yes, more sophisticated pool growth options is something I want longer term. We can also provide knobs that will let you drive the pool growth logic yourself if needed.
Cold start time is the issue with most serverless runtimes.
Your own mission statement states: "We want on-demand, granular elastic scale of specific parts of our app code." Doing that correctly is fundamentally a question of how long you need to wait for cold starts, because if you have a traffic spike, the spiked part of the traffic is simply not being served until the cold start period elapses. If you're running hot runners with no load, or if you have incoming load without runners (immediately) serving them, then you're not really delivering on your goal here. AWS EC2 has had autoscaling groups for more than a decade, and of course, a VM is essentially a more elaborate wrapper for any kind of application code you can write, and one with a longer cold-start time.
> Under load you are rarely paying the cold start time because the pool is already hot.
Depending of course on the workload and request volume, I imagine you could apply a strategy where code is run locally while waiting for a remote node to start up, so you can still serve the requests on time?
No, because then you're dividing the resources allocated to the function among the existing run + the new run. If you over-allocate ahead of time to accommodate for this, you might as well just run ordinary VMs, which always have excess allocation locally; the core idea of scaling granularly is that you only allocate the resources you need for that single execution (paying a premium compared to a VM but less overall for spiky workloads since less overhead will be wasted).
We took an alternative approach with https://www.windmill.dev which is to consider the unit of abstraction to be at the source code level rather than the container level. We then parse the main function, and imports to extract the args and dependencies, and then run the code as is in the desired runtime (typescript, python, go, bash). Then all the secret sauce is to manage the cache efficiently so that the workers are always hot regardless of your imports
It's not as integrated in the codebase as this, but the audience is different, our users build complex workflows from scratch, cron jobs, or just one-off scripts with the auto-generated UI. Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required. That's what we are currently exploring for integrating better Windmill with existing codebase instead of having to rely on http calls.
> Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required.
This isn't strictly what is happening. FLAME just uses the BEAM's built in clustering features to call a function on a remote node. That implicitly handles transferring only the context that is necessary. From the article:
> FLAME.call accepts the name of a runner pool, and a function. It then finds or boots a new copy of our entire application and runs the function there. Any variables the function closes over (like our %Video{} struct and interval) are passed along automatically.
Fair point, TIL about another incredible capability of the BEAM. As long as you're willing to write Elixir, this is clearly a superior scheme for deferred tasks/background jobs.
One issue I see with this scheme still is that you have to be careful of what you do at initialization of the app since now all your background jobs are gonna run that. For instance, maybe your task doesn't need to be connected to the db and as per the article it will if your app does. They mention having hot-modules, but what if you want to run 1M of those jobs on 100 workers, you now have a 100 unnecessary apps. It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
This is actually a feature. If you watch the screencast, I talk about Elixir supervision trees and how all Elixir programs carefully specify the order their services stop and stop in. So if your flame functions need DB access, you start your Ecto.Repo with a small or single DB connection pool. If not, you flip it off.
> It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
So, Chris, how do you envision the FLAME child understanding what OTP children it needs to start on boot, because this could be FLAME.call dependent if you have multiple types of calls as described above. Is there a way to pass along that data or for it to be pulled from the parent?
Acknowledging this is brand new; just curious what your thinking is.
EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
Good question. The pools themselves in your app will be per usecase, and you can reference the named pool you are a part of inside the runner, ie by looking in system env passed as pool options. That said, we should probably just encode the pool name along with the other parent info in the `%FLAME.Parent{}` for easier lookup
Ah, that makes a lot of sense - I think the FLAME.Parent{} approach may enable backends that wouldn't be possible otherwise.
For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level.
If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more.
Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off.
EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness)
This looks fantastic! At my last gig we had exactly the “nuts” FaaS setup described in the article for generating thumbnails and alternate versions of images and it was a source of unnecessary complexity.
It's indeed very powerful and there are jobs out there. Besides being an excellent modern toolbox for lots of problems (scaling, performance, maintenance) and having the arguably best frontend-tech in the industry (LiveView), the Phoenix framework also is the most loved web framework and elixir itself the 2nd most loved language according to the stackoverflow survey.
Its still a more exotic choice of a tech stack, and IMO its best suited for when you have fewer but more senior devs around, this is where it really shines. But I also found that phoenix codebase survived being "tortured" by a dozen juniors over years quite well.
I basically make my money solely with Elixir and have been for ~5 years now, interrupted only by gigs as a devops for the usual JS nightmares including serverless (where the cure always has been rewriting to Elixir/Phoenix at the end).
Having dealt with the pain and complexity of a 100+ lambda function app for the last 4 years, I must say this post definitely hits the spot wrt. the downsides of FaaS serverless architectures.
When starting out, these downsides are not really that visible. On the contrary, there is a very clear upside, which is that everything is free when you have low usage, and you have little to no maintenance.
It is only later, when you have built a hot mess of lambda workflows, which become more and more rigid due to interdependencies, that you wish you had just gone the monolith route and spent the few extra hundreds on something self-managed. (Or even less now, e.g. on fly.io)
I talk about FLAME outside elixir in one one of the sections in the blog. The tldr; is it's a generally applicable pattern for languages with a reasonable concurrency model. You likely won't get all the ergonomics that we get for free like functions with captured variable serialization, but you can probably get 90% of the way there in something like js, where you can move your modular execution to a new file rather than wrapping it in a closure. Someone implementing a flame library will also need to write the pooling, monitoring, and remote communication bits. We get a lot for free in Elixir on the distributed messaging and monitoring side. The process placement stuff is also really only applicable to Elixir. Hope that helps!
That's just standard erlang/elixir- because all values are immutable, when a new anonymous function is defined it copies the current value of the external variables into it.
You can do it right now even without Flame, just by opening two Elixir nodes, then it's as simple as
Note that while the string interpolation and `IO.puts` was run on `other_node@localhost`, it still did stdout from the first node- this is because it was the one that called `Node.spawn_link`, making it the 'group leader'. Outside of which stdout it went to, all the work was done in the other node.
JS kind of has inter process built in with Web Workers and the channel messaging API- I wonder whether it'd be possible to essentially create a "FLAME Worker" with the same API as a web worker but backed by distributed execution!
A pattern I see over and over, which has graduated to somewhere between a theorem and a law, is that motivated developers can make just about any process or architecture work for about 18 months.
By the time things get bad, it's almost time to find a new job, especially if the process was something you introduced a year or more into your tenure and are now regretting. I've seen it with a handful of bad bosses, at least half a dozen times with (shitty) 'unit testing', scrum, you name it.
But what I don't know is how many people are mentally aware of the sources of discomfort they feel at work, instead of a more nebulous "it's time to move on". I certainly get a lot of pushback trying to name uncomfortable things (and have a lot less bad feelings about it now that I've read Good to Great). Nobody wants to say, "Oh look, the consequences of my actions."
The people materially responsible for the Rube Goldberg machine I help maintain were among the first to leave. The captain of that ship asked a coworker of mine if he thought it would be a good idea to open source our engine. He responded that nobody would want to use our system when the wheels it reinvented already exist (and are better). That guy was gone within three to four months, under his own steam.
That's why I'm always wary of people who hardly ever seem to stay anywhere more than a couple of years.
There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. Understanding how tech debt comes to be, what tradeoffs were involved and how they came to bite later. Which ideas turned out to be bad in hindsight through the lens of the people making them at the time.
Rather than just painting the previous crowd as incompetent while simultaneously making worse decisions you'll never experience the consequences of.
Moving on every 18-24 months leaves you with a potentially false impression of your own skills/wisdom.
> There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle.
This is so true. It's extremely enlightening to watch a design go from design docs to implementation and then finally the maintenance phase. A lot of problems that could happen never do and some unexpected ones pop up along the way.
>That's why I'm always wary of people who hardly ever seem to stay anywhere more than a couple of years
Do have some empathy for when job markets or life make people move on. I'd like to stay at an employer for more than 1-2 years, but between the layoffs (avoiding them or getting laid off) or the need for a higher salary that often only comes from switching jobs, its not always possible to build tenure.
Frankly its a big issue in the industry at large. I hate interviewing etc. but I'm not going to get paid 20% less because of it. I had to ride the job hopping treadmill for awhile, and I'd like to get off it just as much as you'd like to see people have more tenure.
When I first started out though, I worked for 5 years at the same place until it was very clear I was going to cap out on being able to advance and merit increases of 3-5% a year aren't going to cut it.
Yep. Absolutely this. I get the frustration of people building fragile junk and bouncing before the house of cards falls, but I wouldn’t hold switching jobs every couple of years against a candidate when a significant difference in salary is potentially on the table. A 20% raise, compounded over a couple of switches, is massive.
I got way more by switching jobs (after 4 years) than I was ever going to get by staying.
It definitely colors which questions I want to ask people.
It is possible to avoid these traps, but then there are a lot of traps that we collectively have the wisdom to avoid but individually do not.
I started taking things apart and putting them back together at a very young age. When I was a young man I was in a big hurry to get somewhere, and so I could walk into a brownfield project and slowly reconstruct and deconstruct how we got here, would I have made the same decisions with the same information, and how do I feel about this news. Not only was I not falling into the "1 year of experience 10 times" dilema, I got more like 3-4 years of experience 4 times in 10 years, by playing historian.
My first almost-mentor left me with a great parting gift at the start of the dot-com era. He essentially convinced me of the existence of the hype cycle (in '95!), that we were at the beginning of one/several, he had seen previous ones play out, and they would play out again. Not cycles for new things, mind you, but cycles trying to bring back old things that had been forgotten. Like fashion. If anything it made me more likely to want to excavate the site.
Going into the trap knowing it's a trap doesn't necessarily save you, but it does improve your odds. Of course it also makes you a curmudgeon at a tender age.
I am in a long-term-career place and it has its challenges (mostly no leverage on pay...) but the level of knowledge and expertise and proficiency of most people is astounding, allowing to do a whole lot with very little people, and to build on knowledge, experience and yes, lots of code and tooling too. Most everything you're thinking about in the domain, has been done at least twice, and you can pick people's brains on the most obscure but important topics and get a pretty good idea on the actual challenges.
It can get hard to separate ideas from execution, to 'forget' about the pain of a horrible technical move that still managed to be a commercially success, or to forget the pain of a badly-timed big decision that exploded in your face.
I have spent so much time fighting to do things the boring way.
There's only so much interesting code you can add to interesting code, before every single conversation becomes a proxy discussion of The Mythical Man Month - we can't teach anybody new how to use this code in anything like a reasonable time frame. The best we can do is create new experts as fast as the old ones disappear.
Really Interesting Code is more at home surrounded by Really Fucking Boring code. You get so blinded by the implementation details of the Interesting Code that you cannot see the forest for the trees. That is the secret wisdom of Relentless Refactoring. The more I rearrange the pieces the more things I can see that can be made out of them, some of which are better, and a few of which are much better or brilliant.
Last week I implemented something in 2 days that we wanted years ago and didn't do because it would have taken more than a month of 1.5 people working on it to fix. But after a conversation in which I enumerated all the little islands of sanity I had created between Here and There, it took a couple dozen lines of code to fix a very old and very sore problem.
I couldn't even stand having a dozen lambdas. The app was originally built by someone who didn't think much about maintenance or deployment. Code was copy-pasted all over the place. Eventually, we moved to a "fat lambda" monolith where a single lambda serves multiple endpoints.
> that you wish you had just gone the monolith route
Going from hundreds of lambdas to a monolith is overreacting to one extreme by going the other one. There's a whole spectrum of possible ways to split a project in useful ways, which simplify development and maintenance.
Anything in between is all the downsides of both approaches.
Once you have the flow for deploying always running hot application(s) with autoscaling the benefits of lambda are basically gone.
Low volume scale to zero is just another route. No 15 minute limit, no having to marshal data through other AWS services because that's all lambda talks to, no more eventbridge for cron, no more payload size limits and having to use S3 as buffer, no more network requests between different parts of the same logical app, code deploys are atomic you're either at v1.x or v1.x+1 but never some in-between state.
I really do like Lambda but once you're at the spend where it's the same as some dedicated always-on compute the value drops off.
I'm working on something that I think might solve the problem in any language (currently have an sdk for typescript, and java in the works). You can avoid splitting an application into 100s of small short-running chunks if you can write normal service-orientated code, where lambdas can call each other. But this isn't possible without paying for all that time waiting around. If the Lambdas can pause execution while they are blocked on IO, it solves the problem. So I think durable execution might be the answer!
I've been working on a blog post to show this off for the last couple of weeks:
I don't get why you'd have a 100+ lambda function app... i can see purpose built lambdas (ie we have one for "graphql" and "frontend" and a few backend services) but unless you're at Meta size, why would you have 100 lambdas? Do you have 100 teams?
They handle scaling at only highest level, similar to spinning up more dynos/workers/webservers like I talk about in the intro. FLAME is about elastically scaling individual hot operations of your app code. App Engine and such are about scaling at the level of your entire app/container. Splitting your operations into containers then breaks the monolith into microservice pieces and introduces all the downsides I talk about in the post. Also, while it's your code/language, you still need to interface with the mount of proprietary offerings to actual accomplish your needs.
> Splitting your operations into containers then breaks the monolith into microservice pieces and introduces all the downsides
A pattern here is to not split the monolith and use the same container for your main app and hot operations. The hot operations just need some different configuration eg. container args or env vars
152 comments
[ 2.9 ms ] story [ 181 ms ] threadServerless with a good local dev story. Nice!
If you don’t understand how the OS works at a very fundamental level, you will write sub-optimal code. This is usually fine, because people are smart, compilers are good, and hardware is absurdly fast. However, the fact remains that it’s probably giving up some bits here and there.
More germane to my point, though, the amount of libraries and tooling and abstractions being spun up to avoid having to run a server seems like more effort than its worth. Serverless sometimes has cost merits, but as a counterpoint, you can get a 1 year RI micro EC2 for something like $3/month. If your app is struggling to run on that, either you have the scale to justify more cash outlay, or your app has performance issues – and now we’re full circle back to the previous paragraph.
Everything has trade-offs. You don’t generally get good, fast, and cheap.
- Flame still assumes you have a server somewhere- this is purely for elastic workloads.
- Dynamically scaling horizontally is not something "linux" can do for you: It's a legitimately hard problem because of all the usual distributed systems difficulties.
- Often, there are workloads that are only needed occasionally but also have very different resource needs than the rest of the system. Being forced to not only create a new version of your application, but additionally a bespoke server setup and dedicated VM/Container image for that case takes something that could be trivial in the Flame model, and makes it take hours even for someone experienced.
Basically, you really should've read the article first.
It can, depending on your definition of horizontal, and whether you accept that tools like HAProxy fall under the realm of “Linux administration.”
systemd can spawn N services from a single definition file, which can then register with the load balancer. A small shell script could monitor those service cgroups for basic metrics like memory or CPU utilization, and then command systemd to scale up/down. This is horizontal from the app’s perspective.
I’m not suggesting the above is a great idea, and you’d probably be better off with container orchestration of some kind – but tbf, all that’s doing is encapsulating cgroups and monitoring consumption to command scaling.
> function and have that block of code run in a temporary copy of your app.
That's interesting, sounds like what fork does but for serverless. Great work
One could really up a whole Hetzner/OVH server and create a KVM for the workload on the fly!!
https://www.bbkane.com/blog/azure-functions-with-terraform/
Don’t have strong feelings there. It worked. I did have some issues with upgrading the functions but found the work arounds.
Those "weird assumptions" are what makes the experience wonderful for the happy path. If you use the C#/v4 model, I can't imagine you'd have a hard time. Azure even sets up the CI/CD for you automatically if your functions are hosted in Github.
If your functions need to talk to SQL, you should be using Managed Identity authentication between these resources. We don't have any shared secrets in our connection strings today. We use Microsoft Auth to authenticate access to our HttpTrigger functions. We take a dep on IClaimsPrincipal right in the request and everything we need to know about the user's claims is trivially available.
I have zero experience using Azure Functions outside of the walled garden. If you are trying to deploy python or rust to Az Functions, I can imagine things wouldn't be as smooth. Especially, as you get into things like tracing, Application Insights, etc.
I feel like you should only use Microsoft tech if you intend to drink a large amount of their koolaid. The moment you start using their tooling with non C#/.NET stacks, things go a bit sideways. You might be better off in a different cloud if you want to use their FaaS runners in a more "open" way. If you can figure out how to dose yourself appropriately with M$ tech, I'd argue the dev experience is unbeatable.
Much of the Microsoft hate looks to me like a stick-in-bike-wheels meme. You can't dunk on the experience until you've tried the one the chef actually intended. Dissecting your burger and only eating a bit of the lettuce is not a thorough review of the cuisine on offer.
But Microsoft isn't selling burgers that people are taking a bit of lettuce from. They're selling lettuce, and if that lettuce sucks in any context that isn't the burger that they're also selling, then complaining about the quality of their lettuce is valid.
I think our “best” issue is how sometimes our functions won’t be capable of connecting to their container registry. For no apparent reason. We have an Entra ID group that has Arc Pull access and we add new function slot identities to it automatically, and for the most part it works fine, but then suddenly it won’t.
We currently has a pipeline slot, that is completely the same as the four other slots, which will only deploy a new function version if you deploy it twice through the Azure pipeline. We’ve had Microsoft look at it and their suggestion was to delete the pipeline and create it again.
And that’s just part of it. Then comes the VNETs, the subnets the private endpoints and how both subscriptions and resource groups make everything a tiny battle.
I don’t really mind that much, we’re abandoning it in favour of Azure Container Apps and a fully run Bicep + Dapr pipeline and we’re never looking back.
Though, to be fair to Microsoft, the way they designed Azure Functions makes the migration away from them really, really, easy. Which is frankly a brilliant design for the more managed side of “serverless” in my opinions. It’s just a shame that the managed part just doesn’t work very well. The functions themselves work fine, well maybe not if you don’t use .net in isolation, but I can’t speak about that as we weren’t going to trust Microsoft to update our dependencies (even if they are Microsoft SDKs).
And now I feel (a tiny bit) bad for sniping ffmpeg.fly.dev :)
It’s easy to dismiss these types of articles as straight sales pitches when the problem is presented as “…a fate worse than death.” and the solution so easy and painless, you’d be a fool not to dismiss any other approach! Thats the MO of snake oil sellers.
Instead, I think a more objective comparison with much less editorializing would be less of a turn off for folks curious about a better approach to solve a problem that you did well in outlining at the beginning of the article.
Hopefully you take this as constructive feedback! I think the substance of the article is interesting, I was just turned off by the presentation.
> CPU bound work like video transcoding can quickly bring our entire service to a halt in production
Couldn't you just autoscale your app based on cpu though?
Or they require a GPU.
Or your core service only needs 1-2 servers, but you need to scale up to dozens/hundreds/thousands on demand, for work that only happens maybe once a day.
i think it's cool tech, but none of those things are "hair on fire" problems for me. i'm sure they are for some people.
Serverless implies more that you’re not the one managing the machine. Also connects to the cattle not pets paradigm
I agree that it's a dumb name that doesn't really say what it means and "ephemeral" might be better, but "serverless" is better than "clouds-you-rent-by-the-second".
* OpenFaaS - https://www.openfaas.com/blog/porting-existing-containers-to...
* AWS Lambda - https://docs.aws.amazon.com/prescriptive-guidance/latest/pat...
* Scaleway Serverless Containers - https://www.scaleway.com/en/serverless-containers/
* Azure Container Instances - https://learn.microsoft.com/en-us/azure/container-instances/...
Probably others too, those are just the ones I know off the top of my head. I see very little reason to use traditional Function-based FaaS, which forces you into a special, locked-in framework, instead of using containers that work everywhere.
One potential gotcha I'm curious if there is a good story for is if it can guard against code that depends on other processes in the local supervision tree. I'm assuming since it's talking about Ecto inserts it brings over and starts the whole apps supervision tree on the function executor but that may or may not be desired for various reasons.
(Aside: I wish someone would rethink Serverless, heh.)
> (Aside: I wish someone would rethink Serverless, heh.)
Not sure if you've checked out https://sst.dev/ but I think they've done precisely that. For example, they have Live Lambda Development which makes local dev a real breeze by significantly shortening feedback loops (no need to push your code up to the cloud and wait for it to deploy)
- Trigger the lambda via HTTP endpoint, S3, or API gateway ($)
- Write the bespoke lambda to transcode the video ($) - Place the thumbnail results into SQS ($) - Write the SQS consumer in our app (dev $) - Persist to DB and figure out how to get events back to active subscribers that may well be connected to other instances than the SQS consumer (dev $) So really the issue is:* s3 is our image database
* our app needs to deploy an s3 hook for lambda
* our codebase needs to deploy that lambda
* we might need to listen to SNS
which is still some complexity, but it's not the same and it's not using the wrong technology like some chain of SQS nonsense.
> * Pretending that starting a fly machine doesn't cost the same as triggering via s3 seems disingenuous.
You're going to be paying for resources wherever you decide to run your code. I don't think this needs to be spelled out. The point about costs is rather than paying to run "my app", I'm paying at multiple layers to run a full solution to my problem. Lambda gateway requests, S3 put, SQS insert, each have their own separate costs. You pay a toll at every step instead of a single step on Fly or wherever you host your app.
> * I wouldn't do this at all. There's no reason the results need to be queued. Put them in a deterministically named s3 bucket where they'll live and be served from. Period. This is totally unnecessary. Your application should forget it dispatched work. That's the point of dispatching it. If you need subscribers to notice it or do some additional work I'd do it differently rather than chaining lambdas.
You still need to tell your app about the generated thumbnails if you want to persist the fact they exist where you placed them in S3, how many exist, where you left off, etc.
> * Your lambda really should be doing the DB work not your main application. If you've got subscribers waiting to be informed the lambda can fire an SNS notification and all subscribed applications will see "job 1234 complete"
This is exactly my point. You bolt on ever more Serverless offerings to accomplish any actual goal of your application. SNS notifications is exactly the kind of thing I don't want to think about, code around, and pay for. I have Phoenix.PubSub.broadcast and I continue shipping features. It's already running on all my nodes and I pay nothing for it because it's already baked into the price of what I'm running – my app.
I think this is fine if and only if you have an application that can subscribe to PubSub.broadcast. The problem is that not everything is Elixir/Erlang or even the same language internally to the org that runs it. The solution (unfortunately) seems to be reinventing everything that made Erlang good but for many general purpose languages at once.
I see this more as a mechanism to signal the runtime (combination of fly machines and erlang nodes running on those machines) you'd like to scale out for some scoped duration, but I'm not convinced that this needs to be initiated from inside the runtime for erlang in most cases -- why couldn't something like this be achieved externally noticing the a high watermark of usage and adding nodes, much like a kubernetes horizontal pod autoscaler?
Is there something specific about CPU bound tasks that makes this hard for erlang that I'm missing?
Also, not trying to be combative -- I love Phoenix framework and the work y'all are doing at fly, especially you Chis, just wondering if/how this abstraction leaves the walls of Elixir/Erlang which already has it significantly better than the rest of us for distributed abstractions.
In this case, we give you global event streams with a durable workflow engine that any language (currently Typescript, Python, Go, Elixir) can hook into. Each step (or invocation) is backed by a lightweight queue, so queues are cheap and are basically a 1LOC wrapper around your existing code. Steps run as atomic "transactions" which must commit or be retried within a function, and are as close to exactly once as you could get.
However, the overhead concerns me. This would only make sense in a situation where the function in question takes long enough that the startup overhead doesn't matter or where the main application is running on hardware that can't handle the resource load of many instances of the function in question.
I'm still, I think, in the camp of "monoliths are best in most cases." It's nice to have this in the toolbox, though, for those edge cases.
Resource-wise if you had a ton of unbounded concurrency then that would be a concern as you could quickly hit instance limits in the backend, but the pooling strategy discussed lower in the post addresses that pretty well, and gives you a good monitoring point as well.
That is, we both make your existing code available to serverless functions by wrapping with something that, essentially, makes the code callable via remote-RPC .
Some things to consider, which are called out in the blog post:
Often code like this runs in a series of imperative steps. Each of these steps can run in series or parallel as additional lambdas. However, there's implicit state captured in variables between steps. This means that functions become workflows. In the Inngest model, Inngest captures this state and injects it back into the function so that things are durable.
On the note of durability, these processes should also be backed by a queue. The good thing about this model is that queues are cheap. When you make queues cheap (eg. one line of code) everything becomes easy: any developer can write reliable code without worrying about infra.
Monitoring and observability, as called out, is critical. Dead letter queues suck absolute major heaving amounts of nauseous air, and being able to manage and replay failing functions or steps is critical.
A couple differences wrt. FLAME and Inngest. Inngest is queue backed, event-driven, and servable via HTTP across any language. Because Inngest backs your state externally, you can write a workflow in Elixir, rewrite it in Typescript, redeploy, and running functions live migrate across backend languages, similar to CRIU.
Being event-driven allows you to manage flow control: everything from debounce to batching to throttling to fan-out, across any runtime or language (eg. one Elixir app on Fly can send an event over to run functions on TypeScript + Lambda).
I'm excited where FLAME goes. I think there are similar goals!
https://github.com/inngest/ex_inngest
So for each “Flame.call” it begins a whole new app process and copies the execution context in?
A very simple solution to scaling, but I’d imagine this would have some disadvantages…
Adding 10ms to the app startup time, adds 10ms to every “Flame.call” part of the application too… same with memory I suppose
I guess these concerns just need to be consider when using this system
For hot runners, the only overhead is the latency between the parent and child, which should be the same datacenter so 1ms or sub 1ms.
Your own mission statement states: "We want on-demand, granular elastic scale of specific parts of our app code." Doing that correctly is fundamentally a question of how long you need to wait for cold starts, because if you have a traffic spike, the spiked part of the traffic is simply not being served until the cold start period elapses. If you're running hot runners with no load, or if you have incoming load without runners (immediately) serving them, then you're not really delivering on your goal here. AWS EC2 has had autoscaling groups for more than a decade, and of course, a VM is essentially a more elaborate wrapper for any kind of application code you can write, and one with a longer cold-start time.
> Under load you are rarely paying the cold start time because the pool is already hot.
My spiky workloads beg to differ.
Reminds me of 12-factor app (https://12factor.net/) especially "VI. Processes" and "IX. Disposability"
We took an alternative approach with https://www.windmill.dev which is to consider the unit of abstraction to be at the source code level rather than the container level. We then parse the main function, and imports to extract the args and dependencies, and then run the code as is in the desired runtime (typescript, python, go, bash). Then all the secret sauce is to manage the cache efficiently so that the workers are always hot regardless of your imports
It's not as integrated in the codebase as this, but the audience is different, our users build complex workflows from scratch, cron jobs, or just one-off scripts with the auto-generated UI. Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required. That's what we are currently exploring for integrating better Windmill with existing codebase instead of having to rely on http calls.
This isn't strictly what is happening. FLAME just uses the BEAM's built in clustering features to call a function on a remote node. That implicitly handles transferring only the context that is necessary. From the article:
> FLAME.call accepts the name of a runner pool, and a function. It then finds or boots a new copy of our entire application and runs the function there. Any variables the function closes over (like our %Video{} struct and interval) are passed along automatically.
One issue I see with this scheme still is that you have to be careful of what you do at initialization of the app since now all your background jobs are gonna run that. For instance, maybe your task doesn't need to be connected to the db and as per the article it will if your app does. They mention having hot-modules, but what if you want to run 1M of those jobs on 100 workers, you now have a 100 unnecessary apps. It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
> It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
Exactly :)
Acknowledging this is brand new; just curious what your thinking is.
EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level.
If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more.
Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off.
EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness)
[1] https://github.com/supabase/libcluster_postgres
[2] https://devcenter.heroku.com/articles/platform-api-reference...
[3] https://devcenter.heroku.com/articles/platform-api-reference...
I love the project's goals; I'm really hoping Windmill becomes a superior open-source Retool/Airtable alternative!
Elixir looks ridiculously powerful. How's the job market for Elixir -- could one expect to have a chance at making money writing Elixir?
Its still a more exotic choice of a tech stack, and IMO its best suited for when you have fewer but more senior devs around, this is where it really shines. But I also found that phoenix codebase survived being "tortured" by a dozen juniors over years quite well.
I basically make my money solely with Elixir and have been for ~5 years now, interrupted only by gigs as a devops for the usual JS nightmares including serverless (where the cure always has been rewriting to Elixir/Phoenix at the end).
When starting out, these downsides are not really that visible. On the contrary, there is a very clear upside, which is that everything is free when you have low usage, and you have little to no maintenance.
It is only later, when you have built a hot mess of lambda workflows, which become more and more rigid due to interdependencies, that you wish you had just gone the monolith route and spent the few extra hundreds on something self-managed. (Or even less now, e.g. on fly.io)
A question for author: what if not using Elixir?
Can't wait for the deep dive on how that works
You can do it right now even without Flame, just by opening two Elixir nodes, then it's as simple as
```elixir
iex(first_node@localhost)> name = "Santa"
iex(first_node@localhost)> Node.spawn_link(:other_node@localhost, fn -> IO.puts "Hello #{name}" end)
Hello Santa
#PID<1337.42.0>
```
Note that while the string interpolation and `IO.puts` was run on `other_node@localhost`, it still did stdout from the first node- this is because it was the one that called `Node.spawn_link`, making it the 'group leader'. Outside of which stdout it went to, all the work was done in the other node.
From a quick look at the code, this looks the magic line: https://github.com/phoenixframework/flame/blob/main/lib/flam...
By the time things get bad, it's almost time to find a new job, especially if the process was something you introduced a year or more into your tenure and are now regretting. I've seen it with a handful of bad bosses, at least half a dozen times with (shitty) 'unit testing', scrum, you name it.
But what I don't know is how many people are mentally aware of the sources of discomfort they feel at work, instead of a more nebulous "it's time to move on". I certainly get a lot of pushback trying to name uncomfortable things (and have a lot less bad feelings about it now that I've read Good to Great). Nobody wants to say, "Oh look, the consequences of my actions."
The people materially responsible for the Rube Goldberg machine I help maintain were among the first to leave. The captain of that ship asked a coworker of mine if he thought it would be a good idea to open source our engine. He responded that nobody would want to use our system when the wheels it reinvented already exist (and are better). That guy was gone within three to four months, under his own steam.
There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. Understanding how tech debt comes to be, what tradeoffs were involved and how they came to bite later. Which ideas turned out to be bad in hindsight through the lens of the people making them at the time.
Rather than just painting the previous crowd as incompetent while simultaneously making worse decisions you'll never experience the consequences of.
Moving on every 18-24 months leaves you with a potentially false impression of your own skills/wisdom.
This is so true. It's extremely enlightening to watch a design go from design docs to implementation and then finally the maintenance phase. A lot of problems that could happen never do and some unexpected ones pop up along the way.
Do have some empathy for when job markets or life make people move on. I'd like to stay at an employer for more than 1-2 years, but between the layoffs (avoiding them or getting laid off) or the need for a higher salary that often only comes from switching jobs, its not always possible to build tenure.
Frankly its a big issue in the industry at large. I hate interviewing etc. but I'm not going to get paid 20% less because of it. I had to ride the job hopping treadmill for awhile, and I'd like to get off it just as much as you'd like to see people have more tenure.
When I first started out though, I worked for 5 years at the same place until it was very clear I was going to cap out on being able to advance and merit increases of 3-5% a year aren't going to cut it.
I got way more by switching jobs (after 4 years) than I was ever going to get by staying.
It is possible to avoid these traps, but then there are a lot of traps that we collectively have the wisdom to avoid but individually do not.
I started taking things apart and putting them back together at a very young age. When I was a young man I was in a big hurry to get somewhere, and so I could walk into a brownfield project and slowly reconstruct and deconstruct how we got here, would I have made the same decisions with the same information, and how do I feel about this news. Not only was I not falling into the "1 year of experience 10 times" dilema, I got more like 3-4 years of experience 4 times in 10 years, by playing historian.
My first almost-mentor left me with a great parting gift at the start of the dot-com era. He essentially convinced me of the existence of the hype cycle (in '95!), that we were at the beginning of one/several, he had seen previous ones play out, and they would play out again. Not cycles for new things, mind you, but cycles trying to bring back old things that had been forgotten. Like fashion. If anything it made me more likely to want to excavate the site.
Going into the trap knowing it's a trap doesn't necessarily save you, but it does improve your odds. Of course it also makes you a curmudgeon at a tender age.
It can get hard to separate ideas from execution, to 'forget' about the pain of a horrible technical move that still managed to be a commercially success, or to forget the pain of a badly-timed big decision that exploded in your face.
There's only so much interesting code you can add to interesting code, before every single conversation becomes a proxy discussion of The Mythical Man Month - we can't teach anybody new how to use this code in anything like a reasonable time frame. The best we can do is create new experts as fast as the old ones disappear.
Really Interesting Code is more at home surrounded by Really Fucking Boring code. You get so blinded by the implementation details of the Interesting Code that you cannot see the forest for the trees. That is the secret wisdom of Relentless Refactoring. The more I rearrange the pieces the more things I can see that can be made out of them, some of which are better, and a few of which are much better or brilliant.
Last week I implemented something in 2 days that we wanted years ago and didn't do because it would have taken more than a month of 1.5 people working on it to fix. But after a conversation in which I enumerated all the little islands of sanity I had created between Here and There, it took a couple dozen lines of code to fix a very old and very sore problem.
Going from hundreds of lambdas to a monolith is overreacting to one extreme by going the other one. There's a whole spectrum of possible ways to split a project in useful ways, which simplify development and maintenance.
Once you have the flow for deploying always running hot application(s) with autoscaling the benefits of lambda are basically gone.
Low volume scale to zero is just another route. No 15 minute limit, no having to marshal data through other AWS services because that's all lambda talks to, no more eventbridge for cron, no more payload size limits and having to use S3 as buffer, no more network requests between different parts of the same logical app, code deploys are atomic you're either at v1.x or v1.x+1 but never some in-between state.
I really do like Lambda but once you're at the spend where it's the same as some dedicated always-on compute the value drops off.
I've been working on a blog post to show this off for the last couple of weeks:
https://restate.dev/blog/suspendable-functions-make-lambda-t...
Put another way, you can monolith and minimize spend on AWS; it's not either or.
I'm using asp.net these days and even a chunky app published ready to run with optimized ef models starts relatively quickly.
Is that a fair comparison?
A pattern here is to not split the monolith and use the same container for your main app and hot operations. The hot operations just need some different configuration eg. container args or env vars