Launch HN: Defer (YC W23) – Zero-infrastructure background jobs for Node.js
Background jobs, while being used in all web applications (processing webhooks, interacting with 3rd party APIs, or powering core features), did not benefit from the developer experience improvements that arose in all other layers of the Node.js API stack: quick and reliable databases with Supabase or easy Serverless deployment with Vercel.
Today, even for simple use cases, working with background jobs in Node.js necessarily requires some infrastructure knowledge—either by deploying and scaling an open source solution (ex: BullMQ) or using an IaaS such as AWS SQS with Lambdas, which comes with complexity and limited features (no support for dead letter queues, dynamic concurrency, or throttling).
At a large scale, you will need to solve how to handle rolling restarts, how to auto-scale your workers, how to safely deploy without interrupting long-running jobs, how to safely encrypt jobs’ data, and how to version them. Once deployed, your background job’s code lives in a separate part of your codebase, with its own mental model (queues and workers). Finally, most solutions provide technical dashboards which are not always helpful in debugging production issues, so you end up having to build custom dashboards.
Most companies we talked to try to handle those different aspects, building custom similar solutions and using developers’ time that could have been used on user-facing features.
Bryan and I are technical founders with 10+ years of experience working at start-ups of all stages (e.g. Algolia, home of HN Search!), from tech lead to CTO roles. Like many developers, we got asked many times to work on background job stacks and invest time into tailoring and scaling them for product needs.
I even dedicated most of my time at Algolia to building a custom background jobs pipeline to power the Algolia Shopify integration: ingesting partial webhooks from Shopify, enriching them given customers configuration, in FIFO order per shop, with the Shopify rate limited API, for thousands of shops and the equivalents of 3 millions of jobs per day. Given the complex and unique product requirements of the Algolia Shopify Ingestion Pipeline, the only solution (at the time and context) was to build a custom background jobs stack combining Redis and Kubernetes.
When consulting with some startups, we witnessed some developers choosing to keep some slow API routes calling 3rd party APIs synchronously instead of investing time in setting up background jobs. When looking back to the recent increase of productive zero infrastructure solutions in the Node.js ecosystem, we were surprised that the experience with background jobs remained unchanged. We decided to build Defer, so working with background jobs, CRONs, and workflows would match the current standard of Node.js developer experience.
Inspired by Next.js, Remix, and Netlify design, background jobs in Defer become background functions that live in your application’s code, with direct access to all configuration options: retry, concurrency, and more (https://docs.defer.run/features/retries-concurrency/) , and no specific mental model to learn. Your background functions get continuously deployed from GitHub with support for branch-based environments, allowing you to test new background jobs in no time, before safely moving to production.
Defer works for all kinds of Node.js projects, not only serverless ones. It does not require you to learn any new architectures or adapt your system design—you just ...
113 comments
[ 2.6 ms ] story [ 235 ms ] threadFirst, they allow a maximum execution time comparable to “pure server” environments, instead of 15min.
Then, the code-first approach allows configuring the execution parameters (concurrency, retries, and more) from the function’s code and enables to write workflows by applying well-known code patterns such as recursion or map-reduce (calling child background functions is a workflow).
Finally, you can write a background function that requires any kind of dependencies, whether they be internal, external, or even native.
Does this look nice in the observability layer?
Right now our Executions list is not ideal for such pattern but we will soon release filtering based on arguments which will help to get all the executions linked to a specific sequence.
Can you please elaborate more on the modern development standards bit? I've always been intrigued by changes on development forefront from early 90's time.
Node.js, with the flexibility of the JavaScript language, allowed the rise of great abstraction and other domain-oriented API designs, a bit like Ruby on Rails did with Ruby.
The arrival of React.js Server Components pattern enabled the isomorphic pattern (first applied to mobile and front-end apps) to reach the server side of things with patterns such as Server side rendering, popularized by frameworks like Next.js or Remix.
Those new patterns and abstractions allow Node.js developers to move faster while building more complex applications to match users’ requirements: real-time, performant apps, richly integrated with third-party products.
Beyond code, those new coding habits came with new products such as Vercel or Supabase that help them to get the infrastructure done in no time, without any DevOps knowledge (good article on this topic: https://vercel.com/blog/framework-defined-infrastructure).
“modern development standards”, applied to Node.js, do not only apply to coding experience and productivity (ex: the rise of monorepos, TypeScript, SSR) but also to enabling developers to configure their infrastructure from the code.
[update: typo]
I don't mean to be unkind it's a legitimate question not sarcasm.
It just seems like a subset of Cloud Functions or a subset of Lambda - without being connected to the rest or any of our existing workflow, monitoring, secrets, etc.
a lot of times these specialized services can really nail a more narrowly scoped feature set to the point where it's worth having stuff running outside of your primary cloud
it always depends, for me personally that bar is very high because like you said, now you have to copy paste secrets into yet another service among other things
some services are so good, like planetscale, especially compared to the cloud option that I'm willing to do that
As @thdxr mentioned, our goal is also to provide top-notch queueing/scheduling features that are complicated to achieve with Lambda/SQS/Cloud Functions, such as dead letter queue support, throttling, or a product-oriented dashboard. In the same idea, we plan to provide better integration, for example, as you mentioned on secrets management by integrating with Doppler, AWS Secrets Manager, and more - the same goes with linked deployment pipelines.
Also I don't think this is as much about the "compute" of the background job as much as the nodejs app saying "take care of this for me please, got this? Don't bother me again until I ask for the result".
Endpoint handlers are flaky. They aren't guaranteed to run to completion even through no fault of their own. They should ideally have short and predictable run times, otherwise scaling is much more painful. That's why people separate out certain stuff into those background jobs. And getting that just right, scale it and so on can be quite a bit of work, but yes, something like Cloud Functions or Lambda would serve very similar functions. If you set up all the CI/CD pipelines to make that work.
I've been "abusing" GitLab CI pipelines a lot for running periodic background jobs that I'd like to have separated from the "regular" backend and without worrying much about deployment.
It supports Cron Syntax for scheduling, manual re-runs, provides secret management and download of artifacts, alerts me on failure and allows me to easily use the docker images in our registry.
Sure, I'm not storing anything in a database or calling the pipeline via an endpoint. But if that's needed, it should probably be "part of the regular backend".
I'm guessing this is a "just make sure everything you do is idempotent" approach? Is there any kind of state between attempts?
Also how would you have a task with a unique id / parameter so that when you call it it will only be scheduled once and returns the result of the existing scheduled job?
Another question, how would you have custom retry logic in your job without blocking your concurrency? Or if using the platforms retry mechanism, how can you run some logic when the job fails? Can the job know what attempt it is on? Is there context of any sort? The only solution I can see from the docs would be to execute jobs recursively but that seems like it would be ugly and mess up the observability? I'm looking for a feature like Temporal's sleeping that actually does a delayed reexecution of the job (I think).
Also I can't get the discord link to work, it seems to link to a discussion rather than being an invite link (I'll give it another try on desktop)
Right now, you will need to ensure that your background functions are idempotent but we plan to introduce an API for “retried execution” so you can clean up. Also, by default retry is an opt-in configuration option to avoid any unwanted side effects, the same goes with concurrency.
I'm curious who your target customer for this? As someone who likes to work on side projects, this seems ideal for me, but I would imagine people will graduate to more mature solutions once they have decent volume. The concurrency limit is something that seems especially concerning in that regard.
We want to enable developers working on a side project to build apps faster and new startups to have a faster time to market by avoiding spending time on infrastructure and building custom layers of API and dashboard.
We also want to help bigger companies to take back control of their custom background jobs stack and to be able to ship new features or sub-products quickly. Again for growing companies, we provide Custom Pricing which comes with degressive pricing with custom concurrency to support, for example, high throughput needs.
We use AWS KMS to perform data-at-rest encryption on all data we store. We perform another encryption pass for sensitive data (e.g., Github Token, Secrets, etc.) before storing the data with a symmetric PGP key.
> Is this CronAAS suitable for sensitive workloads?
It should be. Could you elaborate?
I have a project that I've been dreading on how I would tackle its background processing. It is a long-running process that should start and scale based on a user starting his work. I'm AWS certified and still wasn't thrilled about setting this up. Defer would solve this for me.
I can see myself suggesting this solution to clients who want to run complex jobs without the hassle from cloud infra.
I mean, basically anyone paying for this service would have to scramble to find and implement another solution once Defer shuts down.
If you're going to `await` for the contacts import to finish anyway, what's the advantage of separating the import logic from your main API? It's blocked, so might as well be part of the same service, no?
I could see maybe if the API returned right away with a pointer the user can later poll for task progress, but it doesn't seem like this is the case?
Side note: I like this type of web design, is it an in-house job or did you hire someone external?
You shouldn't need to first do a check to see if the task already exists using a different api, and then choose whether or not to run a new task.
At least this is my preference.. a-la Durable Objects.
If you don't specify a custom unique id when calling the task it would then be treated as a task that can (and would make sense to be) run multiple times (i.e. not idempotent).
``` const somethingThatMayHaveBeenCalledBefore = idempotent(myFunc, txnref); const result = await somethingThatMayHaveBeenCalledBefore(params); ```
> If you're going to `await` for the contacts import to finish anyway, what's the advantage of separating the import logic from your main API? It's blocked, so might as well be part of the same service, no?
> I could see maybe if the API returned right away with a pointer the user can later poll for task progress, but it doesn't seem like this is the case?
As you suggest, the API returns right away with a pointer the user can later poll to get the function result. Also, using `await` ensures the function is enqueued on our system.
> Side note: I like this type of web design, is it an in-house job or did you hire someone external?
Happy to hear this! We are working with a friend who is a professional designer.
If your friend is open for business, maybe give them a shoutout ;)
It is true that we don't elaborate on this point on our landing page. We will take that into consideration, thanks.
> If your friend is open for business, maybe give them a shoutout ;)
Again, glad you like his work! He's not available at the moment, but we will stay in touch :)
I used to use Celery a lot and find Hangfire lacking. To get the same value out of Defer, you'd need to "listen" to those handles.
And listening may actually mean writing another local background job...
Just a small note that on the landing page, the snippet under Define has the file path `defer/helloWorld`, while the snippet in Enqueue is importing `importContacts` from `defer/importContacts`. As I was reading it, I thought it seemed as if the Enqueue snippet was supposed to be importing from the snippet written under Define. Just thought I'd mention it in case that's what it's supposed to show.
It's very clearly infrastructure. Putting "zero infrastructure" in the title and using the same API as Lambda invoke etc. doesn't make it true.
I also get double the risk of downtime, security as it's a third party running on top of AWS vs. just running on AWS.
The API looks nice - but no mention of typescript at all in this post or the website, so presumably type-safety isn't the thing.
You are right. We provide an infrastructure service. We mean by "zero infrastructure" that you don't have to implement and/or manage your own. Our service could run on a platform other than AWS, though, as we are not relying on AWS-only specific services (e.g., lambda, SQS, etc.). Of course, like any other cloud or on-premises service, we could have downtime.
> The API looks nice - but no mention of typescript at all in this post or the website, so presumably type-safety isn't the thing.
Glad to hear this. Although we don't mention it, our client is written in Typescript. If you want to know more, you can check out the code: https://github.com/defer-run/defer.client.
To achieve type safety normally involves a shared interfaces folder and has to be specifically implemented
This is imporant as I'm not sure I'd want to trust yet another service provider with my customers' data, especially one that is VC-backed and therefore inherently less trustworthy in the long-term than others.
For our users concerned about data locality, we recommend pushing the minimal data as arguments (ex: ids or external ids) and fetching the data during the execution on our side, if needed, through a dedicated SSH tunneling setup. Once an execution is done, its associated isolated container - gets a dedicated VPC and disk - gets destroyed permanently. We will also provide on-premises solutions for Enterprises.
In local environment (dev), background functions run completely synchronously and locally, no call is made to Defer.
I use postgraphile graphile-worker https://github.com/graphile/worker for this.
For example, every month we roll over credits. For each user, when they signed up, 30 days from that, check. If they are available for roll over, reset and email. Then we have drip campaigns for alerts like running low on credit.
Also, if you upgraded your account, then pause payment, it uses a worker to schedule the date they are paid up to run the SQL to downgrade. With a simple API called 'addJob' that looks for a JavaScript file in a folder called task.
Something something Dropbox, something something rsync.
Charly, Bryan consider yourself flattered.
I could see this selling fast to corporate
It's on a separate network so they may go down and cause outage in your own app, go out of business, etc. (Yes, your IaaS service provider can go down as well, but that's a much larger operation and something that's very-very likely to have a much better uptime than what you could do on-site. And whatever the case is with that, it's still on top of that.)
Abstractions aren't free. (BTW, I don't see it as an abstraction, it's simply outsourcing. You'd use some kind of API anyway even if you hosted the job queue yourself.)
Depends what you're building, but likely a good trade if it's an MVP.
At $30/mo you're already ahead if it saves you 1 hour of dev time.
Most of the time, I don't want control over my infrastructure (boring, low value); I'd rather focus on building features (fun, high value).
You need to put up significant resources in order to beat a dedicated team running this. Many companies can do that. Many can't. Even Twitter increasingly runs into problems with site reliability...
It’s just scary to hand off such an important part of your system to a third party.
Also, you don't know the team size behind this service, as I mentioned in my original comment.
If these start to matter, even picking a service from a list of possible providers takes time. Learning their tools, their APIs takes time, etc. OTOH if you have a project template, a setup you know, doing an MVP with that can be really cheap/quick. If not, I can see how this makes sense. But I'm not sure you can have a sustainable business just by serving people building MVPs.
Defer: Down and to the right
Not a good look. Calling it now.
Sequoia's has a diagonal line through it. Defer's does not.
Thanks for pointing that out g_delgado14 but I still think it could work as their logo, because with a simple logo it's hard to not make it look a little bit like some other logo.
I can't think of a top-down approach working for technical folks, especially cold outreach.
We got our first customers from both channels: network (sales) and inbound (twitter discussions, etc). I agree that top-down is not working well for a newcomers but getting better at a later stage.
1. Data locality
2. Privacy
3. SLAs and Uptimes
While SaaS solutions like these can get us MVP/PoCs fast but a home grown solution is more preferable when SLOs are tight and security is a huge concern.
[edit: formatting]
We address privacy by encrypting all the data on our side (doing a second pass with a symmetric PGP key for tokens such as GH tokens, and environment variables) and advise companies that want to keep their data on their infra to push as minimum data in arguments while leveraging a dedicated SSH tunneling setup between our infra and theirs.
When it comes to the SLA/up-time of home grown, my POV would be that, again, achieving good results on those often requires SRE engineers, which is an investment.
I think there's a huge opportunity for whoever that can create Spark or Flink like framework that works in Node.js or at least has Node.js API.
It's a huge challenge to create a tool like that, but I believe there's an appetite for such tool. JS community is huge. JS is a flexible language that can conceptualize distributed computing well. JS community and developer ecosystem generally has superior emphasis on developer experience, outreach and education than the communities around existing tools.
Given how economically valuable companies that exist around big data distributed computing are, I think a company that can create an open source Node.js distributed computing tool will be like the next MongoDB, Elastic and Databricks.
We aim to provide a similar feature set as Sidekiq (throttling, unique jobs) but with a complete hosted solution.
While Defer and Temporal can both be used for writing background jobs, workflows, and CRONs, there are some core design differences.
Temporal has been created as the Kubernetes of highly distributed systems, enabling developers to write code that runs on multiple regions without worrying about possible termination of the program and interruption of workflows spanning across multiple steps.
While Temporal can be used for background jobs, workflows, and CRONs, its main goal is to ensure that highly distributed tasks will reliably be executed. That's the main reason why Temporal API is so verbose, with many concepts to deal with.
Defer, on the other hand, provides comparable reliability while focusing on the developer experience.
You can write workflows, CRONs, and jobs that run for hours without worrying about them being terminated.
All this, with a simple API that enables you to write some workflows (background functions calling other background functions) in plain TypeScript, with no mental model to fit in.
https://docs.defer.run/features/delay/
Does the "defer" directory make the exports of those modules special? Because the defer function isn't called until the endpoint is hit, there's no way to really know (without static analysis, which is a can of worms) which functions will actually get deferred.
Moreover, how does the defer function know, at runtime, which function on the Defer platform needs to run? I could pass any arbitrary function to defer(), which surely can't be run externally. How does the defer platform know which function it can see (by importing my code) is the function my application has called defer() on?
Without knowing exactly the constraints of this, I have a lot of FUD.
When a background function gets a call from your application, the `defer()` wrapper intercepts this call and pushes an execution to the Defer API with the function’s name and serialized arguments.
I hope it makes thinks clearer, let me know!
[update: grammar]
OK, tell me more...
Is there a local runtime a-la miniflare?
Or do I have to deploy and then run my unit tests against your APIs?