Well spotted but I don’t think it’s bad trade off. A beefy postgres instance (with standby configured), couple of worker nodes running dbos / river directly as a library backed by same db . This system can go surprisingly far.
I’ve seen and used airflow , spark , temporal for many systems. I’d def pick this simpler choice for 95% workloads these days.
How does it compare to a full-fledged durable execution platform like DBOS[0], which follows the same philosophy? Looks like River does have workflows, but it's locked behind Pro [1].
Yes and yes. DBOS self hosted , pointed to same application DB. Systems like these (durable execution) fit well with agents or LLM driven on demand workflows. You might be attaching multiple flaky local or remote api call tools. Many complex LLM workflows that you don’t want to one shot but break down into chain of prompt can be added to river/dbos + LLM , providing much needed retries and bit of concurrency control.
Airflow perhaps fits better with scheduled recurring ETL workflows.
I was asking around a while ago about better single-node solutions to "durable workflows" than Airflow before I was familiar with the terminology. Admittedly, Go + durable workflows in the cloud feels like the perfect marriage. Seeing the gated features makes me hesitant, unfortunately, but definitely considering this!
I’ve used asynq [1] at my past job. And although development is not as active anymore, they have a new contributor and we never encountered any bugs. It has been really solid and we relied on it heavily.
9 comments
[ 4.6 ms ] story [ 31.7 ms ] threadI’ve seen and used airflow , spark , temporal for many systems. I’d def pick this simpler choice for 95% workloads these days.
[0] https://dbos.dev
[1] https://riverqueue.com/docs/pro/workflows
Do you self-host? Do you use the same DB as the application DB? Do you use this for API background jobs, batch jobs, cron jobs?
As someone who uses Airflow a lot but not for background jobs, I am interested in the pros and cons of various approaches.
Airflow perhaps fits better with scheduled recurring ETL workflows.
My only hesitation is that many features are locked behind Pro.
I normally only settle for completely open source tools that I can self host and try to avoid these “partially” open tools.
For now, going to try to migrate my current stack to RabbitMQ. Seems like a very mature and completely open source tool.
River does look cool but I’m not a fan of vendor lock in. Will always avoid when possible
[1] https://github.com/hibiken/asynq