there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here?
you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.
I think this is great. We should see more simple solution to this problem.
I recently started doing something very similar on Postgres [1] and I'm greatly enjoying using it. I think the total solution I ended up with is under 3000 lines of code for both the SQL and the TypeScript SDK combined, and it's much easier to use and to operate than many of the solutions on the market today.
I really enjoyed this post and love seeing more lightweight approaches! The deep dive on tradeoffs between different durable-execution approaches was great. For me, the most interesting part is that Persistasaurus (cool name btw) use of bytecode generation via ByteBuddy is a clever way to improve DX: it can transparently intercept step functions and capture execution state without requiring explicit API calls.
(Disclosure: I work on DBOS [1]) The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX.
One thing that needs to be emphasized with “durable execution” engines is they don’t actually get you out of having to handle errors, rollbacks, etc. Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? You failed while updating the inventory system - did the product get marked out or not? All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.
> One thing that needs to be emphasized with “durable execution” engines is they don’t actually get you out of having to handle errors, rollbacks, etc.
I think this is a gross misrepresentation of what durable executions are. DEs were never expected to magically eliminate the need to handle errors. What DEs do is provide an high-level abstraction of the same pattern that is recurrent on all workflow engines, and they provide a simpler way for developers to implement rollback and compensation steps when workflows fail.
If you are designing and implementing a transaction with a DE, you still need to design and implement a transaction. DEs simplify much of the logic, but you still need to design and implement a transaction. There is no silver bullet.
> Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? (...)
That's immaterial to the discussion on DEs. You, as a software engineer, still need to design and implement a transaction. DEs greatly simplify your job, but you still need to analyze failure modes and perform the necessary compensation steps.
> All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.
I completely disagree, but you do you. Some durable execution engines greatly simplify tracking state and implementing activities and rollback logic. Some cloud providers even provide services that allow you to implement long-running workflows with function-as-a-service components that provide out-of-the-box support for manual approvals. If you feel you are better off rolling your own support, good for you. Meanwhile, everyone around you is delivering the same value with much less work.
Sorry for the off-topic but I have been lately seeing a lot of hype around durable execution.
I still cannot figure out how this is any different than launching a workflow in something like Airflow. Is the novel thing here that it can be done using the same DB you already have running?
Reminds me of IBM TPF (Transaction Processing Facility) - the system that powered airline reservations for decades. TPF used per-transaction logging with restart/recovery semantics at massive scale. You could literally unplug the power mid-transaction, plug it back in, and resume exactly where you left off.
The embedded database approach here is interesting though - low latency, no network calls, perfect for single-agent workflows. TPF assumed massive concurrent load across distributed terminals. Different problems, similar durability patterns.
Serious question: How does "Durable Execution" differ from "Atomic Transaction"? At most, it seems that DE refers to more concrete details around implementing Atomic Transactions.
13 comments
[ 2.7 ms ] story [ 32.2 ms ] threadyou can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.
I recently started doing something very similar on Postgres [1] and I'm greatly enjoying using it. I think the total solution I ended up with is under 3000 lines of code for both the SQL and the TypeScript SDK combined, and it's much easier to use and to operate than many of the solutions on the market today.
[1]: https://github.com/earendil-works/absurd
(Disclosure: I work on DBOS [1]) The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX.
[1]: https://github.com/dbos-inc
I do still think there is sufficient amount of boilerplate to potentially justify some engine like this.
I think this is a gross misrepresentation of what durable executions are. DEs were never expected to magically eliminate the need to handle errors. What DEs do is provide an high-level abstraction of the same pattern that is recurrent on all workflow engines, and they provide a simpler way for developers to implement rollback and compensation steps when workflows fail.
If you are designing and implementing a transaction with a DE, you still need to design and implement a transaction. DEs simplify much of the logic, but you still need to design and implement a transaction. There is no silver bullet.
> Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? (...)
That's immaterial to the discussion on DEs. You, as a software engineer, still need to design and implement a transaction. DEs greatly simplify your job, but you still need to analyze failure modes and perform the necessary compensation steps.
> All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.
I completely disagree, but you do you. Some durable execution engines greatly simplify tracking state and implementing activities and rollback logic. Some cloud providers even provide services that allow you to implement long-running workflows with function-as-a-service components that provide out-of-the-box support for manual approvals. If you feel you are better off rolling your own support, good for you. Meanwhile, everyone around you is delivering the same value with much less work.
I still cannot figure out how this is any different than launching a workflow in something like Airflow. Is the novel thing here that it can be done using the same DB you already have running?
The embedded database approach here is interesting though - low latency, no network calls, perfect for single-agent workflows. TPF assumed massive concurrent load across distributed terminals. Different problems, similar durability patterns.
Does anyone really do this?