i would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model
I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Yeah it's exactly what I've seen as well. Ordewell is close to your high end case: a frontier model builds the plan once, then it's fixed, the executing model can't reinterpret it.
Won't get you to 4B, but should help a small model that only has to execute, not plan and execute at once.
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
I like your analogy. The main problem though is context and keeping it clean as much as possible as long a parallelization. This is what drove to build this tool: having control of everything that the LLMs will do, controlling all with one main planner that orchestrates the rest. This way we can have cheaper LLMs with a short context window used (less intelligence degradation) while still obtaining the same objective.
And again, you can have a clear picture of everything structured as tasks.
until we can get to rely on huge swarms of agents (tasks) being directed on the planner alone I don't see how we can get a better framework.
I've been working on a pi extension to do basically this; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
The gist is that a SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone.
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
I just can’t get excited about any of these meta-frameworks.
Doesn’t everyone get by now that any advance just gets rolled in to Claude and Codex a few months later, then the downstream competitors a few months after that?
indeed, but I also value the freedom of using whichever provider I want and not get locked into Claude Code alone for instance. I want to use both Opus, DeepSeek and GPT 6 at the same time, not just the 2/3 models Anthropic offers.
What I am trying to build is something that gives you absolute control, agnostic of the subscription/API you will use.
How does this compare with Matt Pocock's /grill-me -> /to-spec -> /to-tickets ? I see that Ordewell recommends model and effort, but is the workflow overall the same?
Yeah absolutelly. I just wanted that exact framework in a simple controllable UI, where I could also plug in any of my subscriptions/API keys such as Claude Code or Codex at the same time.
With Ordewell you get all layed out as tasks that are easily manageble (Matt's specs) in a UI.
24 comments
[ 0.26 ms ] story [ 22.8 ms ] threadI’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
The gist is that a SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone.
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
- https://github.com/Dicklesworthstone/beads_rust
https://github.com/awslabs/aidlc-workflows
Doesn’t everyone get by now that any advance just gets rolled in to Claude and Codex a few months later, then the downstream competitors a few months after that?