Lately I'm trying a variation on this: Have the main agent make a phased implementation plan, then for each phase, have it start an implementation subagent with a focused prompt, then review that agent's work in the main session. The theory being that the main session still contains all the research, but it can review just the diff rather than have the entire implementation session in context as well.
The post doesn't include a review in the cost comparison, but I find that immediately doing a review catches lots of mistakes.
Just specify that the more efficient model should only be used if the cost of doing the task would be less than using the higher horsepower model. Fable has no problem routing based on that instruction.
From what I can tell, I have a similar workflow. I get Fable 5 / Sol on High to talk to me about requirements until it's ready to design. I then have it break the design into tickets. I use kata, an agent-oriented issue tracker. If it ever starts to get bloated, I'll just make my own as it doesn't need too many features. Anyway, I tell it to include sufficient context in each such ticket to be picked up by a new implementation agent. Explicit user stories, Cucumber style acceptance criteria, etc.
When it's done, I switch to a smaller model and tell it to start a /goal of calling `kata ready` to get tickets ready to be worked. Work one ticket on each goal iteration, committing changes when done. Stop when all the tickets are either closed out or are blocked on actions from me.
It works fantastically. I can get entire (relatively straightforward) iOS apps done in under my $20/month five hour session window.
I get even better results if I do the QA session with the frontier model with two output artifacts: an implementation spec and a design prompt for Claude Design. I push the design prompt through Claude Design, tweak the results, and get a design spec. When I have the frontier model do the planning, I have it read the implementation spec from before, along with the design spec's overview file.
When I do it this way, I've done A/B tests between Fable 5 and Sol, and the apps they wrote were basically identical. It's so much cheaper than the "let loose the subagent fleet!" form of context management.
I haven't even added any kind of frontier model validation cycle into this loop yet. Everyone keeps talking about a subagent flow in which the big boy reviews the work of the drones, but I've found that if it encodes its acceptance criteria well enough, they do a satisfactory job of it themselves.
Think I unironically spent more time on that than writing the dang thing.
Glad you liked it though! Not sure if there's a name, was the best thing I could come up for visualizing turns.
The reason I use plan then implement is because I can adjust the plan, whereas if I get it to implement straight away, it might (and often does) make the wrong decisions that will be harder to adjust, or I'd have to adjust it after the fact.
I just did this five minutes ago. Started a task with Kimi K3, noticed cost going up, switched to Minimax M3 continuing from the same context. Really easy to do with Crush/OpenCode.
It works quite well most of the time. For very deeply technical tasks, where exploration involves multiple agents (and blowing up context limits a few times), I'll have a plan written down to disk that will be much longer than 2k tokens.
Wow. Someone using Crush in the wild. What do you think of it? I'm having trouble switching to it because the rest work fine, but would love to actually adopt it.
It has the most user-friendly UI of them all, a really usable scroll view. Automatically picks up LSPs. Has a well maintained provider registry. Looks nice and runs fast.
The main downside is that subagents cannot have edit tools, which kills any agent orchestration possibilities. I use it for most one-shot / single task prompts using open models.
this article is full of missing text like this:
" of Opus at of the cost, the speed, points over oneshot Flash."
is it just me or is that page using some brand new js feature or chromium-only hack?
> The mistake is upstream of the architecture diagram. People price agents the way they price people: senior time is expensive, so minimize senior involvement.
> But the expensive part of an agent's day is not the fixing, building, or even the thinking. Opus fixing things does not cost money. Opus reading things costs money.
Heh, another rediscovery that agents and humans are alike. By the time I've researched a Jira ticket and made it unambiguous enough to outsource, I might as well have written the code myself instead.
This is really smart, like the author said, old idea but cleverly applied.
In case anyone wants a summary: don’t one shot, don’t use plan mode and hand off the plan to cheap executors, ask the frontier model to explore, create a todo list, and then start when it feels confident; stop it after first code edit, then prefill the context to cheap executor to continue.
How is that different than just having the frontier model write a build plan and store it, then have a cheaper model execute? That’s pretty normal practice.
The build plan should be better than the context that generated it since it strips out wrong turns and other noise. I think?
So its basically (a plan + a bunch of file reads + a first edit) injected into the context of a cheap model so the cheap model does not feel like it has to re-read the files and just continues executing and editing?
> Below: the share of runs that went poking around the web for it. Filthy cheaters!
I can understand that in their benchmark setting they wouldn't want the model to find an existing solution. But in my day to day work, wouldn't I want the model to search online for the best solution? Am I not shooting myself in the foot by preventing it?
In ML, you want to test general capability of a model (generalizability), because you want it to perform well on unseen tasks. In that benchmark, the literal reference is leaking through web search, the agent can see the matching real codebase online and the commits so that's test set leakage. I know no programmer that was ever paid to rewind an existing codebase to a previous commit and implement a feature/fix a bug that exists in the next commits.
It's hard to argue with the numbers, but starting with a (mostly true!) “research is the most expensive part” premise, this strikes me as an odd direction to go to optimize costs:
1. As others pointed out, we feed all the same research turns to a smaller model, so we pay the uncached price for all of them.
2. During research, the model typically reads more code than is relevant, to figure out what is relevant and what is not. If that's the expensive part, we keep paying for those turns with the most expensive model?
3. There is no quality comparison of the resulting code. Same plan != same code, and AI tokens during the initial implementation phase isn't the only cost attributable to the task.
I do the opposite:
1. Outsource research to a subagent, or several parallel subagents. Let them output the relevant code paths only. Using gpt-5.6-terra-high on Codex and sonnet on Claude. Merge results into a single per-task research file. This saves tokens and context window of the bigger model, and avoids re-researching after compactions and in subagents.
2. Use the smartest agent (Sol xhigh ultra / Fable max) for both planning and execution. Tell it to use the research file where possible.
3. Switch to dumber agents for verbose substeps, e.g. gpt-5.6-luna-medium / sonnet is enough to drive browser use.
Sadly, got no numbers to back this approach.
PS: Of course, different task complexities and codebase complexities demand different approaches. The most expensive part is actually the code review step, which they don't mention/have at all. We should come up with some sort of complexity grading, so that discussions like this can be contextualized.
I'd love nothing more than measure 3), but quality is really hard to assess IMO as even with the frontier models I find myself disagreeing on what is good and maintainable code; so that disqualifies LLM-as-judge, which even at it's best is somewhat RNG and costly.
+ I don't see this replacing anyones workflow to make it clear. This is mostly for fully autonomous agents that are given a task & are asked to execute. For instance, subagents are a perfect example where you can get the aforementioned savings for free, even if you don't run a "software factory" per-se, as you generally do not interact with them.
I mean, this is real. Your KV cache lives for an hour since the last token (on anthropic pro/Max at least, today at least, this has degraded in the past). So once you pay, say, 50k input and 50k output Opus tokens, you don't pay for cache reads of the 100k context while it builds. Switch to another model, you'll start with the cost of 100k input tokens on the smaller model to get that context loaded and its unique KV cache set.
The post may have some real insight here, where this 100k working context is actually better than trying to summarize it's findings into a plan. It's also right that the handoff is a perfect time to edit the context (removing planning instructions). But it doesn't mention it's a trade-off: the plan is smaller, so it's a cheaper "on-boarding" of the next model. Send quite plausible that this is worth it for 1-off tasks. If it's right, this is basically "plans are useless, planning is everything" for LLMs.
My problem is, I think the plans are useful. I want to review and edit them. I want them to give context for the upcoming code review (even if humans aren't reviewing). LLMs are notoriously bad at explaining why they're doing something in the moment. Humans are notoriously bad at accepting there's no reason why. I think the humans have it right here, and to bridge this gap want my PRs, my docs, and my comments teeming with reasons why.
Wait this is new? Large context models figure out the architecture of the request. Miss sized models plan out each requirement, and smaller models implement a tightly detailed task? Isn't this the standard approach?
I‘m really wondering about plan and then agent mode switch in Copilot even with the same model. The switch swaps the prompt so all the learnings during planning which are in the cache will get invalidated.
46 comments
[ 3.3 ms ] story [ 69.6 ms ] threadThere are a lot of interesting ideas in it, mostly none that I have applied to my own workflows. Curious if anyone has used it?
Sounds like a consulting slaughterhouse. Picture Java Enterprise Solutions. As we all know, the pinnacle of software engineering.
The post doesn't include a review in the cost comparison, but I find that immediately doing a review catches lots of mistakes.
When it's done, I switch to a smaller model and tell it to start a /goal of calling `kata ready` to get tickets ready to be worked. Work one ticket on each goal iteration, committing changes when done. Stop when all the tickets are either closed out or are blocked on actions from me.
It works fantastically. I can get entire (relatively straightforward) iOS apps done in under my $20/month five hour session window.
I get even better results if I do the QA session with the frontier model with two output artifacts: an implementation spec and a design prompt for Claude Design. I push the design prompt through Claude Design, tweak the results, and get a design spec. When I have the frontier model do the planning, I have it read the implementation spec from before, along with the design spec's overview file.
When I do it this way, I've done A/B tests between Fable 5 and Sol, and the apps they wrote were basically identical. It's so much cheaper than the "let loose the subagent fleet!" form of context management.
I haven't even added any kind of frontier model validation cycle into this loop yet. Everyone keeps talking about a subagent flow in which the big boy reviews the work of the drones, but I've found that if it encodes its acceptance criteria well enough, they do a satisfactory job of it themselves.
That said, it takes me a while to reads plans and often I'll take a break, by when the cache has expired. At that point it may be better to start over
Therefore token sellers have incentive to produce verbose, unreadable code
Try plan with opus 4.8 and then implement with Deepseek v4 flash - its 35x cheaper for reads, 90x cheaper writes, and 18x cheaper cache reads.
Or plan with Deepseek Pro, or even Flash itself. I've been impressed with both.
It works quite well most of the time. For very deeply technical tasks, where exploration involves multiple agents (and blowing up context limits a few times), I'll have a plan written down to disk that will be much longer than 2k tokens.
The main downside is that subagents cannot have edit tools, which kills any agent orchestration possibilities. I use it for most one-shot / single task prompts using open models.
> But the expensive part of an agent's day is not the fixing, building, or even the thinking. Opus fixing things does not cost money. Opus reading things costs money.
Heh, another rediscovery that agents and humans are alike. By the time I've researched a Jira ticket and made it unambiguous enough to outsource, I might as well have written the code myself instead.
In case anyone wants a summary: don’t one shot, don’t use plan mode and hand off the plan to cheap executors, ask the frontier model to explore, create a todo list, and then start when it feels confident; stop it after first code edit, then prefill the context to cheap executor to continue.
The build plan should be better than the context that generated it since it strips out wrong turns and other noise. I think?
I can understand that in their benchmark setting they wouldn't want the model to find an existing solution. But in my day to day work, wouldn't I want the model to search online for the best solution? Am I not shooting myself in the foot by preventing it?
1. As others pointed out, we feed all the same research turns to a smaller model, so we pay the uncached price for all of them.
2. During research, the model typically reads more code than is relevant, to figure out what is relevant and what is not. If that's the expensive part, we keep paying for those turns with the most expensive model?
3. There is no quality comparison of the resulting code. Same plan != same code, and AI tokens during the initial implementation phase isn't the only cost attributable to the task.
I do the opposite:
1. Outsource research to a subagent, or several parallel subagents. Let them output the relevant code paths only. Using gpt-5.6-terra-high on Codex and sonnet on Claude. Merge results into a single per-task research file. This saves tokens and context window of the bigger model, and avoids re-researching after compactions and in subagents.
2. Use the smartest agent (Sol xhigh ultra / Fable max) for both planning and execution. Tell it to use the research file where possible.
3. Switch to dumber agents for verbose substeps, e.g. gpt-5.6-luna-medium / sonnet is enough to drive browser use.
Sadly, got no numbers to back this approach.
PS: Of course, different task complexities and codebase complexities demand different approaches. The most expensive part is actually the code review step, which they don't mention/have at all. We should come up with some sort of complexity grading, so that discussions like this can be contextualized.
+ I don't see this replacing anyones workflow to make it clear. This is mostly for fully autonomous agents that are given a task & are asked to execute. For instance, subagents are a perfect example where you can get the aforementioned savings for free, even if you don't run a "software factory" per-se, as you generally do not interact with them.
--- Switch model? Your next response will be slower and use more tokens
This conversation is cached for the current model. Switching to Opus 4.8 (1M context) means the full history gets re-read on your next message. ---
Reminds me a bit of airline booking sites ("We noticed you didn't add insurance").
The post may have some real insight here, where this 100k working context is actually better than trying to summarize it's findings into a plan. It's also right that the handoff is a perfect time to edit the context (removing planning instructions). But it doesn't mention it's a trade-off: the plan is smaller, so it's a cheaper "on-boarding" of the next model. Send quite plausible that this is worth it for 1-off tasks. If it's right, this is basically "plans are useless, planning is everything" for LLMs.
My problem is, I think the plans are useful. I want to review and edit them. I want them to give context for the upcoming code review (even if humans aren't reviewing). LLMs are notoriously bad at explaining why they're doing something in the moment. Humans are notoriously bad at accepting there's no reason why. I think the humans have it right here, and to bridge this gap want my PRs, my docs, and my comments teeming with reasons why.
(NB: ranted to long, reposting at top level)
For those not sufficiently cracked... what is omp? Is there a way I can have this command or workflow in Cursor / Claude Code?
I tried going to the homepage (https://stencil.so/). It didn't really help.