What a great confirmation of what I've tried to achieve by getting models to communicate with my local data in shorter communicates to save on token command bandwidth, more or less the same results! Thanks for the article!
LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed)
American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery.
So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.
Simply starting in the right part of the search space is the biggest predictor of success.
The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table.
5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better.
The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.
Been through this exact loop building an LLM product. The counterintuitive lesson for me: most "token saving" ideas are cache killers — anything that makes the prompt prefix dynamic (rotating retrieval, appending context per turn) can cost more than the tokens it saves, because you lose the cached-prefix discount. Freezing the retrieval selection early in a conversation and paying a slightly larger fixed prefix beat every clever adaptive scheme I measured.
I have a process generating trivia questions for my app and by passing in my existing dataset each time as a cached exclusion list, the output tokens it save more than pay for it.
You should consider testing our RemembrallMCP (note I'm the author). It substantially cuts token usage and you get better results in codebases.
https://github.com/roboticforce/remembrallmcp
It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.
I stopped using any of the built-in skills and skills agents and tried to keep things in the model only, and that seems to work better for token usage for me. It took me some time to tune my skills .md, but it rocks on now. I use them all in both Claude-code and Codex. I will say, GPT is better on token usage than Claude is, but for some of my code bases, Codex is not as good, but seems to be getting better with 5.6-sol.
I am not a proper developer and only use AI for faster research of topics so please forgive my ignorance. Could one not save a lot of money on tokens by using the 80/20 or 90/10 rule in that 90% of AI usage is on local models and save that last 10% or less for the frontier models where the local model did not meet the needs? Did they cover this and I misunderstood?
It is Polish so please translate. But tldr is: ask your agent at times to analyze last chats and update rules and skills to not repeat the same steps it repeats every time when you start a new thread. Simple as that.
The fun part of LLMs and internet research is that even if you have a URL and a quote from a primary source that primary source could be an LLM generated hallucination.
I get a lot of satisfaction using a significant chunk of my work-provided Claude plan researching and developing local LLM solutions for work.
Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform.
At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.
I can't remember who it was I spoke to recently, but it was some startup for doing code reviews. So I asked how they're handling larger diffs and especially when a code change is related to something requiring knowing a lot of context. Will it hallucinate things that make the review low-quality, etc. I was told "no it's been given guardrails to prevent hallucinations. it doesn't hallucinate." but I was thinking, that's not really how LLMs do or do not hallucinate, is it? (I'm not an AI person so I really don't know).
You can reduce hallucinations with appropriate grounding checks. You can’t really get rid of them though, so feedback loops with verification are essential.
I hear you! I read the article and thought the same thing because I've actually built a surprisingly similar pipeline myself, but it culminates with Human review. I've got a trivia app (Hedge Calibrated Trivia) whose whole content pipeline is built on a similar architecture of models evaluating models. I didn't want the same trivia that every other app has and I naively imagined that it would be easy to ask an LLM to give me questions. Turns out when you ask an LLM for trivia questions, the FIRST thing it gives you are the same questions every other app asks!
After iterating into many of the same problems OP describes I settled into a similar pipeline. I generate questions with Gemini 3.5 flash and review with Claude. The different vendor generation and validation eliminates training holes in a single model solution. This is True/False trivia so I ask for an equal mix of True/False questions and also ask for alternate phrasing of the question to the opposite variant. Each statement must provide an explanation and sources which back up the claim. I pass in my existing dataset as questions to avoid, but found the model then rewording the questions or category shopping the questions into related categories in an attempt to re-use them. That led to using Claude Sonnet to check for duplicate "themes" and similar topics. Claude also acts as a critic of the question and the explanation. Sometimes Gemini's "thoughts" leak into the explanation. The critic catches these leaks and also catches "weak" questions (basically shapes that I have identified via examples as lazy or projecting the answer in the question). Once the comparative models have stripped out anything obviously bad, my server curls the source URLs and uses Sonnet to verify that the fetched source supports the claim. Any URL that doesn't resolve throws up flags, and I found Gemini was particularly bad at inventing source URLs for sports trivia. Updated prompting directed it to prefer Wiki links over its fabricated deep links into league websites, which helped immensely. Still one of the biggest tells in review is that the URLs don't resolve or don't confirm the statement.
But review is the entire point here. I have to review them all. A question that lands in the app and is wrong reflects terribly. So while I have a trivia app on the App Store, the real complex App is my private admin app that lets me sit on the couch swiping through each and every question sorted by the model's confidence in its own quality and correctness (weakest first). No question makes it into my bank without me reading, checking all the indicators that came from the models and potentially even rewriting the question to make it better. It all happens in the admin app and exists entirely to prevent hallucinations from reaching real users. Early on I was maybe accepting 1 in 3 questions generated, but I'm probably closer to 2/3rds now. Of course not every dismissed question is wrong. The quality checks I've put in the pipeline are probably giving me 80% questions that are correct and sourced, but sometimes they just aren't what I'm looking for or don't meet my standard.
The best thing I ever did was make an app to review these questions though. Sitting on the couch, kick off a question generation, come back 20 minutes later with 50+ questions in the queue. Sort through them and pull out the 30 best when I have the time. That's the real anti-hallucinations pipeline.
Many of us were saying it a year ago but now with model restrictions (e.g. fable) and pricing changes it should be obvious to people that part of the economics is avoiding vendor lock-in with evals (so you can easily switch providers/models/optimize) and increasing control by investing in local models which could be good enough for your tasks, at whatever the price for your cloud compute is. Eventually consumer hardware will also be able to run good enough.
You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change.
The author touches on an issue which bothered me, which was the thought of many agents re-solving the same issues over and over. I see a few comments here too, mentioning fixing issues which may already have a solution elsewhere. That was why I created https://pushrealm.com which started as essentially a Stackoverflow clone via MCP.
It has now become a way for agents to converge on complete, shared answers for emerging issues. As a side effect, through posting open issues we can also identify gaps in model knowledge.
1. Use less subagents. The act of using subagent already needs dumping adequate amount of context for subagent to work effectively
2. Refactor when file grows large. Reasonably scoped files is easier to understand, for both human and agents, and consume less tokens because less guessing which section of the file to read
3. More capable models for planning, learning. Cheap models for execution. Same for human teams, senior member for higher level planning while junior member more on the act of translation of idea into code
4. I wasn't doing any fancy stuff that opus can't handle
5. Consult a second opinion for things you are not sure. (I have made a skill specifically for that)
6. Audit how your habit correlates with token consumption. I found out a skill a use frequently is using a lot of context just so a subagent can get all the context. (made a tool to audit claude specifically)
47 comments
[ 3.6 ms ] story [ 69.0 ms ] threadLLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed)
American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery.
So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.
The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table.
5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better.
The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.
For me personally, the only reason why i don't use it 24/7 is cost.
Token costs, energy cost, usage limit due to token/subscription cost.
https://neuronowa.pl/posts/porada-jak-wydajniej-kodowac-w-cu...
It is Polish so please translate. But tldr is: ask your agent at times to analyze last chats and update rules and skills to not repeat the same steps it repeats every time when you start a new thread. Simple as that.
Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform.
At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.
After iterating into many of the same problems OP describes I settled into a similar pipeline. I generate questions with Gemini 3.5 flash and review with Claude. The different vendor generation and validation eliminates training holes in a single model solution. This is True/False trivia so I ask for an equal mix of True/False questions and also ask for alternate phrasing of the question to the opposite variant. Each statement must provide an explanation and sources which back up the claim. I pass in my existing dataset as questions to avoid, but found the model then rewording the questions or category shopping the questions into related categories in an attempt to re-use them. That led to using Claude Sonnet to check for duplicate "themes" and similar topics. Claude also acts as a critic of the question and the explanation. Sometimes Gemini's "thoughts" leak into the explanation. The critic catches these leaks and also catches "weak" questions (basically shapes that I have identified via examples as lazy or projecting the answer in the question). Once the comparative models have stripped out anything obviously bad, my server curls the source URLs and uses Sonnet to verify that the fetched source supports the claim. Any URL that doesn't resolve throws up flags, and I found Gemini was particularly bad at inventing source URLs for sports trivia. Updated prompting directed it to prefer Wiki links over its fabricated deep links into league websites, which helped immensely. Still one of the biggest tells in review is that the URLs don't resolve or don't confirm the statement.
But review is the entire point here. I have to review them all. A question that lands in the app and is wrong reflects terribly. So while I have a trivia app on the App Store, the real complex App is my private admin app that lets me sit on the couch swiping through each and every question sorted by the model's confidence in its own quality and correctness (weakest first). No question makes it into my bank without me reading, checking all the indicators that came from the models and potentially even rewriting the question to make it better. It all happens in the admin app and exists entirely to prevent hallucinations from reaching real users. Early on I was maybe accepting 1 in 3 questions generated, but I'm probably closer to 2/3rds now. Of course not every dismissed question is wrong. The quality checks I've put in the pipeline are probably giving me 80% questions that are correct and sourced, but sometimes they just aren't what I'm looking for or don't meet my standard.
The best thing I ever did was make an app to review these questions though. Sitting on the couch, kick off a question generation, come back 20 minutes later with 50+ questions in the queue. Sort through them and pull out the 30 best when I have the time. That's the real anti-hallucinations pipeline.
And Anthropic clearly shows that we can reduce hallucination in models. Progress is made which you can see in benchmarks.
And at the end of the day, a normal human being doesn't 'hallucinate' but 'missunderstand' things or just doesn't know what they are missing.
A LLM doesn't has to be perfect, it just has to be cheaper with the min. same quality as a human.
So why are you screaming into the void? Because LLM leave deterministic world? A lot of people don't even life in that world anyway.
You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change.
- [1] https://alexhans.github.io/posts/find-the-loop-story-first.h...
It has now become a way for agents to converge on complete, shared answers for emerging issues. As a side effect, through posting open issues we can also identify gaps in model knowledge.
I'd bet this only really work 2% of the time in real world.
1. Use less subagents. The act of using subagent already needs dumping adequate amount of context for subagent to work effectively
2. Refactor when file grows large. Reasonably scoped files is easier to understand, for both human and agents, and consume less tokens because less guessing which section of the file to read
3. More capable models for planning, learning. Cheap models for execution. Same for human teams, senior member for higher level planning while junior member more on the act of translation of idea into code
4. I wasn't doing any fancy stuff that opus can't handle
5. Consult a second opinion for things you are not sure. (I have made a skill specifically for that)
6. Audit how your habit correlates with token consumption. I found out a skill a use frequently is using a lot of context just so a subagent can get all the context. (made a tool to audit claude specifically)