Perhaps compacting the context can be made in multiple requests over smaller and overlapping chunks to avoid using the 'dumb zone', and for yielding a better result.
I /clear all the time out of habit. I want to be able to get the thing done with minimal context. It also means you can do it again slightly different if needed, you know the seed conditions for the task.
The approach we're taking to deal with this very real context rot is using a bunch of related techniques which we call transposing the agent loop: https://alejo.ch/3jt
In essence, we run many short agent loops, generating their prompts dynamically from structured data. Each loop advances the state in a small step towards the final goal.
I'm getting a lot of mileage out of basically acting like the AI's Product Manager, and insisting that it writes up short PRDs for every feature we propose to build. That gives it a reference over time of everything that has been built, but also makes it less liable to drift with each one. Each one gets its own conversation. For me this is a happy medium between stopping it going off the rails but also making sure it can reference past decisions when it needs to. The one thing I dislike about Pocock's method (not to use PRDs so much but to have an in depth discussion to get alignment) first is it wastes a lot of the best window on that initial back and forth.
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see.
(And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.)
I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name.
Not everybody is using the same model and harness as you, nor using the model the same way as you.
Different models, and versions of models, use different types of attention, which affects their long-context performance, and no doubt also do different amounts/types of long context training.
Different agents build context differently and implement context compaction differently.
Unless someone else is using the same model as you, the same agent/harness as you, and doing very similar tasks, then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours.
I never use Opus etc after 50% token usage (and from reading other devs blogs and on Twitter it seems this is a comment sentiment) because it falls off an intelligence cliff at that point.
I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan.
So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan.
Out of all the methodologies I've tried, this seems to be the best in terms of output quality.
Considerations about what goes on in agents internally will probably not be part of software development for long.
Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request.
To have an ongoing sense of which agents perform best, I keep a log and calculate an ELO score of which agents meet my demands best. This score is imporant to me, not so much how the agent achieves it.
I've had no problem with Claude Code Opus 4.8 effort max using 20% token context (200k) on software development tasks (all stages). I aways load core source files and the ones we are working on up front. Around 20%, I make it autoprepare for a new session and clear.
Admittedly I have been doing this precautiously, based on anecdotal evidence, not because I had bad experiences with longer context deterioration myself.
In the brief time I had access to Fable 5, it went on long running tasks (>45 mins) into the 30-40% zone without apparent context coherence problems.
Even taking the author's criticism about large context windows for granted, which in my experience are exaggerated, they are still a huge UX improvement over short windows. That reason alone is enough for me to support them.
There's an env var you can set in Claude Code to bring the autocompact threshold down, effectively setting your own max context window. I have it at 400k.
In my own testing I have seen peak performance happen usually within 15-20% of the intended context limit, albeit there are a few optimizations depending on the task quality.
I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller.
I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread.
There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time.
Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.
> There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia
Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.
i let the main loop spawn sub terminal via tmux to prevent large contexts. it's great to divide tasks in small patterns and consolidate it step by step.
Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k.
But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results.
The way to have the agent remember things is to have it document its work, like a human developer would do if they wanted their project to be friendly to other developers working on it. Good developer docs with an index page and a good plan with checklists, in concise Markdown files, checked in to the repo is the ideal memory for models and the ideal docs you need to figure out WTF the model has been up to. Helps with code review, too, whether by humans or another model. There's no down side.
I wonder how much this depends on the quality and consistency of the context?
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?
I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important.
This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things.
I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc.
But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.
I built a very small personal extension for Pi [1] that gives me a /last command. It clears the entire session, only retaining the agent's last output message. This allows me to do manual "compaction". Basically I tell the agent something like "state the plan as discussed with references to files that should be edited", and call /last, then tell it to implement.
Considering how expensive context is in terms of compute, I wonder why (and if ) vendors don't invest more into context engineering.
When it comes to source code, I feel like LLMs could just as well work with something like minified source code, if an LLM is trained on programming well, I think there's no reason why something like a variable should be represented by something more than a single token. Comments can be discarded, etc. In fact considering embeddings for LLMs are very rich, I think common ops could be reduced to a single token.
Imo that's why LLMs are soo good at reverse engineering. A lot of the time, assembly (with symbols) is pretty close to the source code, but compressed and encoded, and if you're familiar with the patterns of your compiler, reversing it is not that difficult.
Anyways, context engineering could be huge boon to input token curation imo (and maybe it already is)
67 comments
[ 2.8 ms ] story [ 65.6 ms ] threadIn essence, we run many short agent loops, generating their prompts dynamically from structured data. Each loop advances the state in a small step towards the final goal.
(And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.)
I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name.
Different models, and versions of models, use different types of attention, which affects their long-context performance, and no doubt also do different amounts/types of long context training.
Different agents build context differently and implement context compaction differently.
Unless someone else is using the same model as you, the same agent/harness as you, and doing very similar tasks, then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours.
I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan.
So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan.
Out of all the methodologies I've tried, this seems to be the best in terms of output quality.
Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request.
To have an ongoing sense of which agents perform best, I keep a log and calculate an ELO score of which agents meet my demands best. This score is imporant to me, not so much how the agent achieves it.
Admittedly I have been doing this precautiously, based on anecdotal evidence, not because I had bad experiences with longer context deterioration myself.
In the brief time I had access to Fable 5, it went on long running tasks (>45 mins) into the 30-40% zone without apparent context coherence problems.
I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread.
There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time.
Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.
Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.
But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results.
The way to have the agent remember things is to have it document its work, like a human developer would do if they wanted their project to be friendly to other developers working on it. Good developer docs with an index page and a good plan with checklists, in concise Markdown files, checked in to the repo is the ideal memory for models and the ideal docs you need to figure out WTF the model has been up to. Helps with code review, too, whether by humans or another model. There's no down side.
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?
https://arxiv.org/abs/2506.00069
I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important.
This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things.
I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc.
But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.
[1] https://pi.dev/
When it comes to source code, I feel like LLMs could just as well work with something like minified source code, if an LLM is trained on programming well, I think there's no reason why something like a variable should be represented by something more than a single token. Comments can be discarded, etc. In fact considering embeddings for LLMs are very rich, I think common ops could be reduced to a single token.
Imo that's why LLMs are soo good at reverse engineering. A lot of the time, assembly (with symbols) is pretty close to the source code, but compressed and encoded, and if you're familiar with the patterns of your compiler, reversing it is not that difficult.
Anyways, context engineering could be huge boon to input token curation imo (and maybe it already is)