We try to solve a similar problem to put long documents in context. We built an MCP for Claude to allow you to put long PDFs in your context window that go beyond the context limits: https://pageindex.ai/mcp.
That’s powerful. Most of the differences I can see between AI generated output and human output comes from the « broad but specific » context of the task. I mean company culture, organization rules and politics, larger team focus and way of working.
It may take time to build the required knowledge bases but it must be worth it
Nice. When using OpenAI Codex CLI, I find the /compact command very useful for large tasks. In a way it's similar to the context editing tool. Maybe I can ask it to use a dedicated directory to simulate the memory tool.
Context editing is interesting because most agents work on the assumption that KV cache is the most important thing to optimise and are very hesitant to remove parts of the context during work. It also sometimes introduces hallucinations, because parts of the context are with the assumption that eg tool results are there, but theyre not. Example Manus [0]. Eg, read file A, make changes on A. Then prompt on some more changes. If you now remove the "read file A" tool results, not only you break the cache, but in my own agent implementations(on gpt 5 at least) can hallucinate now since my prompt etc all naturally point to the content of the tool still beeing there.
Plus, the model got trained and RLed with a continuous context, except if they now tune it with messing with the context as well.
I’m trying to understand what part of this is something we could not have hacked together already as clients? Maybe new sonnet is rl’ed to be able to use these memories in a better way?
At Zenning AI, a generalist AI designed to replace entire jobs with just prompts. Our agents typically run autonomously for hours, so effective context management is critical. I'd say that we invest most of our engineering effort into what is ultimately context management, such as:
1. Multi-agent orchestration
2. Summarising and chunking large tool and agent responses
3. Passing large context objects by reference between agents and tools
Two things to note that might be interesting to the community:
Firstly, when managing context, I recommend adding some evals to our context management flow, so you can measure effectiveness as you add improvements and changes.
For example, our evals will measure the impact of using Anthropics memory over time. Thus allowing our team to make a better informed decisions on that tools to use with our agents.
Secondly, there's a tradeoff not mentioned in this article: speed vs. accuracy. Faster summarisation (or 'compaction') comes at a cost of accuracy. If you want good compaction, it can be slow. Depending on the use case, you should adjust your compaction strategy accordingly. For example, (forgive my major generalisation), for consumer facing products speed is usually preferred over a bump in accuracy. However, in business accuracy is generally preferred over speed.
The hardest part of that journey is that there are a _lot_ of people out there now thinking the same thing. It is really difficult to get into these jobs without a network helping you.
I don’t get it. I have been doing something similar for a month with opencode, is the new thing that the new sonnet model is fine tuned to call these tools “better” or simply that they have improved the devex to accomplish these things?
From what I can tell the new context editing and memory APIs are essentially formalization of common patterns:
Context editing: Replace tool call results in message history (i.e replace a file output with an indicator that it’s no longer available).
Memory: Give LLM access to read and write .md files like a virtual file system
I feel like these formalizations of tools are on the path towards managing message history on the server, which means better vendor lock in, but not necessarily a big boon to the user of the API (well, bandwidth and latency will improve). I see the ChatGPT Responses API going a similar path, and together these changes will make it harder to swap transparently between providers, something I enjoy having the ability to do.
> managing message history on the server, which means better vendor lock in
I feel that managing context should be doable with a non-SOTA model even locally. Just need a way to select/deselect messages from the context manually say in Claude-CLI.
I wish every instruction and response had a enable/disable checkbox so that I can disable parts of the conversation in such a way that it is excluded from the context.
Let's say I submit or let it create a piece of code, and we're working on improving it. At some point I want to consider the piece of code to be significantly better that what I had initially, so all those initial interactions containing old code could be removed from the context.
I like how Google AI Studio allows one to delete sections and they are then no longer part of the context. Not possible in Claude, ChatGPT or Gemini, I think there one can only delete the last response.
Maybe even AI could suggest which parts to disable.
Related, it feels like AI Studio is the only mainstream LLM frontend that treats you like an adult. Choose your own safety boundaries, modify the context & system prompt as you please, clear rate limits and pricing, etc. It's something you come to appreciate a lot, even if we are in the part of the cycle where Google's models aren't particularly SOTA rn
I went with an extra CURRENT.md for whatever extra info that might be useful for what I am working on and frequently /clear after each very small task. /compact is rarely used unless there are reasons to maintain a summary on what its working on.
Each new prompt involves asking Claude to read CURRENT.md for additional context.
I'm not sure if I should move this to CLAUDE.md but the stuff in CURRENT.md are very short term information that gets useless after a while.
---
There is one time where Claude entirely messed up the directory when moving things around and it sort of stuck in a weird "panic" loop in chat for quite a while (involving oh no / oh dear in chat), nothing git can't fix, but I suspect is due to the directory info in CLAUDE.md getting stale. Ever since then I moved things that might get stale to a separate file and frequently keep it updated/trimmed as needed.
> I like how Google AI Studio allows one to delete sections and they are then no longer part of the context. Not possible in Claude, ChatGPT or Gemini, I think there one can only delete the last response.
I have the same peeve. My assumption is the ability to freely edit context is seen as not intuitive for most users - LLM products want to keep the illusion of a classic chat UI where that kind of editing doesn't make sense. I do wish ChatGPT & co had a pro or advanced mode that was more similar to Google AI Studio.
The SolveIt tool [0] has a simple but brilliant feature I now want in all LLM tools: a fully editable transcript. In particular, you can edit the previous LLM responses. This lets you fix the lingering effect of a bad response without having to back up and redo the whole interaction.
FWIW, in Claude Desktop you can edit previous user context and Claude will fork the conversation from that point. I know it's not quite what you as asking for, but it's something.
There are 3rd-party chat interfaces out there that have much better context controls if it matters enough for you that you're willing to resort to direct API usage.
I think the main issue with removing certain previous responses from context would be that you no longer hit the cache for a large part of your chat history, which makes responses much more expensive and slow.
Its faster and cheaper (in most cases) to leave the history as is and hit the cache.
I noticed this in the Claude Code interface, I reached "8% context left" but after giving it a huge prompt the warning disappeared, and it kept on working for another 20 minutes before reaching again "10% context left", but it never had to compact the history of the conversation. 10/10 great feature.
> Enable longer conversations by automatically removing stale tool results from context
Boost accuracy by saving critical information to memory—and bring that learning across successive agentic sessions
Funny, I was just talking about my personal use of these techniques recently (tool output summarization/abliteration with memory backend). This isn't something that needs to be Claude Code specific though, you can 100% implement this with tool wrappers.
I've been doing this for a bit, dropping summarized old tool output from context is a big win, but it's still level ~0 context engineering. It'll be interesting to see which of my tricks they figure out next.
I wish claude code supported the new memory tool. The difference is CLAUDE.md is always in your active context while the new memory stuff is essentially local RAG.
I don't think cline ever had to do much context management, which is why i prefer them over cursor. Cline is similar to claude code, but without the vendor lock in.
I'll let them (ai companies) figure it out while i keep coding manually.
I'll wait for the day they will release a js library to programmatically do all this llm context juggling, instead of using a UI, and then I will adopt it by doing what I do now, writting code.
I will write code that orchestrates llms for writting code.
Edit: This is obviously a joke... but is it really a joke?
Why client side? Doesn't that mean re-uploading the whole context whenever you change something? Seems like storing it server-side and providing an API to swap pieces in and out of context would be a lot more efficient. Or if the LLM is the one deciding what to swap, that could be done without client involvement at all. Even better would be if this API was standardized across providers.
Outside of context manipulation, it'd also be nice to standardize a format to label sections of the context and later append keywords to the context to ignore / unignore those sections (or refer back to them, emphasize them, whatever). With that, I imagine we'd be able to have a pretty good set of standard LoRA adapters that enable all LLMs to be controlled in the same fashion. That way agents will be able to manipulate LLMs in a standard way, without having to rewrite context itself.
Context management is one of those things that seems simple until you actually build with it. We've run into issues where Claude loses important context halfway through complex tasks. Love the idea of being able to mark parts of the conversation as 'resolved' or 'outdated' would be huge.
38 comments
[ 2.8 ms ] story [ 67.6 ms ] threadWe try to solve a similar problem to put long documents in context. We built an MCP for Claude to allow you to put long PDFs in your context window that go beyond the context limits: https://pageindex.ai/mcp.
Plus, the model got trained and RLed with a continuous context, except if they now tune it with messing with the context as well.
https://manus.im/blog/Context-Engineering-for-AI-Agents-Less...
1. Multi-agent orchestration 2. Summarising and chunking large tool and agent responses 3. Passing large context objects by reference between agents and tools
Two things to note that might be interesting to the community:
Firstly, when managing context, I recommend adding some evals to our context management flow, so you can measure effectiveness as you add improvements and changes.
For example, our evals will measure the impact of using Anthropics memory over time. Thus allowing our team to make a better informed decisions on that tools to use with our agents.
Secondly, there's a tradeoff not mentioned in this article: speed vs. accuracy. Faster summarisation (or 'compaction') comes at a cost of accuracy. If you want good compaction, it can be slow. Depending on the use case, you should adjust your compaction strategy accordingly. For example, (forgive my major generalisation), for consumer facing products speed is usually preferred over a bump in accuracy. However, in business accuracy is generally preferred over speed.
For context: I have background in CV and ML in general. Currently reviewing and revising RL.
Any idea how I can get into RL?
I have 3 years of industry/research experience.
Whenever I see post like this, it triggers a massive fomo creating a scene of urgency on I should work in these problems.
Not being able to work here is making be anxious.
what does it take for someone in Non-US/Non-EU region to get into big labs such as these?
Do I really have to pursue PhD? I am already old that pursuing PhD is a huge burden that I can't afford.
Context editing: Replace tool call results in message history (i.e replace a file output with an indicator that it’s no longer available).
Memory: Give LLM access to read and write .md files like a virtual file system
I feel like these formalizations of tools are on the path towards managing message history on the server, which means better vendor lock in, but not necessarily a big boon to the user of the API (well, bandwidth and latency will improve). I see the ChatGPT Responses API going a similar path, and together these changes will make it harder to swap transparently between providers, something I enjoy having the ability to do.
I feel that managing context should be doable with a non-SOTA model even locally. Just need a way to select/deselect messages from the context manually say in Claude-CLI.
Let's say I submit or let it create a piece of code, and we're working on improving it. At some point I want to consider the piece of code to be significantly better that what I had initially, so all those initial interactions containing old code could be removed from the context.
I like how Google AI Studio allows one to delete sections and they are then no longer part of the context. Not possible in Claude, ChatGPT or Gemini, I think there one can only delete the last response.
Maybe even AI could suggest which parts to disable.
Each new prompt involves asking Claude to read CURRENT.md for additional context.
I'm not sure if I should move this to CLAUDE.md but the stuff in CURRENT.md are very short term information that gets useless after a while.
---
There is one time where Claude entirely messed up the directory when moving things around and it sort of stuck in a weird "panic" loop in chat for quite a while (involving oh no / oh dear in chat), nothing git can't fix, but I suspect is due to the directory info in CLAUDE.md getting stale. Ever since then I moved things that might get stale to a separate file and frequently keep it updated/trimmed as needed.
I have the same peeve. My assumption is the ability to freely edit context is seen as not intuitive for most users - LLM products want to keep the illusion of a classic chat UI where that kind of editing doesn't make sense. I do wish ChatGPT & co had a pro or advanced mode that was more similar to Google AI Studio.
[0] https://news.ycombinator.com/item?id=45455719
There are 3rd-party chat interfaces out there that have much better context controls if it matters enough for you that you're willing to resort to direct API usage.
Its faster and cheaper (in most cases) to leave the history as is and hit the cache.
Funny, I was just talking about my personal use of these techniques recently (tool output summarization/abliteration with memory backend). This isn't something that needs to be Claude Code specific though, you can 100% implement this with tool wrappers.
I've been doing this for a bit, dropping summarized old tool output from context is a big win, but it's still level ~0 context engineering. It'll be interesting to see which of my tricks they figure out next.
I'll wait for the day they will release a js library to programmatically do all this llm context juggling, instead of using a UI, and then I will adopt it by doing what I do now, writting code.
I will write code that orchestrates llms for writting code.
Edit: This is obviously a joke... but is it really a joke?
Outside of context manipulation, it'd also be nice to standardize a format to label sections of the context and later append keywords to the context to ignore / unignore those sections (or refer back to them, emphasize them, whatever). With that, I imagine we'd be able to have a pretty good set of standard LoRA adapters that enable all LLMs to be controlled in the same fashion. That way agents will be able to manipulate LLMs in a standard way, without having to rewrite context itself.