If you look at how agents navigate source code, they do not look at directory names, and drill down into the ones with plausible names, instead the grep the whole repo for plausible keywords.
Of course, ideally your data would be structured, but the agents will mostly be grepping anyway, and maybe look at sibbling files.
I've come to a similar lofi solution for my agent fleet. Markdown wiki with simple querying is decently effective as a memory system. Setting up a skill that can effectively reduce a session into useful long term lessons is the easiest unlock for these systems.
This was a compelling writeup to me. I read through the spec and found it easy to understand and make sense of.
I wonder how much my system needs something like this. Between the invisible system memory of my random chats with Gippity, my Matt Pocock skills saving terminology and plans, and whatever else Cursor and Codex do, I don't think I feel a need for more agent memory. I do like how it's exposed and searchable, and not invisible. But I honestly just send my questions/tasks away to my magic agent and eventually it gets it right anyway; do I need more discrete memory my team has to maintain? (That's an earnest question, not disregard for this)
that sounds horribly token inefficient, just create a tool call if you are all in on the agentic approach and hide memory retrieval behind an optimized api
What the author suggests is remarkably close to the proposition of OpenViking. I've been testing a few memory solutions and OpenViking is one of my favorites so far.
I see a lot of claims in this article without ... any proof?
Both can be true:
- It's useful to anthropomorphize agents when predicting behavior and
- we have to use specific language to specify what we mean.
What does the author mean by "confuse the models" ? Are they talking about not picking right information? Picking the wrong information? Losing their previous context / task?
Part of setting up a proper eval is also deciding what we actually mean ourself. What are we actually optimizing for? It's not, e.g. % confusion, %rubbish, etc.
The article does point to it: retrieval latency, accuracy, etc.
It’s good that a lot of people are trying a lot of things when it comes to agentic memory. Sadly none of it represents a complete solution at this time. But we need the experimentation.
> How can I judge what is a good memory to store? How can I avoid filling my memory with crap?
> This is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search.
This is so wrong. The Achilles' heel of this approach is the RAG. What makes it worse is having lots of memories that are outdated, wrong, hallucinated, or irrelevant.
Nothing beats curated data. Memory should be regularly reviewed, compacted, and cleaned up if it's no longer valid.
Cheapest version of that review is already sitting in the format. Frontmatter carries created and updated, so you can sort by staleness and drop whatever nothing touched in months
I find once there is one poisoned line of text it negatively affects everything else downstream. Instead, I use a temp/ folder with documents and use different files for different agents and models. Then I have to constantly prune and delete the files. Any information that can be extrapolated is just noise which negatively affects the agent. If you have a definition of a database structure and it has been implemented, that information should not be contained in any text document -- it is noise, will drift, and be impossible to debug why the agent keeps producing undesired behavior.
I have a ~/Projects folder. For example, I use Playwright with Chrome DevTools Protocol in order to do performance testing and leak detection. There is a script that handles this. My prompt is "Search ~/Projects for perf testing with CDP and Playwright and implement here". Point being, if I need anything I point to a resource or ask to search a resource and it will find it quick and, most importantly, tends to improve it every iteration.
If I was in an institution, I would have a repository and would rather just point the resource and say use that than have memory of it locally.
What I propose is only a very slightly more formal version of what you describe.
Just to start with: memoryfields are possible to use in a server/client system. That was a key aim and I do already use them over Amazon S3 (though not always).
I started, like you did, with a personal library of prompts. But the issue is that as your library of little pieces of prompts increases a) you get tired of constantly editing them yourself b) you have no easy way to export and share them with others c) it's frustrating that the agent doesn't "automatically" find your little bit of prompt on X even when clearly it is relevant - hence sem search.
I think a lot of people are still using the "personal library of bits of prompt" model. It is ok. But I wanted to propose an minimal, interchangeable standard for sharing them. So the idea of being an institution and having a shared memoryfield: that's something I want as well!
I have mine self record into Supabase for my projects and a local sqlite for work. It determines a method of record keeping which I audit every week to hone the process. Really makes it so I can move to any provider I want and I have a queryable memory store. Also really helps when someone asks about why some feature was implemented a certain way. I also have it learn from corrections in PRs and comments made overtime in a repo to get the shape of what is important to the team at work.
Your experience mirrors my own. I don't know if he coined the term, but Steve Yegge talks about 'heresies' that creep in to a system -- untrue things that stick around and permanently influence its behavior. I still find that these happen regularly and stopped using self-managing memory systems because they make heresies even harder to diagnose and remove.
Within projects, I make heavy use of path-scoped rules to intentionally bring context where it's needed, and also make heavy use of temp directories. LLMs are more than happy to produce ad-hoc memories/summaries/context docs that I can then point a session to, but I can be selective and intentional about it.
I like that memoryfield is portable, intentional and composable. I'm not convinced that sharing memoryfields between users will be practical, but I keep isolated virtual environments for absolutely everything. I like the idea of being able to intentionally bring collections of managed context around with me. There are other ways to do that, but will keep an eye on this.
This is a real problem, and it’s not just in markdown files and docs. Claude loves to write things we’ve “discovered” in comments and then later in treats the comments as gospel truth.
You have to constantly tend the garden and weed these things out.
Whenever Claude makes some error ask it where and why? And then dig out the weed.
And of course if it’s in the context, probably time for a handover doc (which will need weeding) and started fresh.
> This is a real problem, and it’s not just in markdown files and docs. Claude loves to write things we’ve “discovered” in comments and then later in treats the comments as gospel truth.
There's a simple fix for this: do not let it write comments. Ever. This also has the nice property that there's much less AI slop to clean up afterwards.
Memory is prone to poisoning. Also, I want to be able to take my toys and go elsewhere.
My method is several layers of files, administered differently. I count seven: meta-knowledge, project seed, wiki, code, tickets and todos, chat logs, artifactory. Ordered idea-to-reality. All git repos.
That outgrows any context window pretty soon. My only way out of that trap is to use links, both wiki links and git permalinks.
I have a strict rule no project info should end up in memory, it should be in the docs, memory is just for feedback re interactions - models tend to have weak spots more frequently missing some agents specific instruction than others.
But in general newer models got a bit better at writing their own instructions/docs/memories (still bad, but better), so yeah on all projects started with older models big refactoring or just removing memory seems like a good idea. I don't think companies pay enough attention how negatively bad memories (esp conflicting ones) can impact performance.
It keeps documentation from going out of date by embedding re-runnable verification checks directly inside markdown files (that the agents use, not humans). I use it with a handoff workflow to force Claude/GPT to re-verify facts before handing off documentation to the next session.
I have memory disabled in all my Chat UIs (even though it tends to creep in, looking at you ChatGPT). It’s very helpful until it scales with time, at which point it becomes useless due to staleness or mis-application across contexts.
That being said, in coding over a longer time horizon, having the agent continually re-derive decisions/laws/facts/etc from your code is wasteful of tokens and time, and if your code doesn’t consistently apply them you can’t know the agent will make the correct choices.
You need memory of these important facts to avoid this expense or potential incorrectness. Memory does not itself scale though, without maintenance and pruning, and that has its own impacts on cost and correctness like the Chat memory.
“Damned if you do, damned if you don’t” at least until the agent can itself maintain its memory accurately - or some other non-human effort can achieve that.
e.g. docs/decisions/README.md (index with a blurb about each decision), docs/decisions/01-some-lesson.md (some architectural decision/pattern that you or the agents discovered).
ADR files have important sections like "rejected solutions" and "acceptable risks", and they're live files that can be refined and pivoted over time or retired to docs/decisions/archive/.
It's also nice to give each top-level bullet point some stable ID like "R1" for rejected solution #1, I3 for invariant #3. Agents use this stuff intelligently all the time like "This could be a time to reconsider D4/R2" = ADR #4, rejected solution #2.
The essential part being that your system ratchets into increasingly better decisions and invariants over time, and there's a place to actually put this stuff.
It's essential for automating high-quality software and something we couldn't be arsed to do much less update before AI.
I want to add to my comment the other thing I do is focus on deleting first. In AGENTS.md, which I reference in CLAUDE.md, at the top there is an instruction to first delete anything that is causing a behavior either in agent instructions or code. Then the next step if that doesn't solve changing the behavior is to update the markdown or code to change the behavior. Lastly, the instruction is as a last resort add instructions or code.
It is an elegant strategy in that it first finds the a problem and removes it rather than adding a bandaid.
Dear Claude, what is in the instructions and guidance that is causing this undesired behavior? Can you remove it?
You may be interested in my No3371/projex repo, which is a whole solution based in this idea, I wonder how much do people do differently in this direction
I find that semantic search is substantially better than keyword search even for small corpuses. Being able to find "related" material that doesn't match the keyword is a big advance over traditional full text search.
Was ready to write something snarky because this is essentially RAG, but I think the author is getting at some subtle details which are seemingly important.
- memory systems are a specific type of knowledge base where you generate all the documents. You might as well generate them to be less than your embedding token limit to obviate the need for chunking.
- embedding models are getting better and are no longer just semantic averaging.
- small models are getting dirt cheap, making parallel reads cost manageable
What they describe is sort of the simplest architecture that takes advantage of these observations. I believe them when they say it works well.
I do suspect though that things like keyword lookup will completely fail if every memory is just a vector. Hence why something like Typesense hybrid search can still be useful.
I'm starting to think that 'memory' may be the wrong analogy for what we want.
I do think that having a set of token that are highly personalized to your project and to way you work is beneficial. I also think that the idea that this set of token will be constructed in the background without any work from the user is really appealing. So it's understandable that the 'memory' analogy became so popular.
But in my experience having a really good AGENTS.md file almost always produce better results than enabling memory.
Maybe we should start to think about how we 'train'/'onboard' agents into our projects, in a similar way that we do for new co-workers. Imagine if we could send the agent to our repo and ask it to learn our patterns and in the end we could quiz the agent to gauge how much it actually understood the project. Once he 'understands' the project we can start to use it to help with development.
In a very small scale (example, individual new features) I will sometimes ask the agent to explain me how things work (even though I already know how it works) so I can 'prime' the agent context with good data before starting any real work. But I'm not sure if this approach could be reliably scaled to work with any repo for any kind of work.
When I was doing a really large refactor across the codebase I told Claude Code to explain how certain things worked currently and how I wanted things to look like after the migration and a plan on how to get there. Then I did several further clean sessions where I always started along the lines: "Using this plan {link} do X."
Works pretty well for "non-permanent" instructions (you don't want to put this info into your committed markdowns).
My main motivation was simply to save tokens, but it actually worked really well and improved speed as well.
we want some kind of fractal knowledge graph. It starts coarse at some zoom level and you can move in and out. You can insert knowledge at any level and it's out/in levels adjust accordingly. The search semantics at every level are the same, but whats in the visible area changes based on what your focus is.
One way I've toyed with a graph outline with it using "whitening" (https://arxiv.org/pdf/2104.01767v3) for embeddings rather than just text, so you add things like file path, nearest title/method/const etc. You have to have dummy text though because it fails with "null"; all embeddings need to carry some kind of text and of the same size.
so when the agent rembembers something, the memory would be an embedding that includes where they found the file, what method or const or whatever they're in, what the task they're working on is, etc. That all becomes a single embedding. You could imagine a metadata tag that also describes the tools they're using etc.
Map out a complete space of tags for whitening an embedding and there's surely a proper mix. Then when you're searching for things in the embedding, you also store some of the other metadata as plane strings & edges, which gets you some useful granularity.
It's really about storing institutional context and on-task learnings. The AGENTS.md can do the same thing as memory, but if you have a good memory system, in theory you never need to do any ongoing maintenance of AGENTS.md and the system will do it automatically. And because it's automatic and has a whole harness system built around maintaining the memories, you can engineer it to have not just more convenience of maintenance but also better correctness / recall quality.
Memory systems can also let AI load memories on demand so it's actually analogous to maintaining AGENTS.md plus twenty different "read this if you need to do X" markdown files.
I think the name "memory" makes sense given that the AI system is one singular system with central context (rather than a software org of multiple distinct humans with distinct memories), so the equivalent of institutional knowledge in the software org really is just akin to memory for the AI system.
I just had a thought about similar thing - how to track human decisions on the codebase? Consider you are writing code together with AI, how you understand which code change happened because human asked for it
"Irrelevant material is simply never surfaced by the semantic search."
thats quite optimistic. there's lots of "memory" or past chats with agents that should be suppressed and forgotten because they were looking in the wrong place or were eventually proven wrong. yet semantically they'd look very relevant to a future search. thats why you shouldn't search both textbooks and scifi when trying to solve an examination.
It occurs to me: we have latent embedding giving 'general knowledge' to an LLM. What if we use a 'blank' LLM as well as an agent and train that blank LLM on personal context to query that as memory?
Can an LLM be trained to understand language without remembering anything else from its training data? I thought the intrinsic knowledge and the ability to understand language were tied together.
It would be closer to using an LLM as a RAG for memory, as the reasoning LLM in injected with the return of the 'memory llm' (maybe with a defined number of 'slots' for easy clean up).
All of this stuff seems like a band-aid solution. These things need to be trained ground-up to have some sort of memory, perhaps out-of-context memory. Also seems like a requirement for any sort of continual learning capabilities as well.
Memory is not necessarily a good thing. What we need is a combination of sufficiently large context windows (10-100 million tokens) along with curate, bloat-free data.
You can kludge it together by having the agent keep a log of what it does with any notes and lessons about friction / efficiency, and then on some interval review that + session history for items worth promoting into a topic-based memory, or for things that are needed constantly into the main AGENTS.md or a separate MEMORY.md that's loaded into every session. Reviewing the notes at the same time as session history helps provide enough context to make directionally correct decisions about what is worth keeping in context for future sessions.
92 comments
[ 558 ms ] story [ 928 ms ] threadBut it's still text. You show people how the sausage is made and either they're confused or they're horrified.
The markdown explanation I was also unimpressed with, but RAG over hyperlinks is convincing to me.
Of course, ideally your data would be structured, but the agents will mostly be grepping anyway, and maybe look at sibbling files.
I wonder how much my system needs something like this. Between the invisible system memory of my random chats with Gippity, my Matt Pocock skills saving terminology and plans, and whatever else Cursor and Codex do, I don't think I feel a need for more agent memory. I do like how it's exposed and searchable, and not invisible. But I honestly just send my questions/tasks away to my magic agent and eventually it gets it right anyway; do I need more discrete memory my team has to maintain? (That's an earnest question, not disregard for this)
Incredible to watch things come full circle. Next thing you know, someone is going to figure out a binary encoding.
Both can be true: - It's useful to anthropomorphize agents when predicting behavior and - we have to use specific language to specify what we mean.
What does the author mean by "confuse the models" ? Are they talking about not picking right information? Picking the wrong information? Losing their previous context / task?
Part of setting up a proper eval is also deciding what we actually mean ourself. What are we actually optimizing for? It's not, e.g. % confusion, %rubbish, etc.
The article does point to it: retrieval latency, accuracy, etc.
> This is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search.
This is so wrong. The Achilles' heel of this approach is the RAG. What makes it worse is having lots of memories that are outdated, wrong, hallucinated, or irrelevant.
Nothing beats curated data. Memory should be regularly reviewed, compacted, and cleaned up if it's no longer valid.
I find once there is one poisoned line of text it negatively affects everything else downstream. Instead, I use a temp/ folder with documents and use different files for different agents and models. Then I have to constantly prune and delete the files. Any information that can be extrapolated is just noise which negatively affects the agent. If you have a definition of a database structure and it has been implemented, that information should not be contained in any text document -- it is noise, will drift, and be impossible to debug why the agent keeps producing undesired behavior.
I have a ~/Projects folder. For example, I use Playwright with Chrome DevTools Protocol in order to do performance testing and leak detection. There is a script that handles this. My prompt is "Search ~/Projects for perf testing with CDP and Playwright and implement here". Point being, if I need anything I point to a resource or ask to search a resource and it will find it quick and, most importantly, tends to improve it every iteration.
If I was in an institution, I would have a repository and would rather just point the resource and say use that than have memory of it locally.
Just to start with: memoryfields are possible to use in a server/client system. That was a key aim and I do already use them over Amazon S3 (though not always).
I started, like you did, with a personal library of prompts. But the issue is that as your library of little pieces of prompts increases a) you get tired of constantly editing them yourself b) you have no easy way to export and share them with others c) it's frustrating that the agent doesn't "automatically" find your little bit of prompt on X even when clearly it is relevant - hence sem search.
I think a lot of people are still using the "personal library of bits of prompt" model. It is ok. But I wanted to propose an minimal, interchangeable standard for sharing them. So the idea of being an institution and having a shared memoryfield: that's something I want as well!
The spec, feedback greatly welcome:
https://github.com/calpaterson/memoryfield-spec/blob/main/SP...
Within projects, I make heavy use of path-scoped rules to intentionally bring context where it's needed, and also make heavy use of temp directories. LLMs are more than happy to produce ad-hoc memories/summaries/context docs that I can then point a session to, but I can be selective and intentional about it.
I like that memoryfield is portable, intentional and composable. I'm not convinced that sharing memoryfields between users will be practical, but I keep isolated virtual environments for absolutely everything. I like the idea of being able to intentionally bring collections of managed context around with me. There are other ways to do that, but will keep an eye on this.
You have to constantly tend the garden and weed these things out.
Whenever Claude makes some error ask it where and why? And then dig out the weed.
And of course if it’s in the context, probably time for a handover doc (which will need weeding) and started fresh.
There's a simple fix for this: do not let it write comments. Ever. This also has the nice property that there's much less AI slop to clean up afterwards.
My method is several layers of files, administered differently. I count seven: meta-knowledge, project seed, wiki, code, tickets and todos, chat logs, artifactory. Ordered idea-to-reality. All git repos.
That outgrows any context window pretty soon. My only way out of that trap is to use links, both wiki links and git permalinks.
http://replicated.live/blog/wiki
But in general newer models got a bit better at writing their own instructions/docs/memories (still bad, but better), so yeah on all projects started with older models big refactoring or just removing memory seems like a good idea. I don't think companies pay enough attention how negatively bad memories (esp conflicting ones) can impact performance.
It keeps documentation from going out of date by embedding re-runnable verification checks directly inside markdown files (that the agents use, not humans). I use it with a handoff workflow to force Claude/GPT to re-verify facts before handing off documentation to the next session.
That being said, in coding over a longer time horizon, having the agent continually re-derive decisions/laws/facts/etc from your code is wasteful of tokens and time, and if your code doesn’t consistently apply them you can’t know the agent will make the correct choices.
You need memory of these important facts to avoid this expense or potential incorrectness. Memory does not itself scale though, without maintenance and pruning, and that has its own impacts on cost and correctness like the Chat memory.
“Damned if you do, damned if you don’t” at least until the agent can itself maintain its memory accurately - or some other non-human effort can achieve that.
e.g. docs/decisions/README.md (index with a blurb about each decision), docs/decisions/01-some-lesson.md (some architectural decision/pattern that you or the agents discovered).
ADR files have important sections like "rejected solutions" and "acceptable risks", and they're live files that can be refined and pivoted over time or retired to docs/decisions/archive/.
It's also nice to give each top-level bullet point some stable ID like "R1" for rejected solution #1, I3 for invariant #3. Agents use this stuff intelligently all the time like "This could be a time to reconsider D4/R2" = ADR #4, rejected solution #2.
The essential part being that your system ratchets into increasingly better decisions and invariants over time, and there's a place to actually put this stuff.
It's essential for automating high-quality software and something we couldn't be arsed to do much less update before AI.
It is an elegant strategy in that it first finds the a problem and removes it rather than adding a bandaid.
Dear Claude, what is in the instructions and guidance that is causing this undesired behavior? Can you remove it?
Two agent.MD files that are very small. One on each project. One at parent project level.
Did 30M tokens through glm 5.3 flash today for 52c
Using pi and a few extensions my initial context is always 4k max
- memory systems are a specific type of knowledge base where you generate all the documents. You might as well generate them to be less than your embedding token limit to obviate the need for chunking.
- embedding models are getting better and are no longer just semantic averaging.
- small models are getting dirt cheap, making parallel reads cost manageable
What they describe is sort of the simplest architecture that takes advantage of these observations. I believe them when they say it works well.
I do suspect though that things like keyword lookup will completely fail if every memory is just a vector. Hence why something like Typesense hybrid search can still be useful.
I do think that having a set of token that are highly personalized to your project and to way you work is beneficial. I also think that the idea that this set of token will be constructed in the background without any work from the user is really appealing. So it's understandable that the 'memory' analogy became so popular.
But in my experience having a really good AGENTS.md file almost always produce better results than enabling memory.
Maybe we should start to think about how we 'train'/'onboard' agents into our projects, in a similar way that we do for new co-workers. Imagine if we could send the agent to our repo and ask it to learn our patterns and in the end we could quiz the agent to gauge how much it actually understood the project. Once he 'understands' the project we can start to use it to help with development.
In a very small scale (example, individual new features) I will sometimes ask the agent to explain me how things work (even though I already know how it works) so I can 'prime' the agent context with good data before starting any real work. But I'm not sure if this approach could be reliably scaled to work with any repo for any kind of work.
Works pretty well for "non-permanent" instructions (you don't want to put this info into your committed markdowns).
My main motivation was simply to save tokens, but it actually worked really well and improved speed as well.
One way I've toyed with a graph outline with it using "whitening" (https://arxiv.org/pdf/2104.01767v3) for embeddings rather than just text, so you add things like file path, nearest title/method/const etc. You have to have dummy text though because it fails with "null"; all embeddings need to carry some kind of text and of the same size.
so when the agent rembembers something, the memory would be an embedding that includes where they found the file, what method or const or whatever they're in, what the task they're working on is, etc. That all becomes a single embedding. You could imagine a metadata tag that also describes the tools they're using etc.
Map out a complete space of tags for whitening an embedding and there's surely a proper mix. Then when you're searching for things in the embedding, you also store some of the other metadata as plane strings & edges, which gets you some useful granularity.
Memory systems can also let AI load memories on demand so it's actually analogous to maintaining AGENTS.md plus twenty different "read this if you need to do X" markdown files.
I think the name "memory" makes sense given that the AI system is one singular system with central context (rather than a software org of multiple distinct humans with distinct memories), so the equivalent of institutional knowledge in the software org really is just akin to memory for the AI system.
I just had a thought about similar thing - how to track human decisions on the codebase? Consider you are writing code together with AI, how you understand which code change happened because human asked for it
I.e. for the web we did that with link count etc.
We need some other mechanism for judging and ranking pieces of "memory" for "agents"