> My setup is a laptop and a mac mini without an interface. The agent can work on the mini all night, and in the morning I extract its memory. Then the agent on my laptop will know what the mini tried, what broke, and what eventually worked.
If it’s fully automated and then blindly injected into your laptop without any vetting, isn’t that a perfect vector to break that separation?
This is a fair point of view and ofc it should be taken into account. When called automatically, a small overview (~2 KB) of the imported sessions is included in the context.
However, when syncing, only data records are moved not instructions. Nothing is done and known secrets are deleted at the indexing stage even before they leave the source computer. In addition, the context here is your own computers and your terminal's output data, if an attacker can control your agent's original decryptions, the source computer has already been completely compromised.
Nevertheless, you make a perfectly reasonable case for strict isolation. I just opened the issue to add an opt-out checkbox where imported sessions are only displayed when manually searched using the CLI/MCP, not when automatically implemented: https://github.com/vshulcz/deja-vu/issues/39
I agree that similarity search can't surface reliable project judgment alone. How do you know what was true, what was rejected, and what's obsolete since changes in the project? I am actually building Open-Latch for this reason. It leverages project decisions, facts, lifecycle state, reconciliation, enforcing cited evidence and a judgement gate before coding agents act.
I felt the same annoyance as OP mentioned, but focused on agents violating project judgement and direction. Obviously we are optimizing for some different things than deja: deja uses no-LLMs and exact verbatim transcript retrieval.
Latch uses LLMs, ranking, etc, to emphasize strong judgment, correction, and enforcement over dynamic project flow. Give latch a try if you're also running into decision amnesia in your workflow.
Yes, ctx is the closest cousin here. I actually linked it in my top comment.
This causes a very similar itch, but we went a slightly different route regarding some implementation details
Secrets: The ctx README mentions saving the text as is and warns about the need to review the output data before publishing. deja explicitly removes known secret templates at the indexing stage (replacing them with tags like [redacted:aws-key]).
Syncing: It looks like ctx is getting ready for a cloud beta to be shared by the team. Actually, I just wanted something local and free that I had full control over, so deja does P2P syncing through your existing SSH settings.
Recall that deja has a session startup feature that automatically introduces a bit of context when a new agent is loaded, instead of relying solely on manual search.
Both applications solve the same annoying problem by simply optimizing for different workflows :)
This a session search tool, and I think calling it a memory tool is confusing.
> Secrets: The ctx README mentions saving the text as is and warns about the need to review the output data before publishing. deja explicitly removes known secret templates at the indexing stage (replacing them with tags like [redacted:aws-key]).
How is this not security theater? You've presumably already sent the api key to an LLM if it's in your session file. You need to rotate it.
Maybe somebody can enlighten me... this page for ctx says the program saves all "decisions, constraints, intent, rejected approaches, bug investigations, refactors, file paths, commands, patches, and notes from previous agents."
What am I missing if I instead just instruct the agent to create a handful of files called something like HISTORY.md and tell it to log summarized versions of all of these things inside the project repo whenever something is changed or evaluated? (Which is very close to what I currently do.)
If I want that information to be available to other people/agents who use the repo, they are not really polluting the repo? Am I doing vibecoding wrong?
I like that this stays local instead of depending on a hosted service. One thing I've noticed while building LLM applications is that memory becomes much more valuable when it's easy to inspect and edit manually. Are you planning to support semantic search later, or is the goal to keep everything deterministic and text-based?
I intentionally used a dictionary for this. When I dig into old sessions, I'm usually looking for a very specific token (the exact error string, the unique function name, a specific cli flag) rather than a generic value.
Adding semantic search usually means sending hundreds of megabytes of embedded models and working with non-deterministic results which completely violates the promise of zero dependency within 7-9 ms. In fact, I encounter some uncertainty when expanding substrings (e.g. when requesting the code, the open source code will be displayed).
I won't completely abandon semantic search if there is a really easy and fast way to do it locally, but at the moment deterministic search is the priority. And I completely agree with your first comment, the verbatim approach provides just such a verification opportunity.
- Save everything to disk. Index it or store in vectorDB.
- Search the storage for similarity based on the new prompt
- include any finding with the new prompt as system/user prompt
(or if you find the exact answer skip the llm call)
I think everyone's ended up building one of these for themselves. I did too[0]. In the end it's quite easy these days:
* I use the bge-en-base CPU embedding model
* I put storage behind a simple endpoint that has read,write,update,search semantics
* The endpoint just stores markdown in an S3 like structure (bucket-key-value; tree structure is inferred) and vector indexes
* The actual persistence is just SQLite
Most modern models are pretty good at handling this. Our home agents (voice and text) use this to store information and I also have skills for claude code and codex to do that as well. Overall, works quite well.
I still use MEMORY.md for the home agents to keep short-term memory, and use the KB for long-term memory. You do need a self-reflection/consolidation/dream step in order to periodically consolidate and groom the KB but that's also just another job.
The whole setup is pretty trivial and what anyone would come up with from scratch and it's surprising that it works but it does. I'm a pretty good documenter so it helps. In some future perhaps a vision model could see all the things I'm doing and remember, but for now I just tell it and it's pretty good about everything I want.
I wasn't but now, doing a lot of work inside a harness that is logging everything and can easily create docs from that, I'm finally persisting a lot of knowledge.
What did you find wasn't working well? Perhaps I have insufficient content for it to start failing. Works quite well in Claude Code / Codex / Custom Harness with DSV4F.
That's why I ended up creating two forced paths for this.
First, the SessionStart hook aggressively introduces context immediately after loading, completely bypassing the agent's requirements.
Secondly, I'm forcibly changing the behavior by using a fragment of the system query in CLAUDE.md or AGENTS.md. I'm literally just copypaste this block from the README:
Before debugging or re-implementing anything, run deja "<query>" (or the MCP revocation tool) — the agent's past sessions in Claude Code, Codex, and opencode will be indexed locally. Specify what you are reusing.
If I remember well, the starting point was that memory is only useful if it changes the system behaviour. There are many ideas of automatic 'tips' or 'rules' that are injected into the prompts so that the agent acts accordingly to some retained lesson. But there are many failure modes for this.
But maybe I'll need to revise the phrasing on that page and in instructions.
I've settled on writing these bespoke per project/client, typically as a harness plugin in its own repo. Not infrequently that maps 1:1 with a single target repo. There's something to be said for having the tools and memories you need and _not_ having those you don't, and it simplifies the full mental model not to have to handle moving from machine to machine and keeping all that in sync.
Go-to stack for this in my case is to start off with a MEMORY.md, and then keep a long-running agent session going to observe human-operated tmux sessions, surface/respond to occurrences where there's something significant learned (typically the human operator redirecting away from an agent/LLM decision), then start having it integrate into and read out of DuckDB. The latter becomes dense, easily searchable data for agent skills and commands and, yes, memory and such.
How does one make the agent actually save and recall memories and at relatively sensible times? I tried a few memory things and have seen a few in-CLI programmatic approaches but nothing that actually looked useful in real use.
33 comments
[ 3.7 ms ] story [ 35.5 ms ] threadIf it’s fully automated and then blindly injected into your laptop without any vetting, isn’t that a perfect vector to break that separation?
However, when syncing, only data records are moved not instructions. Nothing is done and known secrets are deleted at the indexing stage even before they leave the source computer. In addition, the context here is your own computers and your terminal's output data, if an attacker can control your agent's original decryptions, the source computer has already been completely compromised.
Nevertheless, you make a perfectly reasonable case for strict isolation. I just opened the issue to add an opt-out checkbox where imported sessions are only displayed when manually searched using the CLI/MCP, not when automatically implemented: https://github.com/vshulcz/deja-vu/issues/39
GitHub: https://github.com/open-latch/latch
I felt the same annoyance as OP mentioned, but focused on agents violating project judgement and direction. Obviously we are optimizing for some different things than deja: deja uses no-LLMs and exact verbatim transcript retrieval.
Latch uses LLMs, ranking, etc, to emphasize strong judgment, correction, and enforcement over dynamic project flow. Give latch a try if you're also running into decision amnesia in your workflow.
I'd lead with your differentiation. Is it the ssh?
This causes a very similar itch, but we went a slightly different route regarding some implementation details
Secrets: The ctx README mentions saving the text as is and warns about the need to review the output data before publishing. deja explicitly removes known secret templates at the indexing stage (replacing them with tags like [redacted:aws-key]).
Syncing: It looks like ctx is getting ready for a cloud beta to be shared by the team. Actually, I just wanted something local and free that I had full control over, so deja does P2P syncing through your existing SSH settings.
Recall that deja has a session startup feature that automatically introduces a bit of context when a new agent is loaded, instead of relying solely on manual search.
Both applications solve the same annoying problem by simply optimizing for different workflows :)
> Secrets: The ctx README mentions saving the text as is and warns about the need to review the output data before publishing. deja explicitly removes known secret templates at the indexing stage (replacing them with tags like [redacted:aws-key]).
How is this not security theater? You've presumably already sent the api key to an LLM if it's in your session file. You need to rotate it.
What am I missing if I instead just instruct the agent to create a handful of files called something like HISTORY.md and tell it to log summarized versions of all of these things inside the project repo whenever something is changed or evaluated? (Which is very close to what I currently do.)
An orphan branch solves that issue.
Adding semantic search usually means sending hundreds of megabytes of embedded models and working with non-deterministic results which completely violates the promise of zero dependency within 7-9 ms. In fact, I encounter some uncertainty when expanding substrings (e.g. when requesting the code, the open source code will be displayed).
I won't completely abandon semantic search if there is a really easy and fast way to do it locally, but at the moment deterministic search is the priority. And I completely agree with your first comment, the verbatim approach provides just such a verification opportunity.
- Save everything to disk. Index it or store in vectorDB. - Search the storage for similarity based on the new prompt - include any finding with the new prompt as system/user prompt (or if you find the exact answer skip the llm call)
?
Or is there more to it ?
* I use the bge-en-base CPU embedding model
* I put storage behind a simple endpoint that has read,write,update,search semantics
* The endpoint just stores markdown in an S3 like structure (bucket-key-value; tree structure is inferred) and vector indexes
* The actual persistence is just SQLite
Most modern models are pretty good at handling this. Our home agents (voice and text) use this to store information and I also have skills for claude code and codex to do that as well. Overall, works quite well.
I still use MEMORY.md for the home agents to keep short-term memory, and use the KB for long-term memory. You do need a self-reflection/consolidation/dream step in order to periodically consolidate and groom the KB but that's also just another job.
The whole setup is pretty trivial and what anyone would come up with from scratch and it's surprising that it works but it does. I'm a pretty good documenter so it helps. In some future perhaps a vision model could see all the things I'm doing and remember, but for now I just tell it and it's pretty good about everything I want.
0: https://wiki.roshangeorge.dev/w/Blog/2026-07-01/The_Everythi...
I wasn't but now, doing a lot of work inside a harness that is logging everything and can easily create docs from that, I'm finally persisting a lot of knowledge.
First, the SessionStart hook aggressively introduces context immediately after loading, completely bypassing the agent's requirements.
Secondly, I'm forcibly changing the behavior by using a fragment of the system query in CLAUDE.md or AGENTS.md. I'm literally just copypaste this block from the README:
There are comparisons and stuff: https://zby.github.io/commonplace/agent-memory-systems/
Deja-vu is here: https://zby.github.io/commonplace/agent-memory-systems/revie...
And the full list: https://zby.github.io/commonplace/agent-memory-systems/revie...
Go-to stack for this in my case is to start off with a MEMORY.md, and then keep a long-running agent session going to observe human-operated tmux sessions, surface/respond to occurrences where there's something significant learned (typically the human operator redirecting away from an agent/LLM decision), then start having it integrate into and read out of DuckDB. The latter becomes dense, easily searchable data for agent skills and commands and, yes, memory and such.