Hi Guys,
like many of you I work with AI agents every day and across several clients.
The major time cost for me is to re-explaining context and past decisions, client rules, that we already do. To finish every tools as its own memory.
So I’m curious how you actually handle this:
1. When you start a new session, how do you give context to the agent (context came from chat, mail, ticket ...) ?
2. Do you use more than one agent type (claude, codex ...) ? And if yes, how do you carry context between agents ?
3. If you work for several clients, how do you keep their context separate ?
4. The last time something went wrong ( an agent used a rejected approach, forgot a decision ...), what happened, and how much time did it cost to you ?
5. Did you paid or built a tool that helped you with these problems ? Do you still use it ?
I’m exploring whether to build something, so I’m trying to learn how other people really work.
Thanks for your return.
I use a gitignored ADR (Architecture Decision Record) directory in all of my repos right now. I mainly use Claude code, but I am dabbling in self hosted llms with oh my pi.
There's no additional tools needed because usually llms "get it" natively.
Also I tend to use a gitignored status doc for resuming where I left off.
I tend to leave this stuff gitignored since it's all pretty "sloppy" and not really for consumption with others.
Spec driven design helps, OpenSpec/Git SDD Kit/ GitShipDone helps with the previous decision problems and design context. Most of these live in the project repos so they won’t pollute each other
re: #1 I have a skill that essentially manages a daily board in json format and pulls in active or potential tasks from all my work streams (email, slack, grafana, ado, or even manual input). I have this management plane keep a journal for each day and then spin off sub-sessions to work individual items. I pretty much just run the thing on a loop, and since context is stored with the board itself there’s no issue closing down and restarting the session or clearing context occasionally.
I let agents read their own pre-compaction history often, as well as history form other sessions. I use both Codex and Claude Code. The main friction right now is keeping an overview of the sessions. Sessions don't have a max length for me. I run them as long as I want, but I keep session scope somewhat limited, usually to one discipline (e.g. software, CAD, electrical, sales, website). But I'll switch to the GUI apps soon, because they seem to have reached parity over the terrible CLIs that I use.
Use my Johnny.Decimal system. [0] (Disclosure: I'm Johnny.)
Every part of my life and business has a well-defined ID. This ID has:
- A defining entry in my index [1], which is an Obsidian vault.
- Therefore, also a Markdown note(s).
- A folder on disk.
I have a CLI [2] and agent skills [3] that help your LLM understand these relationships.
Then you just:
jd 21.15 // or search by title
claude
and now Claude has access to:
- All of your notes from your index. This is your shared memory: it's designed to be human-readable.
- All of its memory and transcripts from the fact that you launch it in the same folder whenever you do this piece of work. This is its internal memory.
- With this comes the ability to easily `--resume` past conversations and so on.
That's … it. Carefully-named Markdown files and folders. My numbers aren't a prerequisite – there's nothing magical about them – but they sure help.
You can do this all for free. It takes about half an hour to set up. My MCP server [4] adds some quality-of-life but is paid (and means I can buy lunch :).
> I’m exploring whether to build something
I really wouldn't. I get 10 emails a week from people who already are and I don't see the point. It's a solved problem.
I recently experimented with breaking my life into segments (like the reporting segments of a company) and organizing Dropbox, text journals, and Claude chats in accordance with those segments. What you are describing is far more advanced and I'm intrigued. Checking out the site now.
I have a small gatekeeper.py script the agent has to run to complete a round or begin a new one. One of the things it does (in addition to record keeping, linting, etc) is validate that the wiki has been updated with all the findings, lessons learned, and documentation for the round. The agent can't proceed until it's done.
Then in the AGENTS.md they're told they can use QMD to perform semantic search of the wiki , and should before starting any significant work.
It's expensive, in terms of tokens, but well worth it. I could reproduce any stage of my project if I wanted. I always know why they did what they did (if they do), and the agent's can't really forget anything important.
I’ve been trying to get something similar to work with local markdowns instead. Getting them to write something is easy. But I cannot reliably trigger them to write when they make “sufficiently important” decisions. There is little self reflection (or actual intelligence).
How do you get them to trigger writing to the log in the right moment and manner?
The karpathy wiki is really just local markdown at its core. The semantic search qmd provides is really the fancy bit.
My system is fairly elaborate, but the simple trick is to build the loop so they have mechanical checks that prevent anything from counting as done until it meets standards. There are many ways to do that. A lot of folks just hook specific tool calls and make it output relèvent instructions.
In my loop I use an Orchestrator and a Builder sub agent. The Orchestrator assigns the Builder work from the spec one phase at a time. The Builder has to build each phase with red then green unit tests as specified by the Orchestrator. The Builder sub agent only knows that one task. It can't consider the task finished until gatekeeper passes.
Gatekeeper is a simple python script that the Orchestrator is allowed to update whenever it sees an anti-pattern. It validates that all the unit tests pass, that it HAS sufficient test coverage, that it lints properly (including custom rules the Orchestrator can add for repeated failure), that the docs exist for that phase and are long enough, and finally does some ast grep searches for common goofs, etc.
Sometimes the builder will have to try many times before it passes, but each time it fails gatekeeper outputs an instruction telling them which rule they violated and the full requirements.
Eventually gatekeeper will pass and the Orchestrator does a final review before assigning the next phase to a new Builder. The most common failure at this point is poor unit tests that don't test for the right thing. The Orchestrator has explicit instructions to check for that and does catch it most of the time.
so i built https://estratos.ai (5.) to ensure context from one client never pollutes a different client (or project, etc.) happy to explain the architecture and save you the sign up. it's set up as a layer-cake, you get layers like `root`, `agent`, `client`, `project`.
1./2. tell your agent to read/store its context from eg. `root.agent_name.client-a.project-a1`
2./3. next client is setup as `root.agent_name.client-b.special-project-b1`
3. when the agent loads the memory from there, it concatenates all the instructions from root, agent, client, project.
3. isolation is enforced by estratos so the llm doesn't get to know that there's a client-a subtree next to the client-b subtree
this works better for me instead of the knowledge graphs, etc. because of the enforced isolation and the domain that it targets (accounting firms, where sharing context from one client to the next is deadly).
I built a tool for myself for this. It runs on localhost and uses sqlite as a storage engine for metadata, and the XDG paradigm for content (eg ~/.local/share, ~/.local/state) I create topics, eg localhost:8080/topic/1. Each topic can have tickets, artifacts and sessions. Tickets are linked task tracking tickets (eg Jira, though not actually Jira in my case), artifacts are documents etc produced during sessions, and sessions are Claude Code sessions (though it wouldn't have to be Claude, that's just what I use).
Sessions run in a terminal emulator in the browser. What makes this work is that I have some general purpose directions/context each topic gets, but there's also a running topic.md that gets updated at the end of each session. This way my context builds up and persists specific to that topic. Artifacts the agent produces (eg specs, plans) open up in a sidebar and render as HTML even though they're stored as markdown, and its easy to go back and find them from previous sessions. Session transcripts also are accessible from the Topic page.
I do regularly run compaction on the topic.md, moving the full content to an archive and keeping the working context smaller.
When I start a new project, I point to other ones where I'm happy with the quality as exemplars. I think its probably simpler to copy relevant stuff allow older projects go out of date until I start working on them again, then I tell the agent to look at the newer project to update its style.
I have 2-4 on going projects I tend to a bit of that, normally 2 at a time, when I pick up one of the others, I really just do that; point to the older one.
But within projects I don't really try to manage context, more so I just have an issue tracker which I add to keep track of stuff. In 2 of my long running projects I have some bootleg issue tracker built in (along with a screenshot QA tool which sends issues to the tracker) into the UI of the tool.
I guess to extent I can, I try to avoid keeping some preferences in the context like formatting and linting (including directory structure and white listing imports in certain parts). I've been trying to find more ways to automate that way, so I keep my prompts and skills and goals limited to objectives where possible, so preferences like styles and structure are automatically enforced.
When I have multiple agents, I just tell them to maintain write only ledgers to coordinate to avoid overlapping work, but it's easier when your project is broken up into self contained components with communication boundaries (defined in protobuf) like a FE/BE split or seperate service codebases (I reassemble it as a google style modular monolith to avoid running too many processes)
It seems many have built similar tools for solving this.
Mine lets me run Claude and Codex (and pi but then you have to pay API rates) from the same localhosted webserver, and also schedule feature implementations to always hit the weekly usage cap. chats logs in jsonl (used by both) are saved per project in a never-pushed git branch. All agents run in Podman containers.
I am just starting to play with a higher-level agent that just pushes the repo in a direction based on the human answering a thorough list of clarification questions.
These have been really useful
- Encourage short, commit-linked chats, to avoid the expensive cache writes.
- Plot of LOC over time. Use better models until this starts not spiking a ton with every new feature.
- Terminals kinda suck for agentic stuff IMO and we should stop cramming agent loops into them. Minimalism doesn't really help here, it's really nice to have built-in STT, real buttons, easy scrolling, etc etc etc.
- The minimal delta is a commit. Every implementation turn must end with one or multiple, and a clean worktree.
- Crystal clear, extremely granular usage data. This is something Claude Code and Codex will never give you, instead you get dark patterns nudging you to forget about the spend, while changing plan quotas every couple weeks.
Here is a good suggestion: try using OKF (Open Knowledge Format) to store your context. Log every transaction in the OKF format as you process tasks. Even if you change your AI, simply reading the OKF file will do the trick.
As many have said - my strong steer build something[^1].
There are _great_ tools for this, but the biggest part of this is managing your own attention, and mental load. You can craft something that really helps _you_ manage all the spinning plates.
In the case of my harness, I hoover up all the PR state into one place and then have a bunch of rules on how I interact with them (or not). Even down to a "swipe right to merge" card system. Works for me.
My recommendation: A cloudflare PWA fully vibe coded with zerotrust on the front. Don't get too attached, tools are changing fast. Make it work and be prepared to ditch it when something better comes along.
I keep my agents strongly sandboxed from each other using byre[0] (of which I am the author). I then use chezmoi[1] to manage my byre config -- including instructions for every box -- across multiple physical machines.
75 comments
[ 3.0 ms ] story [ 35.9 ms ] threadSo I’m curious how you actually handle this:
1. When you start a new session, how do you give context to the agent (context came from chat, mail, ticket ...) ?
2. Do you use more than one agent type (claude, codex ...) ? And if yes, how do you carry context between agents ?
3. If you work for several clients, how do you keep their context separate ?
4. The last time something went wrong ( an agent used a rejected approach, forgot a decision ...), what happened, and how much time did it cost to you ?
5. Did you paid or built a tool that helped you with these problems ? Do you still use it ?
I’m exploring whether to build something, so I’m trying to learn how other people really work. Thanks for your return.
There's no additional tools needed because usually llms "get it" natively.
Also I tend to use a gitignored status doc for resuming where I left off.
I tend to leave this stuff gitignored since it's all pretty "sloppy" and not really for consumption with others.
1. https://github.com/mattpocock/skills/blob/main/README.md
Agent continuously references and updates it.
Every part of my life and business has a well-defined ID. This ID has:
- A defining entry in my index [1], which is an Obsidian vault.
- Therefore, also a Markdown note(s).
- A folder on disk.
I have a CLI [2] and agent skills [3] that help your LLM understand these relationships.
Then you just:
and now Claude has access to:- All of your notes from your index. This is your shared memory: it's designed to be human-readable.
- All of its memory and transcripts from the fact that you launch it in the same folder whenever you do this piece of work. This is its internal memory.
- With this comes the ability to easily `--resume` past conversations and so on.
That's … it. Carefully-named Markdown files and folders. My numbers aren't a prerequisite – there's nothing magical about them – but they sure help.
You can do this all for free. It takes about half an hour to set up. My MCP server [4] adds some quality-of-life but is paid (and means I can buy lunch :).
> I’m exploring whether to build something
I really wouldn't. I get 10 emails a week from people who already are and I don't see the point. It's a solved problem.
[0]: https://johnnydecimal.com/
[1]: https://johnnydecimal.com/documentation/the-jdex
[2]: https://johnnydecimal.com/documentation/jd-cli
[3]: https://johnnydecimal.com/documentation/agent-skills
[4]: https://johnnydecimal.com/software/mcp-server
I have a small gatekeeper.py script the agent has to run to complete a round or begin a new one. One of the things it does (in addition to record keeping, linting, etc) is validate that the wiki has been updated with all the findings, lessons learned, and documentation for the round. The agent can't proceed until it's done.
Then in the AGENTS.md they're told they can use QMD to perform semantic search of the wiki , and should before starting any significant work.
It's expensive, in terms of tokens, but well worth it. I could reproduce any stage of my project if I wanted. I always know why they did what they did (if they do), and the agent's can't really forget anything important.
How do you get them to trigger writing to the log in the right moment and manner?
My system is fairly elaborate, but the simple trick is to build the loop so they have mechanical checks that prevent anything from counting as done until it meets standards. There are many ways to do that. A lot of folks just hook specific tool calls and make it output relèvent instructions.
In my loop I use an Orchestrator and a Builder sub agent. The Orchestrator assigns the Builder work from the spec one phase at a time. The Builder has to build each phase with red then green unit tests as specified by the Orchestrator. The Builder sub agent only knows that one task. It can't consider the task finished until gatekeeper passes.
Gatekeeper is a simple python script that the Orchestrator is allowed to update whenever it sees an anti-pattern. It validates that all the unit tests pass, that it HAS sufficient test coverage, that it lints properly (including custom rules the Orchestrator can add for repeated failure), that the docs exist for that phase and are long enough, and finally does some ast grep searches for common goofs, etc.
Sometimes the builder will have to try many times before it passes, but each time it fails gatekeeper outputs an instruction telling them which rule they violated and the full requirements.
Eventually gatekeeper will pass and the Orchestrator does a final review before assigning the next phase to a new Builder. The most common failure at this point is poor unit tests that don't test for the right thing. The Orchestrator has explicit instructions to check for that and does catch it most of the time.
1./2. tell your agent to read/store its context from eg. `root.agent_name.client-a.project-a1` 2./3. next client is setup as `root.agent_name.client-b.special-project-b1` 3. when the agent loads the memory from there, it concatenates all the instructions from root, agent, client, project. 3. isolation is enforced by estratos so the llm doesn't get to know that there's a client-a subtree next to the client-b subtree
this works better for me instead of the knowledge graphs, etc. because of the enforced isolation and the domain that it targets (accounting firms, where sharing context from one client to the next is deadly).
Sessions run in a terminal emulator in the browser. What makes this work is that I have some general purpose directions/context each topic gets, but there's also a running topic.md that gets updated at the end of each session. This way my context builds up and persists specific to that topic. Artifacts the agent produces (eg specs, plans) open up in a sidebar and render as HTML even though they're stored as markdown, and its easy to go back and find them from previous sessions. Session transcripts also are accessible from the Topic page.
I do regularly run compaction on the topic.md, moving the full content to an archive and keeping the working context smaller.
I have 2-4 on going projects I tend to a bit of that, normally 2 at a time, when I pick up one of the others, I really just do that; point to the older one.
But within projects I don't really try to manage context, more so I just have an issue tracker which I add to keep track of stuff. In 2 of my long running projects I have some bootleg issue tracker built in (along with a screenshot QA tool which sends issues to the tracker) into the UI of the tool.
I guess to extent I can, I try to avoid keeping some preferences in the context like formatting and linting (including directory structure and white listing imports in certain parts). I've been trying to find more ways to automate that way, so I keep my prompts and skills and goals limited to objectives where possible, so preferences like styles and structure are automatically enforced.
When I have multiple agents, I just tell them to maintain write only ledgers to coordinate to avoid overlapping work, but it's easier when your project is broken up into self contained components with communication boundaries (defined in protobuf) like a FE/BE split or seperate service codebases (I reassemble it as a google style modular monolith to avoid running too many processes)
Mine lets me run Claude and Codex (and pi but then you have to pay API rates) from the same localhosted webserver, and also schedule feature implementations to always hit the weekly usage cap. chats logs in jsonl (used by both) are saved per project in a never-pushed git branch. All agents run in Podman containers.
I am just starting to play with a higher-level agent that just pushes the repo in a direction based on the human answering a thorough list of clarification questions.
These have been really useful
- Encourage short, commit-linked chats, to avoid the expensive cache writes.
- Plot of LOC over time. Use better models until this starts not spiking a ton with every new feature.
- Terminals kinda suck for agentic stuff IMO and we should stop cramming agent loops into them. Minimalism doesn't really help here, it's really nice to have built-in STT, real buttons, easy scrolling, etc etc etc.
- The minimal delta is a commit. Every implementation turn must end with one or multiple, and a clean worktree.
- Crystal clear, extremely granular usage data. This is something Claude Code and Codex will never give you, instead you get dark patterns nudging you to forget about the spend, while changing plan quotas every couple weeks.
There are _great_ tools for this, but the biggest part of this is managing your own attention, and mental load. You can craft something that really helps _you_ manage all the spinning plates.
In the case of my harness, I hoover up all the PR state into one place and then have a bunch of rules on how I interact with them (or not). Even down to a "swipe right to merge" card system. Works for me.
My recommendation: A cloudflare PWA fully vibe coded with zerotrust on the front. Don't get too attached, tools are changing fast. Make it work and be prepared to ditch it when something better comes along.
[1] https://jonathannen.com/build-your-own-lightsaber/
0: https://github.com/pjlsergeant/byre
1: https://www.chezmoi.io/
This is what agents.md / claude.md is for, its the first file Claude Code and Codex read.
Bonus: You can just ask Claude / GPT for tips on how to use these things more effectively.