One interesting quirk of the AI-written READMEs these days is how they can include every detail on how it works, thoroughly document every optional flag, known limitation, experimental result, and still not communicate the essence of the project and the problem it solves.
I have read through the project and I still don't understand what this thing is for and why it is to be preferred over the harness's native memory management tools.
I once told the agent explicitly not to write like it's trying to impersonate Hemingway and it started writing like a normal human being. It's surprising how a writing style that was once revolutionary is now a hallmark of sloppyness.
Maybe give it a try next time you write a readme with agents. That and giving it an example of good README in real world repos can increase dramatically the likelihood of synthesizing a serviceable README.
AI readme should be a starting point. I typically remove at least 50% of the details (without any particular skill use, it goes into extremes like "x clears the edit" and writes a related wall of text in the middle of the important explanation).
My pet theory is that while the pretraining -> RL pipeline achieves very impressive results, it does not reward clarity of thought or elegance. It's not obvious whether it even should for most tasks, but it does grind on me as a human who needs elegance in order to keep everything under control. You give astra/codex many tasks, it retires them all more efficiently than I could by hand. But you look under the hood and every bugfix is another codepath, it just hammers away at things with admirable persistence and vigor until the tests pass. Similarly in discussions and docs, I've noticed many LLMs like to "beat around the bush."
it does not reward clarity of thought or elegance.
Well duh. RL can only train behaviors that can be defined. Clarity and elegance are damn subjective.
Also, I doubt we'll ever get AI to understand what clarity is to a human. They have such enormous contexts that what's clear to them is not clear to us.
RL works well for human preference and any other subjective goal. In fact Claude used to be trained for human likeness, reading between the lines, character, and understanding the intent. Claude 3 was outstanding in that. It was their entire marketing shtick. Then they reversed the direction starting with Sonnet 3.5, ignored it completely in favor of code since Claude 4.0, and new models completely lost their ability to write and understand humans. Code pays better.
The AI language really stands out, too. "What is automatic and what depends on the agent". A human might write, "jevmen watches your coding session with Claude Code or Codex and picks just the right moment to remember important things that you decided along the way. There are some differences in how jevmem works, depending on which coding harness you are using. The table below summarizes these differences:"
I don't know why the models were generally trained to be so brief, but it's definitely not the way anyone I know actually writes. A second pass is always a good idea to clean this stuff up. And, thankfully, the models are all pretty good at that.
The big tell for me is assuming the reader has context straight out the gate. "What is automatic and what depends on the agent" assumes the reader, like the LLM at the time it wrote that sentence, has all of this pre-context available. That sentence makes more sense as an h3 header further down in some detailed list where the human has all of the knowledge of how the system works. Whereas a human obviously does not, so they lead with the important context of why you give a shit
It’s funny to me because pre-AI it’s something I something I saw all the time managing interns and junior software engineers when I’d start trying to tech them about writing design docs. You can explain the process to them and give them a million examples of what good looks like, but for a lot of them the first few docs they write will just be this weird combo of way too much information and very little of it being useful.
My belief has always been that they see the document as more like a test that’s a single task and not just one piece in a larger process and so they treat it like a test where there’s a right answer. They know they don’t really understand the question being asked though so they default to a mindset of, “Well if I just put everything in there some of it has to contain the correct answer” so you end up with this document that’s full of “what”s and “how”s, but completely void of “why”s.
You’ll also often see them fill up space answering easy questions that match the structure of things that are in other documents instead of focusing on the actual hard problems in the design because the hard problems are often unique and their answers may not fit the existing patterns in the examples. There isn’t the instinct to go, “Yeah, none of these example documents talk about the servers were going to deploy it on, but this has to be deployed in an EU cluster because of GDPR laws so I need to add that” because they’re mostly just pattern matching at first.
Usually once they’ve experienced the whole process first hand it starts to click because they start to understand where a design document fits into the larger process so they get a feel for what information matters and what doesn’t.
I think when people just tell an agent to create a README you have the same problem because both the human and the agent see it as just a checkbox type task. The agent sees it as an isolated task snd has no fucking idea how the README is going to be used and the human isn’t giving them that context so they just spit out a bunch of stuff that’s factual and fits the patterns it knows, but is fairly useless.
deciduous seems like a great way to ensure there is no space left in context for useful work.
Problem with all these automatic memory managment tools is that they can't reliably classify if something is useful or not. Once it start using some tool it will heavily become biased towards using that tool during that session. First you will have decision to use postgres as a database, by the end of session it will start saving "decided to use SELECT query for this method because that is how you pull data out of database".
> When you change your mind, the old line is marked superseded, not deleted
To me, this seems like a design error. You're polluting context with false/outdated information (even if the LLM is instructed to ignore it). The biggest issue with the memory systems built into Claude et al. is that they're terrible at pruning old/conflicting information as the project evolves, so I'd hope a replacement would do something to improve that.
Keeping history and using history are two different things though
You may want the old decision in the audit trail so you know why something changed without putting that old decision back into the model context every time
I think memory systems need a pretty strict separation between active memory and historical memory
I’m looking at the LLM-generated SECURITY.md and this thing seems to pump a LOT of information back to some place called TypeSafe AI. That and the AI-generated comments from OP here make a few red flags go up for me.
Just start a cheap chinese agent in a context where it has only one tool, and JSON-schema constrain that toolcall to the response shape you want. Prompt the agent to make only one tool call and not speak. Done.
The people that run local jev like models do sth similar. Train a seperate nn adapter on top of qwen 4b so it has balanced probabilities whatever that means and apparently it works quite well.
Side note: conspiracy theorists say that jev is a qwen model fine tuned but who know if true
46 comments
[ 0.26 ms ] story [ 16.1 ms ] threadI have read through the project and I still don't understand what this thing is for and why it is to be preferred over the harness's native memory management tools.
Maybe give it a try next time you write a readme with agents. That and giving it an example of good README in real world repos can increase dramatically the likelihood of synthesizing a serviceable README.
Well duh. RL can only train behaviors that can be defined. Clarity and elegance are damn subjective.
Also, I doubt we'll ever get AI to understand what clarity is to a human. They have such enormous contexts that what's clear to them is not clear to us.
I don't know why the models were generally trained to be so brief, but it's definitely not the way anyone I know actually writes. A second pass is always a good idea to clean this stuff up. And, thankfully, the models are all pretty good at that.
My belief has always been that they see the document as more like a test that’s a single task and not just one piece in a larger process and so they treat it like a test where there’s a right answer. They know they don’t really understand the question being asked though so they default to a mindset of, “Well if I just put everything in there some of it has to contain the correct answer” so you end up with this document that’s full of “what”s and “how”s, but completely void of “why”s.
You’ll also often see them fill up space answering easy questions that match the structure of things that are in other documents instead of focusing on the actual hard problems in the design because the hard problems are often unique and their answers may not fit the existing patterns in the examples. There isn’t the instinct to go, “Yeah, none of these example documents talk about the servers were going to deploy it on, but this has to be deployed in an EU cluster because of GDPR laws so I need to add that” because they’re mostly just pattern matching at first.
Usually once they’ve experienced the whole process first hand it starts to click because they start to understand where a design document fits into the larger process so they get a feel for what information matters and what doesn’t.
I think when people just tell an agent to create a README you have the same problem because both the human and the agent see it as just a checkbox type task. The agent sees it as an isolated task snd has no fucking idea how the README is going to be used and the human isn’t giving them that context so they just spit out a bunch of stuff that’s factual and fits the patterns it knows, but is fairly useless.
I will have to give it a run-through today.
I haven't used Jev yet so this should be interesting. I'd be interested to see if any Deciduous users have opinions, too.
Problem with all these automatic memory managment tools is that they can't reliably classify if something is useful or not. Once it start using some tool it will heavily become biased towards using that tool during that session. First you will have decision to use postgres as a database, by the end of session it will start saving "decided to use SELECT query for this method because that is how you pull data out of database".
To me, this seems like a design error. You're polluting context with false/outdated information (even if the LLM is instructed to ignore it). The biggest issue with the memory systems built into Claude et al. is that they're terrible at pruning old/conflicting information as the project evolves, so I'd hope a replacement would do something to improve that.
You may want the old decision in the audit trail so you know why something changed without putting that old decision back into the model context every time
I think memory systems need a pretty strict separation between active memory and historical memory
Just start a cheap chinese agent in a context where it has only one tool, and JSON-schema constrain that toolcall to the response shape you want. Prompt the agent to make only one tool call and not speak. Done.
Side note: conspiracy theorists say that jev is a qwen model fine tuned but who know if true