We really need a "memory arena" to serve two important purposes:
1. List all the known agent memory projects (of which there are hundreds)\
2. Objectively compare and score them both against each other and vanilla harnesses like Claude Code
Only then can I have the cognitive capacity to decide which one makes sense for me.
Agreed, and point number two is the tricky one. Creating a list of tasks is easy; evaluating them is not. You need a consistent task set, a "clean slate" control (i.e., Claude code without memory is your proper control) and an evaluation criteria which differentiates "uses fewer tokens" from "produces better results," otherwise you end up with vendors evaluating their own work.
Currently constructing a repeatable test harness for PMB: Fixed task, with/without memory, repeated N times, giving number of tokens/turns/passed/not passed with a subjective quality score too. Would be happy to share the task set and evaluation criteria for testing on anyone else's memory server or clean slate control, not just mine.
Proper Framing - AGENTS.md is fixed and always contextual irrespective of its relevance; memory is the exact opposite of it. PMB does not dump the store in, rather extracts only a small top-k relevant chunk per task (a few hundred tokens generally), hence, more storage in the form of facts does not necessarily translate to more context. "Useless facts" are precisely what it is grading against, i.e., items which do not serve any purpose will simply be considered dead and decayed.
this looks very interesting to me and I plan to try it. I was looking at the web site (pmbai.dev) earlier today and it looked very thorough but when I went to show someone this afternoon it no longer loads in my browser, saying it can't establish a secure connection. Verified by clicking the website link on the repo page on github so I am pretty sure I have the url right...
35ms local aint a brag (I am suprised it is not < 1ms naturally so maybe check your bench), but arent there tonnes of agent memory options so why is this different from those?
12 comments
[ 0.17 ms ] story [ 27.3 ms ] thread1. List all the known agent memory projects (of which there are hundreds)\ 2. Objectively compare and score them both against each other and vanilla harnesses like Claude Code
Only then can I have the cognitive capacity to decide which one makes sense for me.
Currently constructing a repeatable test harness for PMB: Fixed task, with/without memory, repeated N times, giving number of tokens/turns/passed/not passed with a subjective quality score too. Would be happy to share the task set and evaluation criteria for testing on anyone else's memory server or clean slate control, not just mine.
My main concern is that it can overwhelm the context window with useless facts.