I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
The memory invalidation based on tying memories to pieces of code (e.g. functions) is the big thing here, I think, though I see why it wouldn't be front and center.
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.
22 comments
[ 3.4 ms ] story [ 41.6 ms ] threadLetting another LLM decide quality probably isn't the only meterstick you want, either.
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
For the evals I've run (albeit, not super scientific), these memory systems did worse than the raw harness given ripgrep.
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.