I see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself.
What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.
Coding agents don't really need memory. Agent skills, rules, git history, documentation is all far more efficient, transparent and easier to manage. These memory frameworks only really makes sense if you are building a consumer facing agent with managed context and limited capabilities.
There’s probably never going to be one answer. The most fascinating thing about this quest for memory is that it’s a Rorschach test. Exploring the myriad attempts to implement memory shows that everyone has a slightly different itch they’re trying to scratch, but we talk about it like we all want the same thing.
The obvious energy saving step would be to utilise previous searches by others. Many of the tasks people do are rather similar, it is such an energy waste to start again each time.
(Obviously ignoring the huge energy saver, which is to observe if you even need to bother doing the task at all.)
> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning
This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching.
This paper might do a thing or two approximating the compression limit for context windows, but there’s a fundamental limit on how much information can go into it.
What you really need is contextual search, as in, different events and objects with the same abstractions and semantic lead to same response, so you can cache effectively… on this front the paper does little to improve “memory” in a meaningful way
I wrote something about it trying to look other way around the context or memory data in models. The gravitational pull of information stills very hard to manage. Ive been using "functional scars" about 30 days now and getting good results in repetitive mistakes across sesions. https://github.com/VDP89/fscars
While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high.
A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a reasonable estimate), then the state can encode 100M tokens worth of information.
Real models obviously aren't powerful enough to operate at the limit, but you can see why this is a promising research direction.
Papers being voted high on Hacker News are usually uncorrelated with their actual importance. It's basically a lottery. There are regularly more interesting papers going semi viral on Twitter.
Hmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.
I would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above.
Essentially, many people want to know what the minimum amount of memory is to run a particular model.
Parameter count obscures important details: what are the sizes of the parameters? A parameter isn't rigorously defined. This also gets folks into trouble because a 4B param model with FP16 params is very different from a 4B param model with INT4 params. The former obviously should be a LOT better than the second.
This would also help with MOE models: if memory is my constraint, it doesn't matter if the (much larger RAM required) MOE version is faster or has better evals.
I'm waiting for someone in anger to ship the 1 parameter model where the parameter according to pytorch is a single parameter of size 4GB.
Is there some kind of memory enabling, for instance, an agent to remember guidelines on a repo without having to feed at the beginning of each session 4 markdown files and spending the corresponding tokens each time ?
The future is fixed size state with a massive token history that the model can look back at like reading a journal. A reframing of the model this way opens a new kind of agent, one with essentially unlimited context, that packs perfectly on a GPU, can be stored/retrieved fairly effortlessly and can essentially be run forever. Fixed size means theta 1 tokens. A model that can look around also means essentially unlimited memory can be bolted on with the model learning to look around memory like it is looking around at the journal of past tokens. Guided windows of attn can do most of this, some other tricks can do the rest.
23 comments
[ 3.3 ms ] story [ 43.9 ms ] threadWhat I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.
(Obviously ignoring the huge energy saver, which is to observe if you even need to bother doing the task at all.)
- fixed size of the memory seems like a good idea to overcome the current limitations
- skimming through the thing, I can't find any mention of the cost?
- I would need more time to read it in-depth to see if this is legitimate and not just fancy form of overfitting or training on testing data
This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching. This paper might do a thing or two approximating the compression limit for context windows, but there’s a fundamental limit on how much information can go into it. What you really need is contextual search, as in, different events and objects with the same abstractions and semantic lead to same response, so you can cache effectively… on this front the paper does little to improve “memory” in a meaningful way
A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a reasonable estimate), then the state can encode 100M tokens worth of information.
Real models obviously aren't powerful enough to operate at the limit, but you can see why this is a promising research direction.
Is it a lowercase to uppercase conversion going on here?
Essentially, many people want to know what the minimum amount of memory is to run a particular model.
Parameter count obscures important details: what are the sizes of the parameters? A parameter isn't rigorously defined. This also gets folks into trouble because a 4B param model with FP16 params is very different from a 4B param model with INT4 params. The former obviously should be a LOT better than the second.
This would also help with MOE models: if memory is my constraint, it doesn't matter if the (much larger RAM required) MOE version is faster or has better evals.
I'm waiting for someone in anger to ship the 1 parameter model where the parameter according to pytorch is a single parameter of size 4GB.
Nothing super novel or groundbreaking, but a moderately interesting read.