Oh interesting, from what I understand then, this is great for small context size models, compared to RAG? Is there research into how to make this more effective for large context size models (since context size of major models seems to be 4xing every 6 months at this point)?
It appears that RAG actually dominates for 2k context lengths compared to this method, but that this method outperforms it more and more the longer the context gets (see the graph titled "Retrieval Benchmark Results, by Document Length")
"Document length" is the length of the text that contains the answer. "Context length" is how much text the model can process to produce the answer, and this number is fixed across their experiments.
When the document length is 2k, it's likely smaller than the context and RAG can just retrieve the entire document to have the model read it. When the document is longer, RAG needs to actually do some work to pick the parts that contain the answer.
The "extended mind" can always query tokens across the entire document, though evidently worse than if they were included in the context.
With almost all of these papers, "RAG" is mentioned as a well-defined, absolute strategy. Is there an agreed upon implementation for it? Because as I have been building my own implementation, I have found that finding the right things to retrieve and augment the prompt with is incredibly challenging.
It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the student a note with the answer right before the exam, compared to letting them read the whole curriculum, but I am very much still learning..
I think it's still a developing technique. I'm not following RAG super closely, but I keep myself somewhat updated through Sam Witteveen's videos on YouTube: https://www.youtube.com/@samwitteveenai
On the note of youtube channels, I can wholeheartedly recommend Prompt Engineering [1]. He has great videos and hands-on tutorials, not just about RAG, and also maintains the LocalGPT repo [2] with 17k stars.
The blog post format is so much nicer than a PDF paper!
More seriously, this does feel like a real advance. Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. I've been wondering for the last year whether it's possible to extend the attention mechanism to more naturally connect to a bigger set of per-session weights or activations. The moment you encounter the query/key/value analogy it's an obvious idea, the problem being that you need a very strong grip on the low level details of neural architecture to actually do it. Now apparently it is possible! And the way the topk knob actually maps to abstraction is quite amazing.
Still, this doesn't eliminate context window constraints. The memories themselves have a form of context window in this technique. Context size (what they call sequence length) does still matter.
Additionally, the memories have to actually fit in GPU memory (at least in their implementation). And the memories appear to be nearly full snapshots of the network, so they will get quite large (much larger than text grabbed using RAG). So there's going to be a painful tradeoff here for the forseeable future where you'll have to decide whether you want a bigger smarter base model with a bigger context window but less space for memory, or a smaller model with a smaller context window but bigger memory.
This is the first I've heard of Normal Computing, who are these guys/gals exactly?
> Normal is a deep-tech startup founded by former Google Brain & X engineers
Ah. That explains it. "X" here also refers to Google, not Twitter/Musk devs.
Well, RAG is a form of lookup operation from an external set of documents that tries to work around anterograde amnesia. Humans don't suffer from that condition unless something has gone badly wrong. We don't carry around sets of documents and constantly consult them because we forgot what we read 10 seconds ago. Instead we have instead short term memory (e.g. what was the last paragraph I just read), a sort of medium-term memory that can absorb lots of information (e.g. the contents of a textbook if you're studying a subject) but which will eventually be forgotten, and a general long term memory that is rarely forgotten (e.g. riding a bike, how to speak your mother tongue ....).
In LLM terms the parameters are the long term memory, the activations whilst processing a prompt are the short term memory, but there's no medium term.
This paper is about adding that sort of medium term memory, if I understood it correctly.
RAG is quite the hack. All you start with is the question, but why should the documents you need for the answer be similar to the question? It can be true for some kinds of search-engine like queries, but it'll be false for others. And then the results are attempted to be stuffed into the short term memory, of course it doesn't fit so then RAG gets done at the paragraph level or whatever, at which point you may have lost vital context. It's just going to be a set of paragraphs that happen to resemble the question. And finally word vector similarity isn't necessarily useful and can easily hurt, e.g. France and Germany are considered close conceptually but if your question is about France then stuffing the context with stuff related to Germany is going to fail much harder than a classical keyword based search engine would.
If you integrate memory with the attention mechanism then you can use "memories" captured at higher levels of abstraction and not just word-level documents, so it is potentially much more powerful.
I don't know, it actually feels quite similar to something that seems to happen in my head. In a certain context, my brain resurfaces a bunch of memories and thoughts of things that are similar, even if they are not necessarily valuable to the current situation. I am not saying that the how is equivalent, but the what certainly seems at least related.
Yes but RAG isn't re-surfacing memories, it's re-surfacing documents i.e. the source of memories. That's what this new paper tackles: how do you resurface actual memories or the closest analogy at the neural architecture level.
> Instead we have short term memory [...], a sort of medium-term memory that can absorb lots of information [...], and a general long term memory that is rarely forgotten
This is basically folk psychology though, not necessarily representative of the actual mechanisms that are being used. This is a common mistake I see in comparisons of ML and neuroscience or psychology.
ML is all about mechanistic models for things, where we know all the details. We simply don't have any such mechanistic models of how the brain works, all we have are speculative models and folk concepts, ie. how things seem to us not how they are. I'm just very skeptical that these are enough to be confident in claims like this not being how we actually do things in reality.
For instance, the anteretrograde amnesia you describe certainly isn't a common feature of the brain as a whole, but can you really say that it isn't a feature of some parts of the brain responsible for cognition, and then some RAG-like mechanism isn't used to eliminate that problem to give the outward appearance of no amnesia?
Sure it's just an analogy. We clearly don't have 3 distinct separate levels, it's more like one memory where recalling memories strengthens them (but can also change them). But whatever we have, it's clearly very different to an LLM.
We don't even want to replicate the human mind exactly. If we did that what'd be the point? AI is useful exactly because it's not the same as how people work.
I don't think RAG is a good analogy for how the brain works because RAG is about finding original concrete sources (like books), not about matching and recalling digested and abstracted thoughts inside the "mind" itself. The article does talk about this a bit at the beginning.
> But whatever we have, it's clearly very different to an LLM.
My only real quibble with your post is, again, that it seems different to an LLM. Current ML models are still multiple orders of magnitude away from being on the same order of complexity as the human brain, and scaling doesn't yet seem to have hit a limit (more a data limit).
I just don't think we can draw such conclusions at this time.
Sure, but what I'm getting at is that even if you scale a GPT-4 style transformer up 1000x, it won't suddenly develop a human-like memory. It will still be read only, whereas there's no such thing as a read only human unless they are unfortunate enough to have severe brain damage.
> RAG is quite the hack. All you start with is the question, but why should the documents you need for the answer be similar to the question?
I have always thought that a better RAG system would be to let the LLM use the search in the same way that a human would - think about good keywords and search the needed information iteratively until everything needed is retrieved. I have even wrote a proof of concept: https://github.com/zby/answerbot and wrote a blog post (mostly) about that approach: https://zzbbyy.substack.com/p/design-principles-for-llm-base...
But then I discovered that it is very hard to tell the LLM to both use its reasoning and the search - for example if I ask GPT-4 "What is the weight proportion of oxygen in water? Think step by step" it lays out a very good thought process that takes the facts that it knows about the chemical formula of water, the atomic weights of hydrogen and oxygen and does the math to arrive at the good answer. But if I just add wikipedia search functions to the llm call (https://github.com/zby/answerbot/blob/ea792fbc7a84c6683dc2f3...) even without any additional prompting or examples in the prompt it starts searching and rather blindly - it stops any reasoning and tries to find the final answer in wikipedia instead of looking for the intermediate information - like the chemical formula, the atomic weights. I guess I'll find a workaround that problem with some clever prompting - but this difficulty surprised me.
I think it will have been fine-tuned to use functions if they exist, as for most use cases if you specify a function then it means the other end of the connection is an automated program and not a human, so returning pure text without a function call would be a failure.
I've always considered the human ability to abstract things away - the capacity to say "this is like that" - as a kind of data compression mechanism. Instead of memorizing everything, we remember the first few things of that kind that we encountered, and then the differences between them and everything else that's sort of like them. The core memories being the root, and the differences being steps along the path of a trie -- both space and time efficient in storage and retrieval.
I think the addition of medium-term memory to an LLM system would mean an individualization / giving that system instance the means to become unique from its common starting point. We can already consider the context window as a short-term memory of sorts, but it's lobotomized every time. Having a persistent medium-term memory window (eg, via collecting summaries / tries of abstraction "keyframes" + diffs) would begin to pose some serious questions. If a mechanism was further developed to persist these medium-term data summaries into "long term memory" (adjusting weights and parameters), we would have continuous learning.
> Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality.
Its actually a very close model of how people use external data sources to supplement thought in reality. Its a hack, but it is very much the same hack (indexing external data, and looking up and stuffing some in our brain temporarily at need) that humans have used forever (with a novel indexing method.)
Its not (intentionally, at least) a model of anything that happens inside the human brain, but what humans do in reality extends beyond what happens inside the brain.
The most important bits about what they do to make this happen:
> In addition to the causal self-attention integral to transformers, we also allow each query token to attend to a fixed number of “external memories”. These memories are stored in a non-differentiable cache. The choice of which memories to attend to is made using cosine similarity within each decoder layer and attention head.
[...]
> We create our external memories (at each layer) by passing those external contexts through our model, just like inference. Then we save the internal representations the model generated, and attend to them later.
What an extremely clever approach!
If, in a chatbot setting, you update the external memory cache during inference, this means the thing immediately retains memory of the discussion.
Maybe this is an alternative (quicker? more exact?) to LoRA finetuning, to give a foundational model some specific personality and experiental history?
This is great. Ive been using GPT based tools extensively for my side project which I have a demo for(http://drophere.co see code at https://github.com/itissid/drop_webdemo) one thing I think that is missing is a couple of agents that can pick out the right context from my code depending on what needs to get done. Let me explain:
They are a couple of patterns that I used to design chunks of the web app:
One is to do a high level design brainstorming with an agent, write some pseudo code. Which is fine with regular GPTs with limited context lengths.
As I got into the implementation details, there are several sub components which themselves require a dedicated agent because they become 1 week sprints to complete with 100's of loc. But I need them to pull context from other agents of the project, which is missing.,
This latter part could be done by along the lines of this research.
I think a couple of projects are trying to do this using traditional RAG like GPTEngineer can benefit from this.
The article links to their huggingface page[0], which offers both chat and non-chat models, and it appears that they come with the code necessary to be run, but I have not actually tried to run them.
> Finetuning seeks to extend the length of the context window itself.
Does it? I'd thought fine-tuning was more like transfer learning, adding highly specific data to the training set (e.g. your internal codebase) rather than actually modifying the architecture.
Their experiment seems to handle less than 100,000 tokens of text. I wonder if this method could scale to more than 1,000,000 or 10,000,000 tokens.
It seems like it's limited to context length? So with 512 token chunks at 4096 context length that is 2 million tokens. Which could be good for some things but is not going to help for a really large knowledgebase.
Edit: I see the source code is in the Files section on the Hugging Face page.
Wouldn't this require retraining the model each time your "extended mind" needs more information? One benefit of RAG in my mind is that the "database" can be updated in real-time without needing any model retraining
I additionally noticed an Apache license on their models at Huggingface, but the blog post states the work described by that blog post is open source for noncommercial use.
41 comments
[ 3.2 ms ] story [ 77.7 ms ] threadWhen the document length is 2k, it's likely smaller than the context and RAG can just retrieve the entire document to have the model read it. When the document is longer, RAG needs to actually do some work to pick the parts that contain the answer.
The "extended mind" can always query tokens across the entire document, though evidently worse than if they were included in the context.
It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the student a note with the answer right before the exam, compared to letting them read the whole curriculum, but I am very much still learning..
His channel might be a useful resource for you.
[1] https://www.youtube.com/@engineerprompt/videos
[2] https://github.com/PromtEngineer/localGPT
Getting RAG to work right depending on the context is tricky. I like RAG fusion approach with rerankers.
More seriously, this does feel like a real advance. Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. I've been wondering for the last year whether it's possible to extend the attention mechanism to more naturally connect to a bigger set of per-session weights or activations. The moment you encounter the query/key/value analogy it's an obvious idea, the problem being that you need a very strong grip on the low level details of neural architecture to actually do it. Now apparently it is possible! And the way the topk knob actually maps to abstraction is quite amazing.
Still, this doesn't eliminate context window constraints. The memories themselves have a form of context window in this technique. Context size (what they call sequence length) does still matter.
Additionally, the memories have to actually fit in GPU memory (at least in their implementation). And the memories appear to be nearly full snapshots of the network, so they will get quite large (much larger than text grabbed using RAG). So there's going to be a painful tradeoff here for the forseeable future where you'll have to decide whether you want a bigger smarter base model with a bigger context window but less space for memory, or a smaller model with a smaller context window but bigger memory.
This is the first I've heard of Normal Computing, who are these guys/gals exactly?
> Normal is a deep-tech startup founded by former Google Brain & X engineers
Ah. That explains it. "X" here also refers to Google, not Twitter/Musk devs.
How do you know?
In LLM terms the parameters are the long term memory, the activations whilst processing a prompt are the short term memory, but there's no medium term. This paper is about adding that sort of medium term memory, if I understood it correctly.
RAG is quite the hack. All you start with is the question, but why should the documents you need for the answer be similar to the question? It can be true for some kinds of search-engine like queries, but it'll be false for others. And then the results are attempted to be stuffed into the short term memory, of course it doesn't fit so then RAG gets done at the paragraph level or whatever, at which point you may have lost vital context. It's just going to be a set of paragraphs that happen to resemble the question. And finally word vector similarity isn't necessarily useful and can easily hurt, e.g. France and Germany are considered close conceptually but if your question is about France then stuffing the context with stuff related to Germany is going to fail much harder than a classical keyword based search engine would.
If you integrate memory with the attention mechanism then you can use "memories" captured at higher levels of abstraction and not just word-level documents, so it is potentially much more powerful.
This is basically folk psychology though, not necessarily representative of the actual mechanisms that are being used. This is a common mistake I see in comparisons of ML and neuroscience or psychology.
ML is all about mechanistic models for things, where we know all the details. We simply don't have any such mechanistic models of how the brain works, all we have are speculative models and folk concepts, ie. how things seem to us not how they are. I'm just very skeptical that these are enough to be confident in claims like this not being how we actually do things in reality.
For instance, the anteretrograde amnesia you describe certainly isn't a common feature of the brain as a whole, but can you really say that it isn't a feature of some parts of the brain responsible for cognition, and then some RAG-like mechanism isn't used to eliminate that problem to give the outward appearance of no amnesia?
We don't even want to replicate the human mind exactly. If we did that what'd be the point? AI is useful exactly because it's not the same as how people work.
I don't think RAG is a good analogy for how the brain works because RAG is about finding original concrete sources (like books), not about matching and recalling digested and abstracted thoughts inside the "mind" itself. The article does talk about this a bit at the beginning.
My only real quibble with your post is, again, that it seems different to an LLM. Current ML models are still multiple orders of magnitude away from being on the same order of complexity as the human brain, and scaling doesn't yet seem to have hit a limit (more a data limit).
I just don't think we can draw such conclusions at this time.
I have always thought that a better RAG system would be to let the LLM use the search in the same way that a human would - think about good keywords and search the needed information iteratively until everything needed is retrieved. I have even wrote a proof of concept: https://github.com/zby/answerbot and wrote a blog post (mostly) about that approach: https://zzbbyy.substack.com/p/design-principles-for-llm-base... But then I discovered that it is very hard to tell the LLM to both use its reasoning and the search - for example if I ask GPT-4 "What is the weight proportion of oxygen in water? Think step by step" it lays out a very good thought process that takes the facts that it knows about the chemical formula of water, the atomic weights of hydrogen and oxygen and does the math to arrive at the good answer. But if I just add wikipedia search functions to the llm call (https://github.com/zby/answerbot/blob/ea792fbc7a84c6683dc2f3...) even without any additional prompting or examples in the prompt it starts searching and rather blindly - it stops any reasoning and tries to find the final answer in wikipedia instead of looking for the intermediate information - like the chemical formula, the atomic weights. I guess I'll find a workaround that problem with some clever prompting - but this difficulty surprised me.
I think the addition of medium-term memory to an LLM system would mean an individualization / giving that system instance the means to become unique from its common starting point. We can already consider the context window as a short-term memory of sorts, but it's lobotomized every time. Having a persistent medium-term memory window (eg, via collecting summaries / tries of abstraction "keyframes" + diffs) would begin to pose some serious questions. If a mechanism was further developed to persist these medium-term data summaries into "long term memory" (adjusting weights and parameters), we would have continuous learning.
It's pretty wild stuff.
Its actually a very close model of how people use external data sources to supplement thought in reality. Its a hack, but it is very much the same hack (indexing external data, and looking up and stuffing some in our brain temporarily at need) that humans have used forever (with a novel indexing method.)
Its not (intentionally, at least) a model of anything that happens inside the human brain, but what humans do in reality extends beyond what happens inside the brain.
I end up with a lot of book marks as a result.
I’m now a professional and I carry my books, notes, articles, papers, etc… with me everywhere via my iPad and margin notes.
I’ve found, since a young age that knowing where to find the information I need vs memorization is how my brain works best.
Wish it was the opposite though.
> In addition to the causal self-attention integral to transformers, we also allow each query token to attend to a fixed number of “external memories”. These memories are stored in a non-differentiable cache. The choice of which memories to attend to is made using cosine similarity within each decoder layer and attention head.
[...]
> We create our external memories (at each layer) by passing those external contexts through our model, just like inference. Then we save the internal representations the model generated, and attend to them later.
What an extremely clever approach!
If, in a chatbot setting, you update the external memory cache during inference, this means the thing immediately retains memory of the discussion.
Maybe this is an alternative (quicker? more exact?) to LoRA finetuning, to give a foundational model some specific personality and experiental history?
They are a couple of patterns that I used to design chunks of the web app: One is to do a high level design brainstorming with an agent, write some pseudo code. Which is fine with regular GPTs with limited context lengths.
As I got into the implementation details, there are several sub components which themselves require a dedicated agent because they become 1 week sprints to complete with 100's of loc. But I need them to pull context from other agents of the project, which is missing.,
This latter part could be done by along the lines of this research.
I think a couple of projects are trying to do this using traditional RAG like GPTEngineer can benefit from this.
[0]: https://huggingface.co/normalcomputing
Does it? I'd thought fine-tuning was more like transfer learning, adding highly specific data to the training set (e.g. your internal codebase) rather than actually modifying the architecture.
It seems like it's limited to context length? So with 512 token chunks at 4096 context length that is 2 million tokens. Which could be good for some things but is not going to help for a really large knowledgebase.
Edit: I see the source code is in the Files section on the Hugging Face page.