One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything! This is a very important open question to me, as I have been building a fairly nuanced RAG pipeline for AI coding.
With GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand.
I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.
If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).
Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens.
I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it.
Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now
The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do.
In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.
If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)
I took the text of Macbeth and changed the dialect of one line:
Original: "The rest, that are within the note of expectation, alreadie are i'th' Court"
New: "It looks like everyone is here already"
Then I gave the entire text to Gemini, ~400000 tokens, and asked it to analyze the play's diction and style, and to identify any lines that didn't match the prevailing dialect.
I think this kind of haystack recall is going to be extremely valuable to large enterprises with massive volumes of standards/practices/technical documents.
The status quo is that you have employees ingest and internalize this material over years of employment, then have those experienced employees manage large projects by telling the project if they're in compliance with the organization's standards.
A reliable recall system like this isn't just a drop in replacement for new employees, its a drop in replacement for a lot of what makes the expensive employees important: the backwards-and-forwards knowledge of the standards/practices.
Although our standard template constructors store all of mankind's knowledge and obviate the need for dangerous thinking, only a tech-priest of great experience may adequately consecrate them.
It's a reference to WH40k, a fictional universe where faster-than-light travel survived the collapse of civilization in the form of pragmatic tools called "standard template constructors," made to take the burden of science and engineering staff off the shoulders of small planetary colonies, and the formalized rituals of the post-apocalyptic martian cults that remembered in some ways how to use them.
That means the models still confabulate and make other errors, and for many cases it's a problem, so there should be solutions to control model output quality
19 comments
[ 2.7 ms ] story [ 45.7 ms ] threadWith GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand.
I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.
I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it.
Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now
In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.
It is unable to find edits: https://twitter.com/mengdi_en/status/1763338245696274469
It hallucinates changes badly: https://twitter.com/thomasahle/status/1763408041960231010
To be fair, finding diffs is an odd way to use LLMs, but does show the highly variable abilities of these models over bigger contexts.
In other models, reasoning performance declines with more tokens (no data yet here for Gemini 1.5 Pro): https://arxiv.org/html/2402.14848v1
Note in that last chart that significant declines in reasoning are apparent over just a few thousand tokens.
Original: "The rest, that are within the note of expectation, alreadie are i'th' Court"
New: "It looks like everyone is here already"
Then I gave the entire text to Gemini, ~400000 tokens, and asked it to analyze the play's diction and style, and to identify any lines that didn't match the prevailing dialect.
It found that one line I'd changed.
My test demonstrates that even with a large context, Gemini 1.5 finds the needle in the haystack.
And it's not even searching for specific terms or meanings, the only clues are subtle differences in style and diction.
It also seems to line up with random paragraph location doing the worst in that paper.
The status quo is that you have employees ingest and internalize this material over years of employment, then have those experienced employees manage large projects by telling the project if they're in compliance with the organization's standards.
A reliable recall system like this isn't just a drop in replacement for new employees, its a drop in replacement for a lot of what makes the expensive employees important: the backwards-and-forwards knowledge of the standards/practices.
> 6) A very important and at the same time difficult class of solutions is model result validation
Thanks.