A promising and potentially useful approach for adding additional context to your prompts.
How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?
> In practice, plain, well-designed summaries should be optimal to fit larger documents in the context.
> This concept has potential, though; building lookup tables seems to outperform long text summarization.
It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.
ChatGPT 3.5 in response to a brief analysis of the community gave out:
"It seems like the conversation you provided is discussing the idea of compressing prompts or summaries of larger documents in order to fit them into a smaller context. While the idea has potential, it appears that people have had difficulty reproducing it and that lookup tables and external storage of memory may be more effective in the long term. Additionally, there is discussion about the possibility of greatly increasing context size in the future, with the example given that there was an almost 10x jump between GPT-3.5-turbo and GPT-4 in terms of token capacity."
A more accurate title would be "decrease GPT4's context size by asking it to obfuscate your prompts". You're getting a prompt that consumes more tokens and isn't particularly faithful to the original.
The context length includes all of GPT4's replies, is that correct? If I want to minimize the context length, should I be asking GPT4 to limit its replies to just the fix that is requested?
GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.
14 comments
[ 0.17 ms ] story [ 43.1 ms ] threadHow likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?
Or is context likely to stay fixed around 32k for the foreseeable future?
That is the question.
A) people have had a hard time reproducing it, and
B) more damning, the "compressed" version uses more tokens than the original (https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d...)
> In practice, plain, well-designed summaries should be optimal to fit larger documents in the context.
> This concept has potential, though; building lookup tables seems to outperform long text summarization.
It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.
Sessions are getting treated as more valuable than necessary.
This should look more like functions.
Needs lower prices and greater availability for that.
But until then… smashing duplos together.
Obvious ideas being held up as something more than that.
If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things.
It helps set a more reasonable scale for these sensational sounding snippets.
https://twitter.com/gfodor/status/1643444605332099072?s=20
https://twitter.com/dogeofcoin/status/1642918892602290179
GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.