The key seems to be that you take the transcript of a model working within a problem domain that it’s not yet good at or where the context doesn’t match it’s original training and then you continually retrain it based on its efforts and guidance from a human or other expert. You end up with a specialty model in a given domain that keeps getting better at that domain, just like a human.
The hard part is likely when someone proves some “fact” which the models knows and has had reinforced by this training is no longer true. The model will take time to “come around” to understand this new situation. But this isn’t unlike the general populous. At scale humans accept new things slowly.
It's basically continual learning. This is beyond a hard problem it's currently an impossible one. I know of no system that solve CL even at small scale let alone large models.
Annoyingly, they have SOME inherent capability to do it. It's really easy to get sucked down this path due to that glimmer of hope but the longer you play with it the more annoying it becomes.
SSI seems to be focused on this problem directly so maybe they discover something?
Because we don't experience reality through language but direct sensory perception. Language is arbitrary bird song and visual representations dragged forward from history, accepted definitions never uniformly distributed.
Testing based on contextual correctness makes no sense when there is no center to the universe. No "one true context to rule them all".
We learn from hands on sensory experiences. Our bodies store knowledge independent of the brain; often referred to as muscle memory.
Gabe Newell mentioned this years ago; our brain is only great at some things like language and vision processing but the rest of our body is involved in sensory information processing too: https://en.wikiquote.org/wiki/Gabe_Newell
The most potent evidence the brain is not the center of the universe we commonly think it to be is that patient with 90% of their skull filled with fluid while they carried out a typical first worlder life: https://www.sciencealert.com/a-man-who-lives-without-90-of-h...
I'm conflicted. I don't know that I would necessarily want a model to pass all of these. Here is the fundamental problem. They are putting the rules and foundational context in "user" messages.
Essentially I don't think you want to train the models on full compliance to the user messages, they are essentially "untrusted" content from a system/model perspective. Or at least it is not generally "fully authoritative".
This creates a tension with the safety, truthfulness training, etc.
Their example usecases are pretty obvious and clear human needs from an LLM. The semantics of system/user messages and how that affects “safety” doesn’t change the need to fix this crucial problem of “in-context learning” that we all have felt while using LLMs.
Don't always trust everything you read in papers. Researchers are usually under incredible pressure to publish something, anything. Wait a few years and see if the paper survives the test of time. LLMs work reasonably fine for me in new domains.
This is quite on brand for China. I think they are experts at reverse engineering and learning 'from context' rather than by formal consumption of foreign training material.
The fictional training data with a made up country and laws was a very interesting experiment design, I can imagine that's how they approach making business with other countries. Like an alien made up system they have to learn on the spot.
> experts at reverse engineering and learning 'from context' rather than by formal consumption of foreign training material
China (as with other Asian cultures like India) is well known for their schooling involving extreme amounts of formal training material consumption. The reverse-engineering is performed with a solid foundation of theoretical understanding.
Bit by bit, we need to figure out how to rebuild human contextual understanding in a way that LLMs can understand. One thing that gets overlooked is the problem if incorrect data. You can provide all of the context in the world but LLMs tend to choke on contradictions or, at the minimum, work a whole lot harder to determine how to ignore or work around incorrect facts.
"Forgetting" and "ignoring" are hugely valuable skills when building context.
It is weird to read because they bring up many things a lot of people have been critiquing for years.
> But as impressive as these feats are, they obscure a simple truth: being a "test-taker" is not what most people need from an AI.
> In all these cases, humans aren't relying solely on a fixed body of knowledge learned years ago. We are learning, in real-time, from the context right in front of us.
> To bridge this gap, we must fundamentally change our optimization direction.
I'm glad the conversation is changing but it's been a bit frustrating that when these issues were brought up people blindly point to benchmarks. It made doing this type of research difficult (enough to cause many to be pushed out). Then it feels weird to say "harder than we thought" because well... truthfully, they even state why this result should be expected
> They rely primarily on parametric knowledge—information compressed into their weights during massive pre-training runs. At inference time, they function largely by recalling this static, internal memory, rather than actively learning from new information provided in the moment.
And that's only a fraction of the story. Online algorithms aren't enough. You still need a fundamental structure to codify and compress information, determine what needs to be updated (as in what is low confidence), to actively seek out new information to update that confidence, make hypotheses, and so so much more.
So I hope the conversation keeps going in a positive direction but I hope we don't just get trapped in a "RL will solve everything" trap. RL is definitely a necessary component and no doubt will it result in improvements, but it also isn't enough. It's really hard to do deep introspection into how you think. It's like trying to measure your measuring stick with your measuring stick. It's so easy to just get caught up in oversimplification and it seems like the brain wants to avoid it. To quote Feynman: "The first principle is to not fool yourself, and you're the easiest person to fool." It's even easier when things are exciting. It's so easy because you have evidence for your beliefs (like I said, RL will make improvements). It's so easy because you're smart, and smart enough to fool yourself. So I hope we can learn a bigger lesson: learning isn't easy, scale is not enough. I really do think we'll get to AGI but it's going to be a long bumpy road if we keep putting all our eggs in one basket and hoping there's simple solutions.
LLMs of the future will need good data for proper context, but it is less and less making it onto the internet. Unpublished data stores like Discord or meeting recordings are going to be the only way forward. How else can you get up to date information except to be where the people are.
Is this correct? My assumption is that all the data collected during usage is part of the RLHF loop of LLM providers. Assumption is based on information from books like empire of ai which specifically mention intent of AI providers to train/tune their models further based on usage feedback (eg: whenever I say the model is wrong in its response, thats a human feedback which gets fed back into improving the model).
These will just drown in their own data, the real task is consolidating and pruning learned information. So, basically they need to 'sleep' from time to time. However, it's hard to sort out irrelevant information without a filter. Our brains have learned over Milenial to filter because survival in an environment gives purpose.
Current models do not care whether they survive or not. They lack grounded relevance.
I don't understand why that's on the critical path. I'd rather a frozen Ramanujan (+ temporary working memory through context) than a midwit capable of learning.
We need models that are smarter than humans. So far, the cost of an AI query + training is dwarfing the effort it would take to teach an intelligent human how to do a task. We are dumping an incredibly amount of money/effort into making AI do stuff when it's still not competitive with humans, because dumbass people are controlling investment. The stock market is not a replacement for competent investment. The fact people buy meme coins shows how fucked we are.
Deceiving people is not a sustainable business model, but it is the most prominent one in the US right now. Lie to the public, sell them stuff that's bad for them at too high of a price, get rich quick, then act confused when your economy collapses because the victims of your grift can't spend anymore.
> Without any context provided, the state-of-the-art model, GPT-5.1 (High), is only able to solve less than 1% of tasks. This starkly demonstrates that the data is contamination-free, as the model is almost entirely incapable of solving the tasks without learning from the context.
[...]
[With context provided,] on average, models solve only 17.2% of tasks. Even the best-performing model, GPT-5.1 (High), achieves just 23.7%.
It would be interesting to see the results of the latest models. At least, it would allow us to see whether there is progress. Human baseline would be interesting to see too.
It's a very interest benchmark. Much more impressive than needle in haystack benches or just tuneable benches.
I wonder if it's somewhat incompatible with some domains.
I.e. perhaps coding models need to rigidly stick to what they know and resist bad ideas in their contexts - I don't want my mistakes to be replicated by the model.
Still I agree with the premise that learning in session is what I want from a model.
Perhaps once models mature they will diverge even more than just having sophistication and coding or not. But creative, coding, rule based etc models
>> Current language models do not handle context this way. They rely primarily on parametric knowledge—information compressed into their weights during massive pre-training runs. At inference time, they function largely by recalling this static, internal memory, rather than actively learning from new information provided in the moment.
>> This creates a structural mismatch. We have optimized models to excel at reasoning over what they already know yet users need them to solve tasks that depend on messy, constantly evolving context. We built models that rely on what they know from the past, but we need context learners that rely on what they can absorb from the environment in the moment.
>> To bridge this gap, we must fundamentally change our optimization direction.
All this is right and what critics of the deafening over-hyping of LLMs have long pointed out.
So what do the authors propose we do? Currently, they propose ... a benchmark. But, what's that going to achieve? We know very well that LLMs, neural nets in general, are masters at saturating benchmarks without actually mastering the abilities that the benchmarks are meant to be measuring.
What happens if in a year or so, as will definite happen, LLMs saturate this benchmark too? Will we all have to agree that LLMs can now do "context learning" and then move on to the next big thing? This year it's "world models", last year it was "reasoning" and next year it's going to be "context learning"? And then, what? Where is this all leading to, if after all the billions spent and all the benchmarks beaten conclusively, LLMs still can't do reasoning, can't do world-modelling, can't do context learning and so on, and so forth?
25 comments
[ 3.4 ms ] story [ 72.5 ms ] threadThe hard part is likely when someone proves some “fact” which the models knows and has had reinforced by this training is no longer true. The model will take time to “come around” to understand this new situation. But this isn’t unlike the general populous. At scale humans accept new things slowly.
Annoyingly, they have SOME inherent capability to do it. It's really easy to get sucked down this path due to that glimmer of hope but the longer you play with it the more annoying it becomes.
SSI seems to be focused on this problem directly so maybe they discover something?
Testing based on contextual correctness makes no sense when there is no center to the universe. No "one true context to rule them all".
We learn from hands on sensory experiences. Our bodies store knowledge independent of the brain; often referred to as muscle memory.
Gabe Newell mentioned this years ago; our brain is only great at some things like language and vision processing but the rest of our body is involved in sensory information processing too: https://en.wikiquote.org/wiki/Gabe_Newell
The most potent evidence the brain is not the center of the universe we commonly think it to be is that patient with 90% of their skull filled with fluid while they carried out a typical first worlder life: https://www.sciencealert.com/a-man-who-lives-without-90-of-h...
States are banning a reading education framework that's been linked to lower literacy scores in younger generations; 3-cueing relies on establishing correctness via context assessment: https://www.edweek.org/teaching-learning/more-states-are-tak...
"Establishing context" is a euphemism for "arguing semantics".
Putting the brain at the root of of human intelligence is a relic of hierarchical and taxonomical models. There are no natural hierarchies.
I'm conflicted. I don't know that I would necessarily want a model to pass all of these. Here is the fundamental problem. They are putting the rules and foundational context in "user" messages.
Essentially I don't think you want to train the models on full compliance to the user messages, they are essentially "untrusted" content from a system/model perspective. Or at least it is not generally "fully authoritative".
This creates a tension with the safety, truthfulness training, etc.
The fictional training data with a made up country and laws was a very interesting experiment design, I can imagine that's how they approach making business with other countries. Like an alien made up system they have to learn on the spot.
China (as with other Asian cultures like India) is well known for their schooling involving extreme amounts of formal training material consumption. The reverse-engineering is performed with a solid foundation of theoretical understanding.
"Forgetting" and "ignoring" are hugely valuable skills when building context.
So I hope the conversation keeps going in a positive direction but I hope we don't just get trapped in a "RL will solve everything" trap. RL is definitely a necessary component and no doubt will it result in improvements, but it also isn't enough. It's really hard to do deep introspection into how you think. It's like trying to measure your measuring stick with your measuring stick. It's so easy to just get caught up in oversimplification and it seems like the brain wants to avoid it. To quote Feynman: "The first principle is to not fool yourself, and you're the easiest person to fool." It's even easier when things are exciting. It's so easy because you have evidence for your beliefs (like I said, RL will make improvements). It's so easy because you're smart, and smart enough to fool yourself. So I hope we can learn a bigger lesson: learning isn't easy, scale is not enough. I really do think we'll get to AGI but it's going to be a long bumpy road if we keep putting all our eggs in one basket and hoping there's simple solutions.
Norms will shift, be prepared.
There's pretraining, training, and finetuning, during which model parameters are updated.
Then there's inference, during which the model is frozen. "In-context learning" doesn't update the model.
We need models that keep on learning (updating their parameters) forever, online, all the time.
These will just drown in their own data, the real task is consolidating and pruning learned information. So, basically they need to 'sleep' from time to time. However, it's hard to sort out irrelevant information without a filter. Our brains have learned over Milenial to filter because survival in an environment gives purpose.
Current models do not care whether they survive or not. They lack grounded relevance.
Deceiving people is not a sustainable business model, but it is the most prominent one in the US right now. Lie to the public, sell them stuff that's bad for them at too high of a price, get rich quick, then act confused when your economy collapses because the victims of your grift can't spend anymore.
[...]
[With context provided,] on average, models solve only 17.2% of tasks. Even the best-performing model, GPT-5.1 (High), achieves just 23.7%.
I wonder if it's somewhat incompatible with some domains.
I.e. perhaps coding models need to rigidly stick to what they know and resist bad ideas in their contexts - I don't want my mistakes to be replicated by the model.
Still I agree with the premise that learning in session is what I want from a model.
Perhaps once models mature they will diverge even more than just having sophistication and coding or not. But creative, coding, rule based etc models
>> This creates a structural mismatch. We have optimized models to excel at reasoning over what they already know yet users need them to solve tasks that depend on messy, constantly evolving context. We built models that rely on what they know from the past, but we need context learners that rely on what they can absorb from the environment in the moment.
>> To bridge this gap, we must fundamentally change our optimization direction.
All this is right and what critics of the deafening over-hyping of LLMs have long pointed out.
So what do the authors propose we do? Currently, they propose ... a benchmark. But, what's that going to achieve? We know very well that LLMs, neural nets in general, are masters at saturating benchmarks without actually mastering the abilities that the benchmarks are meant to be measuring.
What happens if in a year or so, as will definite happen, LLMs saturate this benchmark too? Will we all have to agree that LLMs can now do "context learning" and then move on to the next big thing? This year it's "world models", last year it was "reasoning" and next year it's going to be "context learning"? And then, what? Where is this all leading to, if after all the billions spent and all the benchmarks beaten conclusively, LLMs still can't do reasoning, can't do world-modelling, can't do context learning and so on, and so forth?