Yesterday I asked one LLM about certain provisions of Ukrainian law, where severity threshold for a financial crime was specified indirectly through a certain well known constant. The machine got it wrong and when asked to give sources cited the respective law referencing a similary sounding unit. Amazingly it gave the correct English tranlation but gave me the wrong original in Ukrainian.
I guess it merged two tokens why learning the text.
Amazingly it also knows about difference between two constants, but referrs to the wrong one in both calculations and in hallucinating the quote.
It's tedious to always check for stuff like this.
Then I asked a different LLM and it turned out that actually the constant is monkey patched for specific contexts and both me and the first lying machine were wrong
I'm not convinced the brain stores memories, or that memory storage is required for human intelligence. And we "hallucinate" all the time. See: eye witness testimony being wrong regularly, "paranormal" experiences etc.
It's a statement that /feels/ true, because we can all look "inside" our heads and "see" memories and facts. But we may as well be re-constructing facts on the fly, just as re-construct reality itself to sense it.
> I’ll remind you that biologists do not, in the year 2025, know memory’s physical substrate in the brain! Plenty of hypotheses — no agreement. Is there any more central mystery in human biology, maybe even human existence?
Did they not recently transfer memory of how to solve a maze from one mouse to another, giving credibility to what can store information?
Searching, I only find the RNA transfers done in 60s, which ran into some problems. I thought a recent study did transfer proteins.
A model is capable of learning the "calibration" during the reinforcement learning phase, in this "old" post from OpenAI: https://openai.com/index/introducing-simpleqa/ you can see the positive correlation between stated confidence and accuracy.
You know, whatever memory is or where it’s at and however the mind works, I’m grateful I’ve got mine in tact right now and I appreciate science’s inability to zero in on these things.
It’s nice to know that this sort of appreciation is becoming more common. Somewhere between tech accelerationism and protestant resistance are those willing to re-interrogate human nature in anticipation of what lies ahead.
OpenAI just recently took a systematic look into why models hallucinate [0][1].
The article you shared raises an interesting point by comparing human memory with LLMs, but I think the analogy can only go so far. They're too distinct to explain hallucinations simply as a lack of meta-cognition or meta-memory. These systems are more like alien minds, and allegories risk introducing imprecision when we're trying to debug and understand their behavior.
OpenAI's paper instead identifies hallucinations as a bug in training objectives and benchmarks, and is grounding the explanation in first principles and the mechanics of ML.
Metaphors are useful for creativity, but less so when it comes to debugging and understanding, especially now that the systematic view is this advanced.
I haven't read the full paper yet, but my intuition is that hallucinations are a byproduct of models having too much information that needs to be compressed for generalizing.
We already know that larger models hallucinate less since they can store more information, are there any smaller models which hallucinate less
Hallucinations are actually not a malfunction or any other process outside of the normal functioning of the model. They are merely an output that we find unuseful, but in all other ways is optimal based on the training data, context, and model precision and parameters being used.
I honestly have no idea why OAI felt that they needed to publish a “paper” about this, since it is blazingly obvious to anyone who understands the fundamentals of transformer inference, but here we are.
The confusion on this topic comes from calling these suboptimal outputs “hallucinations” which drags anthropomorphic fallacies into the room by their neck even though they were peacefully minding their own business down the corridor on the left.
“Hallucination” implies a fundamentally fixable error in inference, a malfunction of thought caused by a pathology or broken algorithm.
LLMs “Hallucinating” are working precisely as implemented, only we don’t feel like the output usefully matches the parameters from a human perspective. It’s just unhelpful results from the algorithm, like any other failure of training, compression, alignment, or optimisation.
Did you read TFA? It gives concrete advice on how to change the training and eval of these models in order to decrease the error rate. Sure, these being stochastic models, the rate will never reach zero, but given that they are useful today, decreasing the error rate is worthy cause. All this complaining on semantics is just noise to me. It stems from being fixated on some airy-fairy ideas of AGI/ASI, as if anything else doesn't matter. Does saying that a model "replied" to a query mean we are unwittingly anthropomorphizing them? It's all just words, we can extend their use as we see fit. I think "confabulation" would be a more fitting term, but beyond that, I'm not seeing the problem.
Everybody quickly learns to take note of who-claims-what: in kindergarten kids learn to detect each others lies, and learn to attach more credibility to the teachers / caretakers than dubious claims "I am allowed to hit you", to the point that precisely because they learn it they start crafting insidious lies "the teacher said I'm allowed to hit you", after which you learn to be skeptical of meta-claims etc.
I agree with the folks who call these screwups rather than hallucinations because the point of LLMs isn't to be right, it's to be statistically highly likely. If making something up fits that model, then that's what it will do. That's literally how it works.
I believe this is why the importance of written (human) knowledge is only increasing, especially internally at companies. Written knowledge (i.e documentation) has always served as a knowledge cache and a way to transfer knowledge between people.
Without fundamental changes to the LLMs or the way we think about agentic systems, high quality, comprehensive written knowledge is the best path to helping agents "learn" over time.
The fundamental limitation of LLMs is that they represent knowledge as parametric curves, and their generalization is merely interpolation of those curves. This can only ever produce results that correlate with facts (training data), not ones that are causally derived from them, which makes hallucinations inevitable. Same as with human memory.
As a memory neuroscientist, I enjoyed the shoutout here to episodic memory. It strikes me, however, that a feature that I've noticed when observing "reasoning" models is that they may explicitly search for evidence for intermediate pieces of their responses. If we're following the "remember"/"know" distinction developed by Squire and others, perhaps the more apt analogy might be that a singly pass through an LLM is similar to a "I know this" result, prone to hallucination, conflation, etc., and the multipass reasoning model might be more akin to the "I remember this" result, where primary evidence serves as a substrate for the response?
"I’ll remind you that biologists do not, in the year 2025, know memory’s physical substrate in the brain! Plenty of hypotheses — no agreement. Is there any more central mystery in human biology, maybe even human existence?"
A hypothesis is very distinct from theoretical knowledge. A hypothesis lacks empirical evidence. A theory uses empirical information. That CS personnel are lacking both the scientific method and the ability to discern the current state of the art empirical research to disprove such wildly unsupported statements speaks to the field's total failure to develop present-day relevant tools. I would direct the author to two critical books
Sorry for harping on it, but I think this clearly reflects the difference between 2 approaches to storing knowledge, lossy but humongous, and lossless but limited.
LLMs - Lossy highly compressed knowledge which when prompted "hallucinates" facts. LLMs hallucinations are simply how the stored information is retrieved.
Memory (human in this case) - Extremely limited, but almost always correct.
I have temporal lobe epilepsy. The premise of this article seems quite wrong to me.
From the article:
"I think it’s because I don’t only know things: I remember learning them. My knowledge is sedimentary, and I can “feel” the position and solidity of different facts and ideas in that mass. I can feel, too, the airy disconnect of a guess."
I think this can be rephrased as a statement that episodic memory (i.e. recalling the act of learning) is associated with semantic memory (i.e. recalling the fact itself). And for people with more normal brains, it seems it often is.
For people with temporal lobe epilepsy, in many cases the episodic memory isn't there. TLE frequently damages the hippocampus. Often, immediately after taking my meds, I can't recall doing so. I only know I did because I mark it in a notebook when I do it. Things like that are common. However, I can absolutely learn things like structure of systems, API calls, and the like. I believe that I perceive when I'm not sure of something, or that I'm guessing.
Of course, this kind of perception is difficult to verify outside of a study. A lot of memory is very unreliable, even for normal people. Us lucky people with TLE are often more aware of this, because in my case, I have very vivid memories of events that certainly did not occur. These memories are some of the most vivid I have; they seem more real than reality. I have memories of events that I can't verify by comparing them to the current state of the world; I have no idea if they happened or not. Even with all that, I believe my semantic memory is reasonably good; I can do my job as a software engineer.
21 comments
[ 3.3 ms ] story [ 51.6 ms ] threadI guess it merged two tokens why learning the text.
Amazingly it also knows about difference between two constants, but referrs to the wrong one in both calculations and in hallucinating the quote.
It's tedious to always check for stuff like this.
Then I asked a different LLM and it turned out that actually the constant is monkey patched for specific contexts and both me and the first lying machine were wrong
It's a statement that /feels/ true, because we can all look "inside" our heads and "see" memories and facts. But we may as well be re-constructing facts on the fly, just as re-construct reality itself to sense it.
Did they not recently transfer memory of how to solve a maze from one mouse to another, giving credibility to what can store information?
Searching, I only find the RNA transfers done in 60s, which ran into some problems. I thought a recent study did transfer proteins.
It’s nice to know that this sort of appreciation is becoming more common. Somewhere between tech accelerationism and protestant resistance are those willing to re-interrogate human nature in anticipation of what lies ahead.
A different blog post from this month detailing an experience with ChatGPT that netted a similar reflection: https://zettelkasten.de/posts/the-scam-called-you-dont-have-...
The article you shared raises an interesting point by comparing human memory with LLMs, but I think the analogy can only go so far. They're too distinct to explain hallucinations simply as a lack of meta-cognition or meta-memory. These systems are more like alien minds, and allegories risk introducing imprecision when we're trying to debug and understand their behavior.
OpenAI's paper instead identifies hallucinations as a bug in training objectives and benchmarks, and is grounding the explanation in first principles and the mechanics of ML.
Metaphors are useful for creativity, but less so when it comes to debugging and understanding, especially now that the systematic view is this advanced.
[0] https://openai.com/index/why-language-models-hallucinate/?ut... [1] https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4a...
We already know that larger models hallucinate less since they can store more information, are there any smaller models which hallucinate less
I honestly have no idea why OAI felt that they needed to publish a “paper” about this, since it is blazingly obvious to anyone who understands the fundamentals of transformer inference, but here we are.
The confusion on this topic comes from calling these suboptimal outputs “hallucinations” which drags anthropomorphic fallacies into the room by their neck even though they were peacefully minding their own business down the corridor on the left.
“Hallucination” implies a fundamentally fixable error in inference, a malfunction of thought caused by a pathology or broken algorithm.
LLMs “Hallucinating” are working precisely as implemented, only we don’t feel like the output usefully matches the parameters from a human perspective. It’s just unhelpful results from the algorithm, like any other failure of training, compression, alignment, or optimisation.
The analogy may be much less wishy-washy than you imagine: check out "Source-aware training"
https://arxiv.org/abs/2404.01019
Everybody quickly learns to take note of who-claims-what: in kindergarten kids learn to detect each others lies, and learn to attach more credibility to the teachers / caretakers than dubious claims "I am allowed to hit you", to the point that precisely because they learn it they start crafting insidious lies "the teacher said I'm allowed to hit you", after which you learn to be skeptical of meta-claims etc.
Sadly no SOTA LLM's use source-aware training.
Perhaps start a petition on
change.org openpetition.eu openpetition.org
etc.
Without fundamental changes to the LLMs or the way we think about agentic systems, high quality, comprehensive written knowledge is the best path to helping agents "learn" over time.
"I’ll remind you that biologists do not, in the year 2025, know memory’s physical substrate in the brain! Plenty of hypotheses — no agreement. Is there any more central mystery in human biology, maybe even human existence?"
A hypothesis is very distinct from theoretical knowledge. A hypothesis lacks empirical evidence. A theory uses empirical information. That CS personnel are lacking both the scientific method and the ability to discern the current state of the art empirical research to disprove such wildly unsupported statements speaks to the field's total failure to develop present-day relevant tools. I would direct the author to two critical books
Evolution of Memory Systems
https://academic.oup.com/book/26033
How we remember: brain mechanisms of episodic memory
https://direct.mit.edu/books/monograph/2909/How-We-RememberB...
let's stop taking opinions on ai from randoms. please. they haven't a fkin clue.
LLMs - Lossy highly compressed knowledge which when prompted "hallucinates" facts. LLMs hallucinations are simply how the stored information is retrieved.
Memory (human in this case) - Extremely limited, but almost always correct.
Just an observation. No morals.
From the article: "I think it’s because I don’t only know things: I remember learning them. My knowledge is sedimentary, and I can “feel” the position and solidity of different facts and ideas in that mass. I can feel, too, the airy disconnect of a guess."
I think this can be rephrased as a statement that episodic memory (i.e. recalling the act of learning) is associated with semantic memory (i.e. recalling the fact itself). And for people with more normal brains, it seems it often is.
For people with temporal lobe epilepsy, in many cases the episodic memory isn't there. TLE frequently damages the hippocampus. Often, immediately after taking my meds, I can't recall doing so. I only know I did because I mark it in a notebook when I do it. Things like that are common. However, I can absolutely learn things like structure of systems, API calls, and the like. I believe that I perceive when I'm not sure of something, or that I'm guessing.
Of course, this kind of perception is difficult to verify outside of a study. A lot of memory is very unreliable, even for normal people. Us lucky people with TLE are often more aware of this, because in my case, I have very vivid memories of events that certainly did not occur. These memories are some of the most vivid I have; they seem more real than reality. I have memories of events that I can't verify by comparing them to the current state of the world; I have no idea if they happened or not. Even with all that, I believe my semantic memory is reasonably good; I can do my job as a software engineer.