I came across the above on a twitter thread by Shashank Joshi where he posted comments given by a GCHQ data scientist at a Turing Institute presentation. The thread link below contains some of the purported quotes.
1. They claim that "an LLM does not understand the semantic meaning of a sentence in a linguistic sense, but rather calculates mathematically what the most likely next word should be based on the input to the model." My opinion, shared by many others in the field: In order to compute an accurate probability distribution over all possible next words, an LLM must understand semantic meaning, internally, in some fashion. How else could an LLM do that?
2. The authors claim that an LLM "is extremely good at determining the most likely next sequence – and convincingly so – but has no inherent representation of what those words mean." My opinion, shared by many others in the field: In order to compute accurate joint probability distributions over all possible subsequent sequences of words, an LLM must have some kind of internal representation of what those words mean. How else could an LLM do that?
3. The authors claim that "LLMs do not encode an understanding of our world." My opinion, shared by many others in the field: In order to be able chat with people, write essays, pass exams, etc., LLMs must encode an understanding of our world, even if that understanding is obtained second-hand from statistical patterns seen in language, images, etc. How else could an LLM do all those things?
The difference between your views and those claims from the OP are understandable. To answer your questions ("How else could an LLM do that?"), does this nuance help?:
- The LLM is always answering "what it interprets that YOU want to hear", including all the input biases (its training data, its construction and settings, your prompt, etc.).
- Instead, if the LLM "thought" ("understood" with an "inherent representation" to "encode an understanding of our world"), then the LLM could answer "what IT thinks and applied reasoning (in some way, consciously or not) to arrive at", then.
To restate that, it's the difference between GUESSING WHAT YOU WANT TO HEAR, GIVEN ALL INPUTS versus GIVING ITS OWN HONEST ASSESSMENT GIVEN ALL OF ITS INPUTS.
In other words, IT HAS NO HONEST ASSESSMENT OF ITS OWN, per se. It cannot assess on its own behalf. It only reacts in the way it has been configured to.
Am I on the right track, trying to tease apart the disparity of opinion on those 3 claims?
For a machine to be able to say "what you want to hear," it must understand what you're talking about, internally, in some fashion. How else could it do that?
As Hans Moravec wrote in 1998, when he predicted intelligent machines would appear in the 2020's:
> Only on the outside, where they can be appreciated as a whole, will the impression of intelligence emerge. A human brain, too, does not exhibit the intelligence under a neurobiologist's microscope that it does participating in a lively conversation.
Perhaps by having a dataset of symbols that corresponds to symbols that you similarly understand how they can be put together, as well by having a model for how to respond to inputs given some prompts.
You can fill a SQL database with different kinds of apples and their prices.
You can "ask" the price of a Golden Delicious apple, and the DBMS responds--intelligently, with the "right" answer given the question asked in semantic business language.
How could the DBMS do that, if it didn't "understand" the data?
The answer is, the machine system contained a system of symbols and a model for how to give you want you want. But the system didn't utilize any kind of first principles in its "understanding".
I hypothesize there is a "true sense of understanding" or "understanding before anything else" that machines can mimic but that--currently--only humans are good at (even with LLM advances). At least, I haven't seen any evidence to the contrary.
As an aside, if there really were an LLM that could understand, reason, "think" (so that it "understood" with an "inherent representation" to "encode an understanding of our world"), then it would, by all likelihood, be spouting some disruptive-seeming output (in the best sense possible--as in, output supportive of paradigm shifts from status quo positions in place for no reason other than momentum) across domains, and I'm just not seeing that. (Sure, the machine might be coerced to APPEAR to do such things, but that would be an illusion, and not doing it on its own--not true intelligence analysis.)
> Perhaps by having a dataset of symbols that corresponds to symbols that you similarly understand how they can be put together, as well by having a model for how to respond to inputs given some prompts.
That's the same as saying the machine has some kind of internal model of the world, with symbols it has learned standing in for concepts in the world, as understood by the machine. Representing things with symbols is in fact what GOFAI tried to do, but explicitly instead of via learning.
>> Perhaps by having a dataset of symbols...a model for how to respond to inputs....
> That's the same as saying the machine has some kind of internal model of the world...as understood by the machine.
An object can contain a model of (a part of) the world without actually understanding it (for example, a printed photograph or an abacus does this).
The LLM or some machine can have a detailed model of the world--which humans can meaningfully interact with to get outputs--without the machine actually having any true understanding itself.
This thread makes me wonder if there are different (nuanced) usages of the word "understanding" at play (and maybe related words too).
It also makes me curious--not to pick on the subject, but I think it is relevant: Do you happen hold the view that humans can be described with a purely physical model (mechanical, chemical, electric, or whatever materialistic form[s] of choice)? E.g., that humans are also just machines, however advanced/complex? I hypothesize that such a view might constrain the definition of "understanding" to a subset of its possible meanings, to which the article in the OP might not be constrained.
7 comments
[ 3.2 ms ] story [ 30.3 ms ] threadhttps://x.com/shashj/status/1783872773355373039
1. They claim that "an LLM does not understand the semantic meaning of a sentence in a linguistic sense, but rather calculates mathematically what the most likely next word should be based on the input to the model." My opinion, shared by many others in the field: In order to compute an accurate probability distribution over all possible next words, an LLM must understand semantic meaning, internally, in some fashion. How else could an LLM do that?
2. The authors claim that an LLM "is extremely good at determining the most likely next sequence – and convincingly so – but has no inherent representation of what those words mean." My opinion, shared by many others in the field: In order to compute accurate joint probability distributions over all possible subsequent sequences of words, an LLM must have some kind of internal representation of what those words mean. How else could an LLM do that?
3. The authors claim that "LLMs do not encode an understanding of our world." My opinion, shared by many others in the field: In order to be able chat with people, write essays, pass exams, etc., LLMs must encode an understanding of our world, even if that understanding is obtained second-hand from statistical patterns seen in language, images, etc. How else could an LLM do all those things?
- The LLM is always answering "what it interprets that YOU want to hear", including all the input biases (its training data, its construction and settings, your prompt, etc.).
- Instead, if the LLM "thought" ("understood" with an "inherent representation" to "encode an understanding of our world"), then the LLM could answer "what IT thinks and applied reasoning (in some way, consciously or not) to arrive at", then.
To restate that, it's the difference between GUESSING WHAT YOU WANT TO HEAR, GIVEN ALL INPUTS versus GIVING ITS OWN HONEST ASSESSMENT GIVEN ALL OF ITS INPUTS.
In other words, IT HAS NO HONEST ASSESSMENT OF ITS OWN, per se. It cannot assess on its own behalf. It only reacts in the way it has been configured to.
Am I on the right track, trying to tease apart the disparity of opinion on those 3 claims?
As Hans Moravec wrote in 1998, when he predicted intelligent machines would appear in the 2020's:
> Only on the outside, where they can be appreciated as a whole, will the impression of intelligence emerge. A human brain, too, does not exhibit the intelligence under a neurobiologist's microscope that it does participating in a lively conversation.
Source: https://www.jetpress.org/volume1/moravec.htm
Perhaps by having a dataset of symbols that corresponds to symbols that you similarly understand how they can be put together, as well by having a model for how to respond to inputs given some prompts.
You can fill a SQL database with different kinds of apples and their prices.
You can "ask" the price of a Golden Delicious apple, and the DBMS responds--intelligently, with the "right" answer given the question asked in semantic business language.
How could the DBMS do that, if it didn't "understand" the data?
The answer is, the machine system contained a system of symbols and a model for how to give you want you want. But the system didn't utilize any kind of first principles in its "understanding".
I hypothesize there is a "true sense of understanding" or "understanding before anything else" that machines can mimic but that--currently--only humans are good at (even with LLM advances). At least, I haven't seen any evidence to the contrary.
As an aside, if there really were an LLM that could understand, reason, "think" (so that it "understood" with an "inherent representation" to "encode an understanding of our world"), then it would, by all likelihood, be spouting some disruptive-seeming output (in the best sense possible--as in, output supportive of paradigm shifts from status quo positions in place for no reason other than momentum) across domains, and I'm just not seeing that. (Sure, the machine might be coerced to APPEAR to do such things, but that would be an illusion, and not doing it on its own--not true intelligence analysis.)
That's the same as saying the machine has some kind of internal model of the world, with symbols it has learned standing in for concepts in the world, as understood by the machine. Representing things with symbols is in fact what GOFAI tried to do, but explicitly instead of via learning.
> That's the same as saying the machine has some kind of internal model of the world...as understood by the machine.
An object can contain a model of (a part of) the world without actually understanding it (for example, a printed photograph or an abacus does this).
The LLM or some machine can have a detailed model of the world--which humans can meaningfully interact with to get outputs--without the machine actually having any true understanding itself.
This thread makes me wonder if there are different (nuanced) usages of the word "understanding" at play (and maybe related words too).
It also makes me curious--not to pick on the subject, but I think it is relevant: Do you happen hold the view that humans can be described with a purely physical model (mechanical, chemical, electric, or whatever materialistic form[s] of choice)? E.g., that humans are also just machines, however advanced/complex? I hypothesize that such a view might constrain the definition of "understanding" to a subset of its possible meanings, to which the article in the OP might not be constrained.
Regardless, I appreciate the discussion.