Thinking/intelligence is like magic, to use Dennett's analogy. Any time a machine can do it, it becomes engineering. It's just a bag of tricks. The circle of true thinking/intelligence is an ever shrinking bag of tricks that only carbon-based minds produced through sexual intercourse can do. In fact I'm surprised they call test tube babies intelligent. Do they have souls?
Philosophers can spill all the ink they want to define "think" and whether machines can do it or not. Given some input, the machine takes some time, and then comes up with some output. Coloquially, the machine is thinking during that time. This has been true since there have been computers and long before LLMs. Now that computers can generate essays about anything, maybe it becomes a question that people feel is important to answer for their day to day life, but I doubt it.
Inference means synthesizing new facts from facts already known. A large language model only knows facts about language elements in its training corpus, therefore any reasoning based on such a model can only ever derive facts about language.
Hmm, I don't know if the example really shows what the article argues it does.
If someone came up to me and told me the altered version of the joke I have heard many times, I might answer exactly like Chat GPT did. I would hear the beginning of the story, say "wait, i know this one", and not really pay attention to the rest because I would be all ready to respond with what I think is the right answer.
I bet if you point out the mistake, the LLM will say "oh you are right, this story actually does specify the gender of the doctor" or something to that effect... just like you or I would.
Now, I am not arguing that LLMs are really 'thinking' like humans, but I also find the second argument a bit tenuous. The article conjectures that humans reason from ideas to symbols while LLMs go from symbols to ideas, but couldn't that just be a post hoc interpretation we have for how our ideas come to our brain? We think we have ideas first, but an idea is just the result of neurons firing in our brains... and neurons are really just a probability function connected to other probability functions, just like an LLM... we give it symbols we understand to represent those probabilities, but that is just for our benefit so we can understand it.
It could be that the only difference between us and an LLM is processing power and the training data generated over billions of years of evolution.
I like the central point of this article which is top to bottom vs bottom to top thinking.
But I wonder if there is a falsifiable, formal definition to suggest that models (or anything for that matter) _do_ think.
The normal reply to chatgpt getting a question right is that it simply extrapolated what was already in the training data set. But I feel like the degree to which something "thinks" is the ability to generalise what it already knows.
This generalisation needs some formality - maybe some mathematical notation (like the opposite of overfitting). By generalisation I mean the ability to get something correct that exists pretty far from the training data.
The reason I suggest this is because GPT can solve pretty much any high school math problem you throw at it and it can do it better than 99% of humans. This is clearly not just memorising training data but doing something more. If it were not generalising, it couldn't possibly solve all new high school level mathematics.
But the extent decreases as you go higher level into undergraduate mathematics where it can still solve most problems you throw at it but not all. And still lower in PhD level mathematics. So the "thinking" ability of GPT exists somewhere in between - in some spectrum. But I don't think you can directly say that it can never generalise PhD level mathematics.. it could do it for high school so why not PhD?
If hypothetically it can solve PhD level mathematics, would people still claim that LLM's don't think?
This debate is a huge red herring. No one is ever going to agree on what 'thinking' means, since we can't even prove that other people are thinking, only that one's self is.
What we should concentrate on is agency. Does the system have its own desires and goals, and will it act on its own accord to achieve them? If a system demonstrates those things, we should accord it the benefit of the doubt that it should have some rights and responsibilities if it chooses to partake in society.
So far, no AI can pass the agency test -- they are all reactive such that they must be given a task before they will do anything. If one day, however, we wake up and find that an AI has written a book on its own initiative, we may have some deciding to do.
The notion of "thinking" is not clear. I'll agree thinking with symbols is powerful and something (adult) humans and computers can do, but is is not the only way of making decisions. I'm going to suggest LLMs are not thinking this way, but that indeed "glorified auto complete" (c.f. Hinton) is far more useful than it seems. Https://arxiv.org/abs/2402.08403
A human who is familiar with the original surgeon riddle could also be tricked the same way that ChatGPT was tricked here. I don't think all LLMs would consistently fall for that one either.
"My personal opinion is that LLMs are autocomplete on steroids."
Yes, and OpenAI's legal docs concur. From their Privacy Policy.
"Services like ChatGPT generate responses by reading a user’s request and, in response, predicting the words most likely to appear next. In some cases, the words most likely to appear next may not be the most factually accurate."
We don't understand how humans think, and we don't yet understand completely how LLMs work. It may be that similar methods are being used, but they might also be different.
What is certain is that LLMs can perform as if they are doing what we call thinking, and for most intents and purposes this is more than enough.
Nice that LLMs can now argue to defend themselves. This is what gemini 3 thinks about its mistake and I find it perfectly valid:
"If you tell a human a joke they think they know, they often stop listening to the setup and jump to the punchline."
And when we say "stop listing" we don't actually mean that they shut their ears, but that they activate a once established neural shortcut - just as the LLM did.
LLMs _can_ think top-to-bottom but only if you make them think about concrete symbol based problems. Like this one: https://chatgpt.com/s/t_692d55a38e2c8191a942ef2689eb4f5a
The prompt I used was "write out the character 'R' in ascii art using exactly 62 # for the R and 91 Q characters to surround it with"
Here it has a top down goal of keeping the exact amount of #'s and Q's and it does keep it in the output. The purpose of this is to make it produce the asciii art in a step by step manner instead of fetching a premade ascii art from training data.
What it does not reason well about always are abstract problems like the doctor example in the post.
The real key for reasoning IMO is the ability to decompose the text into a set of components, then apply world model knowledge to those components, then having the ability to manipulate those components based on what they represent.
Humans have an associative memory so when we read a word like "doctor", our brain gathers the world knowledge about that word automatically. It's kind of hard to tell exactly what world knowledge the LLM has vs doesn't have, but it seems like it's doing some kind of segmentation of words, sentences and paragraphs based on the likelihood of those patterns in the training data, and then it can do _some_ manipulation on those patterns based on other likelihood of those patterns.
Like for example if there is a lot of text talking about what a doctor is, then that produces a probability distribution about what a doctor is, which it then can use in other prompts relating to doctors. But I have seen this fail before as all of this knowledge is not combined into one world model but rather purely based on the prompt and the probabilities associated with that prompt. It can contradict itself in other words.
No they don't. When queried how exactly did a program arrive to a specific output it will happily produce some output resembling thinking and having all the required human-like terminology. The problem is that it doesn't match at all with how the LLM program calculated output in reality. So the "thinking" steps are just a more of the generated BS, to fool us more.
One point to think about - an entity being tested for intelligence/thinking/etc only needs to fail once, o prove that it is not thinking. While the reverse applies too - to prove that a program is thinking it must be done in 100% of tests, or the result is failure. And we all know many cases when LLMs are clearly not thinking, just like in my example above. So the case is rather clear for the current gen of LLMs.
A real debate is possible on the subject but this blog post worth nothing on the subject.
From my side, I don't really know if what does LLM is thinking, but what amaze me is that:
- It is clear to me the way the LLM operate that things are generated token after token, without really a pre-existing plan on what comes next. So, more like a probabilistic repeating machine.
- But it the same time, I can see in action LLM capable to create things or reply to questions that clearly does not exist in the training corpus. So it shows a behavior that is similar to thinking to complete tasks.
For example, let's suppose you give him specific tools to your own custom API, you ask him to do a task, and we can observe that it is capable of mixing multiple calls and combination of the tools results to achieve a given purpose.
Otherwise, when you ask LLM to do math operations like 3123454*2030+500 and it is capable to give the good reply (not all the cases but sometimes). Where, despite the huge size of the corpus, there is not exactly all the operations that are exactly available in the corpus for sure.
So, my best guess is that a lot of things in our world are based on "semantic" patterns that we don't know. Especially for math and logic that are bound to the language. To me it is similar to the mentral trick used by "fast calculator".
If the AI truly becomes human like, then it should also make mistakes like a human does even on simple arithmetic.
Now, AI might make mistakes on simple arithmetic just like humans. However, once the human is given a hint on the fact that there is some simple arithmetic mistake to be corrected without any details, then the human corrects it. But the AI never does.
25 comments
[ 5.4 ms ] story [ 56.0 ms ] threadHope you like this week's post.
If someone came up to me and told me the altered version of the joke I have heard many times, I might answer exactly like Chat GPT did. I would hear the beginning of the story, say "wait, i know this one", and not really pay attention to the rest because I would be all ready to respond with what I think is the right answer.
I bet if you point out the mistake, the LLM will say "oh you are right, this story actually does specify the gender of the doctor" or something to that effect... just like you or I would.
Now, I am not arguing that LLMs are really 'thinking' like humans, but I also find the second argument a bit tenuous. The article conjectures that humans reason from ideas to symbols while LLMs go from symbols to ideas, but couldn't that just be a post hoc interpretation we have for how our ideas come to our brain? We think we have ideas first, but an idea is just the result of neurons firing in our brains... and neurons are really just a probability function connected to other probability functions, just like an LLM... we give it symbols we understand to represent those probabilities, but that is just for our benefit so we can understand it.
It could be that the only difference between us and an LLM is processing power and the training data generated over billions of years of evolution.
But I wonder if there is a falsifiable, formal definition to suggest that models (or anything for that matter) _do_ think.
The normal reply to chatgpt getting a question right is that it simply extrapolated what was already in the training data set. But I feel like the degree to which something "thinks" is the ability to generalise what it already knows.
This generalisation needs some formality - maybe some mathematical notation (like the opposite of overfitting). By generalisation I mean the ability to get something correct that exists pretty far from the training data.
The reason I suggest this is because GPT can solve pretty much any high school math problem you throw at it and it can do it better than 99% of humans. This is clearly not just memorising training data but doing something more. If it were not generalising, it couldn't possibly solve all new high school level mathematics.
But the extent decreases as you go higher level into undergraduate mathematics where it can still solve most problems you throw at it but not all. And still lower in PhD level mathematics. So the "thinking" ability of GPT exists somewhere in between - in some spectrum. But I don't think you can directly say that it can never generalise PhD level mathematics.. it could do it for high school so why not PhD?
If hypothetically it can solve PhD level mathematics, would people still claim that LLM's don't think?
What we should concentrate on is agency. Does the system have its own desires and goals, and will it act on its own accord to achieve them? If a system demonstrates those things, we should accord it the benefit of the doubt that it should have some rights and responsibilities if it chooses to partake in society.
So far, no AI can pass the agency test -- they are all reactive such that they must be given a task before they will do anything. If one day, however, we wake up and find that an AI has written a book on its own initiative, we may have some deciding to do.
Yes, and OpenAI's legal docs concur. From their Privacy Policy.
"Services like ChatGPT generate responses by reading a user’s request and, in response, predicting the words most likely to appear next. In some cases, the words most likely to appear next may not be the most factually accurate."
https://openai.com/en-GB/policies/row-privacy-policy/
What is certain is that LLMs can perform as if they are doing what we call thinking, and for most intents and purposes this is more than enough.
"If you tell a human a joke they think they know, they often stop listening to the setup and jump to the punchline."
And when we say "stop listing" we don't actually mean that they shut their ears, but that they activate a once established neural shortcut - just as the LLM did.
Here it has a top down goal of keeping the exact amount of #'s and Q's and it does keep it in the output. The purpose of this is to make it produce the asciii art in a step by step manner instead of fetching a premade ascii art from training data.
What it does not reason well about always are abstract problems like the doctor example in the post. The real key for reasoning IMO is the ability to decompose the text into a set of components, then apply world model knowledge to those components, then having the ability to manipulate those components based on what they represent.
Humans have an associative memory so when we read a word like "doctor", our brain gathers the world knowledge about that word automatically. It's kind of hard to tell exactly what world knowledge the LLM has vs doesn't have, but it seems like it's doing some kind of segmentation of words, sentences and paragraphs based on the likelihood of those patterns in the training data, and then it can do _some_ manipulation on those patterns based on other likelihood of those patterns. Like for example if there is a lot of text talking about what a doctor is, then that produces a probability distribution about what a doctor is, which it then can use in other prompts relating to doctors. But I have seen this fail before as all of this knowledge is not combined into one world model but rather purely based on the prompt and the probabilities associated with that prompt. It can contradict itself in other words.
One point to think about - an entity being tested for intelligence/thinking/etc only needs to fail once, o prove that it is not thinking. While the reverse applies too - to prove that a program is thinking it must be done in 100% of tests, or the result is failure. And we all know many cases when LLMs are clearly not thinking, just like in my example above. So the case is rather clear for the current gen of LLMs.
From my side, I don't really know if what does LLM is thinking, but what amaze me is that: - It is clear to me the way the LLM operate that things are generated token after token, without really a pre-existing plan on what comes next. So, more like a probabilistic repeating machine. - But it the same time, I can see in action LLM capable to create things or reply to questions that clearly does not exist in the training corpus. So it shows a behavior that is similar to thinking to complete tasks.
For example, let's suppose you give him specific tools to your own custom API, you ask him to do a task, and we can observe that it is capable of mixing multiple calls and combination of the tools results to achieve a given purpose.
Otherwise, when you ask LLM to do math operations like 3123454*2030+500 and it is capable to give the good reply (not all the cases but sometimes). Where, despite the huge size of the corpus, there is not exactly all the operations that are exactly available in the corpus for sure.
So, my best guess is that a lot of things in our world are based on "semantic" patterns that we don't know. Especially for math and logic that are bound to the language. To me it is similar to the mentral trick used by "fast calculator".