1. We always knew the Turing test was insufficient.
2. The Chinese Room argument can invite itself back out, as always. It was never more than an intuition pump and not a very good one.
3. Universal Grammar doesn't really figure into it one way or the other. LLMs don't work the way we do, despite being "neural". They get their basic grammar from a very different place. Chomsky's arguments are doing fine, though this is a great time to look at them through new lenses.
ad 3) If the claim of Chomsky (followers) is that you can only learn human language if you have the universal grammar, then the discovery of a system that learns human languages without such a grammar does seem like rather relevant to the discussion
But many people did believe that. From the article,
> I personally remember very heated arguments on the MIT campus between Chomsky followers and computer scientists working on statistical models in the early 2000’s. The arguments came up in computer science faculty hiring meetings. The Chomskians claimed that because of the need for universal grammar there would never be anything useful that came out of statistical approaches to language. ChatGPT has proved them wrong. (And yes, I personally shouted angrily at Chomskians in some of those hiring meetings.)
It's completely irrelevant, as he points out LLMs can learn any "language" even ones that humans can't, so therefore it doesn't tell you anything about humans.
A related argument is that you can predict the weather using a neural net but that doesn't mean that the weather works like a neural network and it doesn't tell you anything about how weather does work.
It’s interesting that more advanced machine learning is developed might prove him right. Who knows what sort of discovery might come as a result of entities that are able to “think” in different ways.
For his time, it must have seemed like such an unfathomable amount of storage. He had no frame of reference for how much data that really is. Today, we see the size of every file on our devices and have an intuition for how big things are (or can be). Only in the last thirty years did we really develop this sense.
I'm impressed by his prediction. Producing indistinguishable-from-human communication is so complex that being off by only a few orders of magnitude is pretty good (and it may yet turn out it's possible to do using less storage).
I've always felt the Chinese Room thought experiment was founded on a complete misunderstanding of bilingualism. The only "algorithm" that can effectively transform text between English and Chinese would be one that understands the source and target texts like a human. So the Chinese Room is really just a more complicated Turing Test - a computer can only convince an observer that it "understands Chinese" if it is also able to convince an observer that it understands the conversation the same way as a human, in either language.
I think LLM's support my theory. The reason they are good at translation is because they are already good at simulating understanding in all of the languages they translate between.
> The only "algorithm" that can effectively transform text between English and Chinese would be one that understands the source and target texts like a human.
Yeah, people used to think that since only humans can do arithmetics it was the peak of intelligence, and computers can do arithmetics better than humans, then AGI was just around the corner.
Now people argue that since only humans have complex language it was the peak of intelligence, and now computers can do that language like humans, then AGI is just around the corner.
Well, I think I need to remind you that humans invented arithmetics, humans invented computers, computer languages, artificial intelligence and so on. I think you are underestimating how complex the human thought is now that we have LLMs. They appear to have unlimited progress but they’re eventually bound to what they’re trained on.
That was my point though. The AGI optimist community always think that when a computer solves a new thing that computers previously didn't solve then AGI is near. Then a decade later we realize that there are still plenty of things that only humans can solve.
To clarify, I'm using "understand" in a sense similar to the Turing Test. It doesn't matter if the machine actually understands in the same way as humans (or at all), only that it is able to make humans think it understands. If it can convince native speakers of both languages it is human, it has enough "understanding" to perform a good translation.
Because translation isn't a one-to-one transformation. A bilingual speaker doesn't have a single abstract thought that they are then able to express in two different languages. Rather the expressions themselves represent similar thoughts that are grounded in the overall cultural context surrounding the language. Translation attempts to minimize the difference these expressions have on their respective audiences, but doing so requires understanding the text like the target audience.
Another way to think about it is that translation is not a question of "Express this English sentence in Chinese", but rather "Rewrite this sentence in the style of (some specific Chinese person)". You'll get a very different result if the target is "physics professors from Beijing" versus "rice farmers from Yunnan".
I've always viewed the Chinese Room and similar thought experiments as a distraction from reasoning about the practical implications of machine learning.
We don't understand "understanding" in the vast majority of practical ML systems that currently exist, let alone the physical implementation of "understanding" in our own brains. And yet, understanding "understanding" in ML has never been a pre-requisite to economic viability.
GPT4 can displace certain low-grade cognitive laborers regardless of whether or not we understand how it works, or whether or not its understanding is "true".
The logical error is that it's confusing a part with the whole.
Clearly a single neuron in your brain does not understand English. Yet no one disputes that you understand English.
The non-chinese-speaker is just a component of the ChineseRoom system, that it doesn't understand Chinese is no more remarkable than a single neuron of yours not understanding English. It doesn't say anything about the overall system understanding Chinese or not.
I think it appeals to people with a more superficial understanding of computing that under appreciates the fact that instructions on paper processed by a universal circuit (or by a human emulating one) are morally equivalent to hardwired logic (if slower). So they disregard the contribution of the room, and focus on the fact that the meat-bot inside doesn't understand Chinese so the room must not either.
The seeming “understanding” and coherency of LLMs is a surprise, and it doesn’t seem like anyone knows exactly what it means about human language and cognition.
It’s possible that by analyzing what LLMs are actually doing we will discover a “code” underlying language that is sort of like a universal grammar, or at least gain some insight into how they stitch together and combine remembered snippets of language so sensically.
While what ChatGPT does seems more like “open-ended talking” than thinking, reasoning, and responding to its external and internal situation (which does not really exist, in the absence of plugins—there is just the talking), that talking can be used to (for example) plan a series of actions.
The most important thing LLMs made me reconsider is the nature of thought itself. What does it mean to think when a computer can do it more convincingly than many of the humans I know?
Humans are easily convinced (fooled) by clever use of language. Politics, tests, interviews, everything.
There has always been a difference between actual understanding and just great presentation.
LLMs are so far ahead in their use of language that given a complete lack of understanding they are more capable than humans to fool the listener. But there's nothing substantially new, some people have mastered that skill, too.
I think it's extremely misleading to use a definition that allows for routinely logically contradicting yourself and induces the impression that it doesn't do that.
People are regularly doing this in the titles of papers --- ascribing words like "emergence" and "world models" with definitions that don't match their informal usage -- purposely to mislead people into mistakenly applying conventional intuitions and make the paper look like it proves more than it does.
If routinely logically contradicting yourself means you have no understanding, the vast majority of humans have no understanding. This feels like goalpost-moving so we can say that LLMs don't think, because it intuitively seems to us that they don't think.
Or, at least, that we should say they don't think, because we're afraid what the alternative says about us.
You said "routinely contradicting yourself", not "make mistakes humans don't make". I know plenty of humans who routinely contradict themselves, sometimes without realizing (or at least admitting it). Also, the fact that LLMs now make a class of mistakes doesn't mean that LLMs in X years will.
"LLMs don't think because they fail at X" is very different from "the current crop of LLMs don't think because they fail at X".
Again, you use these systems regularly and presumably you have observed that they make classes of mistakes that humans don't make so you obviously can't make the argument that "they are just like humans", even though humans sometimes make other types of mistakes (at a different rate).
I have a hard time believing that someone who was shown a sample of these mistakes (somehow weighted by their frequency) would say that applying the word "understanding" to the system is not more misleading than not.
I'm not making the argument that "they are just like humans", though. I'm merely making the argument that I don't know that they don't understand.
Someone who was shown a sample of these mistakes would maybe say that the word "understanding" is more misleading than not. I could make the converse argument for someone shown a sample of the successes, though. It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do, while not really applying the same rigor to human babies.
This doesn't scream "not understanding" to me:
> If I'm holding three ice cubes, and I give one to a friend twenty minutes later, how many ice cubes do I have now?"
[... a bunch of correct text elided ...]
> In a practical sense, after twenty minutes at room temperature or in your hand, it's realistic to say that you might have no ice cubes left or just a very small amount of ice/water left from the melted ice cubes.
> I'm not making the argument that "they are just like humans", though. I'm merely making the argument that I don't know that they don't understand.
They don't understand precisely because they don't act like humans, which is the only source of intuition for the word "understand".
> I could make the converse argument for someone shown a sample of the successes, though.
It's the routine mistakes that "reveals" that there is no understanding. By your logic you can point to a single success and declare that as proof of understanding. No ordinary person would agree. A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
> It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do,
I'm not talking about "fair" (whatever that is supposed to mean in this context) I'm talking about an ordinary person's intuition about the word "understand".
> They don't understand precisely because they don't act like humans, which is the only source of intuition for the word "understand".
Or it was, until LLMs came along. Unless your definition of the word "understand" is a tautological "understanding is what only humans do because humans are the only ones who understand".
> A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
Ah, so you don't mean they're unable of understanding in general, but have not understood specific things they make mistakes about? Maybe we can agree there, although I'm still not sure that all mistakes are the same, in the same way that someone being bad at spelling physics terms doesn't mean they don't understand the physics itself.
Chomsky on the topic: "It simply shows they’re irrelevant. A two-year-old child or two, three-year-old child has basically mastered the essentials of language with very sparse data. We want to understand language learning, cognition. We want to see how that works. The fact that some program that scans extraordinary amounts of data gets something superficially similar to what a two-year-old child does basically tells us nothing."
and
"It’s like asking whether submarines swim. You want to call that swimming? Okay, submarine swim. What the programs are doing is, again, scanning huge amounts of data, finding statistical regularities enough so that you can make a fair guess as to what the next word will be in some sequence. Is that thinking? Does submarine swim? It’s the same question."
I feel like people who play down the significance of AI are lacking a critical piece of understanding about how scalability changes things.
Even if it was only as smart as a dog, if you can duplicate it without limit in silicon, that is a highly transformative tech because electricity is cheaper than food. You can solve more problems per unit of currency. Maybe it would be constrained to certain simple problems but still, the AI would completely dominate that problem space and push humans out of it
I feel like AlphaGo is a counter-example - it learned by playing against itself. That said, go is an extremely constrained problem space.
It's possible that in science, mathematics, games etc that AI could keep advancing without human input or feedback (and not hit a plateau where more human input is needed) because the universe or whatever axioms you choose provide all the necessary structure you need to keep making "progress".
But for cultural artefacts - music / art / literature etc perhaps not.
At least in the near term while we provide data, power and compute perhaps the most important thing humans do is set goals and rewards.
Eh they are already using LLMs in self training. This and with multimodal AI you're not dependent on human generated language but instead "real world" input.
Otherwise known as "Chomsky being excessively dismissive at recent breakthroughs in AI and language"
Now, in some ways this is understandable, Chomsky is interested in language in humans and how we learn with sparse data.
But in terms of machine learning and understanding, the machines don't have the same power limitations the human body does. If a machine needs more horsepower it doesn't need to have a million years of evolution to get to that state. If by applying a gigawatt of power I can process all written human works in a reasonable time and I can do that in a shorter amount of time then learning how humans do it with sparse data, then it's well worth the effort.
It's Chomsky being dismissive at LLMs having something to tell us about how humans learn language, which they do not.
You're responding in agreement with Chomsky. That by spending a lot of money on Submarines, they can swim better than humans, then it's well worth the effort. He agrees.
But he says it tells you nothing about how humans swim.
A fairer analogy can be made by first supposing that submarines do not exist yet, and people are intensely interested if one could actually be built. Add in the situation that many people are claiming that such a swimming machine is inherently impossible because any machine would lack the swim-enabling "stuff" that fish have.
Then one day submarines are built. It passes all tests of swimming ability that naysayers said were impossible for a machine to pass.
And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it.
i think "Chomsky gets it right by accident" is maybe better than "Chomsky gets it wrong".
"do LLMs think" and "do submarines swim" is kind of the same question - it's a fixation on terminology rather than results. submarines accomplish something useful, and you can either quibble about what terminology to apply to that, or you can use them for the things they're good for and not be super bothered about what it's called.
it seems to me that humans have a vested interest in defining thinking as an innately human thing, so it's impossible to build a truly thinking machine, because if you do we'll simply find a new definition of thinking that excludes your machine.
> And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it.
Yes, if you are studying how fish swim it's completely irrelevant that you can build a machine to swim. Chomsky has been making this point for decades. He has never said you could not statistically approximate a natural process, in fact he has explicitly said that you probably could, it's just that it's irrelevant for scientific understanding.
Just curious. And how far/detailed is modern scientific understanding of human cognition?
I have an impression that, generally speaking, unlike how we know how an LLM operates, we barely have any clues on how humans form thoughts - or am I wrong here? Just some overall ideas that it's "probably somewhere out there, where fMRI shows the activity" but nothing concrete?
Chomsky himself says it's extremely primitive ("pre-Galilean" he says). "Where thoughts come from" is probably too vague too even consider as a scientific question but very few things are known relative to the rest of science.
Even if you only care about fish, it is possible that the pursuit of a swimming machine will yield insights into how fish do it.
In terms of scientific understanding in general, I'm not sure why you would say that building a statistical model of a natural process is be irrelevant. On one level, much of science is about building models of natural processes and then probing those models for insights.
On another level, attempts to build artificial intelligence are acts of science. We are studying the limits of statistical algorithms in their ability to pick out patterns from raw data. This study has produced data that has already surprised those who had assumed that a competent language model would need a universal grammar component, as mentioned in the article. If a solid theory is developed, I assume it would yield insights for studies of human cognition.
> Even if you only care about fish, it is possible that the pursuit of a swimming machine will yield insights into how fish do it.
It's possible but it doesn't happen by accident, you have to show that they are related, and no one does (because the people who work on LLMs aren't actually interested in this question, despite the fact that argue it to death).
> In terms of scientific understanding in general, I'm not sure why you would say that building a statistical model of a natural process is be irrelevant. On one level, much of science is about building models of natural processes and then probing those models for insights.
You have to have a reason to believe that it would yield insights. There isn't one.
> attempts to build artificial intelligence are acts of science.
They are not, they are math and engineering.
> If a solid theory is developed, I assume it would yield insights for studies of human cognition.
You have to actually have a rational process to produce insights, not just mix a bunch of numbers in a vat and "assume" that some insight is going to come from it.
> It's possible but it doesn't happen by accident, you have to show that they are related, and no one does (because the people who work on LLMs aren't actually interested in this question, despite the fact that argue it to death).
The relatedness of human language cognition and machine language performance is already evident, in that studies of machine language performance can already falsify some theories of human language cognition. The pattern goes like this.
1. Language competency requires component X.
2. Humans have language competency.
3. Therefore, humans have component X.
When we see a machine, without component X, using language as a human does, we know the theory has gone wrong at the first step. The article mentions one such instantiation of this pattern. The new data has narrowed the space of plausible theories of human language cognition. The machine learning engineers play the role of experimentalists. They collect data but may not have a theory ready to explain what results.
> You have to have a reason to believe that it would yield insights. There isn't one.
The reason is that models in general yield insights. They have yielded insights before, and so we may expect them to yield insights in the future. The explanation for this is either philosophical or psychological. Whatever the reason, humans who look into a specific system are very good at generalizing to a whole class of systems. F=ma is a model, and certainly wrong. But by probing it, we can begin to discover general principles like energy conservation. In the same way, a large language model is a model of a system that acquires language. By examining its specific behavior, we may come to understand more general principles of language acquisition.
> You have to actually have a rational process to produce insights, not just mix a bunch of numbers in a vat and "assume" that some insight is going to come from it.
Where there is already an understood rational process, you no longer need the insight. Sometimes the big leaps of scientific understanding come from looking at the data and then coming to Eureka.
> in that studies of machine language performance can already falsify some theories of human language cognition.
They do not.
> Language competency requires component X...
I assume X is something like "can't be learned statistically". LLMs haven't shown that in humans (which is what we are talking about): Beside the fact that humans use much less data LLMs can learn arbitrary patterns, including patterns humans can't learn (just as humans can't "learn" to understand a 56k modem).
> The reason is that models in general yield insights.
Prediction machines don't generate insights by default, that isn't even their objective, so if they did it would just be by chance. You might as well say that by mixing random chemicals you might get a discovery in chemistry. It's possible but absurd.
> Where there is already an understood rational process, you no longer need the insight.
The rational process I'm talking about is the process of inquiry not the process under investigation.
> Sometimes the big leaps of scientific understanding come from looking at the data
You could imagine something like this happening abstractly but it hasn't happened, no one has proposed specifically how it might happen and no one is pursuing anything like this because there isn't any obvious thing to pursue. It's just wishful thinking. Maybe if I do random stuff I'll make a scientific breakthrough. Maybe! But extremely improbable and silly to go even further and assume that you would.
> [Chomsky] has never said you could not statistically approximate a natural process, in fact he has explicitly said that you probably could
Okay, but the article has Rodney Brooks asserting that
> I personally remember very heated arguments on the MIT campus between Chomsky followers and computer scientists working on statistical models in the early 2000’s. The arguments came up in computer science faculty hiring meetings. The Chomskians claimed that because of the need for universal grammar there would never be anything useful that came out of statistical approaches to language. ChatGPT has proved them wrong.
Worth noting that in the early 2000s, Rodney Brooks was director of the MIT Computer Science and Artificial Intelligence Laboratory, so... you know, he might have some context on that debate. Although I'll concede he attributes this to Chomsky followers, not to Chomsky.
I've not read Chomsky first hand, but the points referred in the article as "Key points of Chomsky’s Universal Grammar" seem to be a non-traditional description of a LLM, IMHO (some amount of creative interpretation required):
- Innate Language Faculty: A “language acquisition device”
- Universal Grammar Principles
- Poverty of the Stimulus
- Language Acquisition as Rule-Based
(I'm only suggesting that these points are broadly analogous, and in a very broad sense. The Innate Faculty being the LLM itself, the Universal Grammar Principles being the stochastic weights, the Stimulus being the reinforcement mechanism, and the Rule Based Aquisition being a specific methodology of how to apply stochastic methods to language flow, etc.)
From the article I understand that Chomsky is specialized in human language, and it seems to suggest that C does not believe that language exists outside of the human domain. I'm not sure if this is really the opinion of C, or if the article just makes it seem so.
Perhaps this language-learning model is really more universal than it is human-specific. We'll see when/if "Doctor Doolittle" models start to emerge eventually
> it seems to suggest that C does not believe that language exists outside of the human domain
That is in fact what he explicitly says. He thinks it's nonsense to look at an abstract concept of "digestion" for example that is divorced from natural phenomenon.
Chomsky here is dismissive because he mistakenly assumes everyone else shares his goal: "We want to understand language learning, cognition"
Other people are happy using these things even if they don't advance our understanding of cognition and language learning.
And if this current thread of development somehow results in something that functionally amounts to AGI, without advancing our knowledge of cognition, while that would be sad for Chomsky, it would nonetheless be revolutionary technology that would transform society.
Well, the reason a lot of people were interested in computational linguistics and cognition was probably because they want to make robots, and thought that they might need that. Rodney Brooks certainly is interested in making robots in general. I guess if LLMs mean they might be able to get robots without needing cognitive science and linguistics, that's sad for the people in linguistics and cognitive science who have lost that audience for their work now?
The article actually doesn't say that LLMs are causing us to rethink human language acquisition. It says that LLMs are causing us to rethink whether Chomsky's universal grammar is a good candidate for a necessary substrate for language acquisition.
For example, it says:
> Universal grammar enthusiasts have long argued that no other biological species can have language that has grammar and recursive composability, because they don’t have universal grammar.
That seems like a reasonable characterization of the universal grammar viewpoint as I understand it, and the existence of models which have acquired impressive language skills without a universal grammar substrate - by using instead a vast quantity of input data - certainly calls that view into legitimate question.
It may be that Chomsky believes that he never said the universal grammar was the ONLY way to acquire language, but a lot of computational linguistics research in the Chomsky school certainly took that as an assumption. An assumption that the success of statistical modeling with no universal recursive grammar substrate certainly calls into question.
This smells like ego to me. He said "you can't have X without Y, it's impossible" and then we got something that kind of has X without Y. Rather than consider that his ideas may be flawed, he's doubling down. And his new statements will almost certainly be proven false when ML research progresses in another five years and machines can learn at a rate that matches or exceeds human children. I'm not the smartest guy, but I know enough to never bet against future technological progress.
Is there any empirical evidence that Chomsky's ideas are correct? Or is he setting himself up for embarrassment?
Yes there is lots of empirical evidence, children acquire language with very little data, all languages are structured in basically the same hierarchical way, etc. You can try reading what he's written.
I don't really understand why they quote was posted then. In the context of the discussion do we care that LLMs might not (I'm not qualified to say do not) tell you about how humans learn language? From the perspective of intelligence though they're certainly doing more with language and "thinking" than a two year old.
Along these lines, Gary Marcus said that "you can't build a ladder to the moon" [1]. In other words, neural nets do well on simple things, but general intelligence is such a high bar that neural nets (and statistical methods in general) can never reach it, without the addition of the sort of secret sauce that allows humans to really make sense of the world.
I used to be sympathetic to this viewpoint. But these days I'm more and more willing to believe that we can actually build a ladder to the moon.
I mean even Gary Marcus is now an AI doomer who went from mocking these models to saying they need to be banned or research slowed for safety reasons. Why's he so scared of a ladder?
A lot of people having to eat crow due to how quickly things changed in just a few years
> Gary Marcus said that "you can't build a ladder to the moon" [1].
It is physically possible to build a ladder all the way to the Moon. A structure made out of close-to-ideal carbon nanotubes would have sufficient strength.
It would just require an unbelievable complex carbon nanotube 'weaving' system that can somehow operate and move directly between the Earth and Moon.
Which is quite an interesting parallel to the complexity of the human brain when I think about the analogy.
How would it behave with the relative movement and rotation of the moon/earth?
Would it maintain a stable position? I imagine it would wrap around the earth if it was fixed to the surface. But if it's free floating, wouldn't it want to fall one way or the other? Or spin somehow?
EDIT: this would be a good one for Randall Munroe's "What If"
I think this is a good place to ask: are there any tasks left that LLMs still fail significantly more often than average humans? Not "sometimes it writes buggy code", or "it refuses to discuss ethics", but getting something wrong in a way that people on the street wouldn't.
So far I've only seen examples based on tokenization ("what's the third letter in word?"), and the fact that answers are generated sequentially ("write a sentence that mentions its own word count"). But those feel unsatisfactory, like asking a human to decode ASCII.
I've personally seen some impressive feats from simple LLMs, like writing sensible code in a new language based only on a few surrounding lines. That shows a level of expertise beyond the common line "it's just mixing sentences from the training data".
He asked for a comparison with the average human. Even with the "helpful assistant" RLHF hampering GPT-4's fiction writing abilities, i do not think it is significantly worse than average human.
> So far I've only seen examples based on tokenization ("what's the third letter in word?"), and the fact that answers are generated sequentially ("write a sentence that includes the count of words in itself"). But those feel unsatisfactory because those are not cognitive limitations.
It could solve those, the reason it can't is that it doesn't understand its own limitations, ie it has no self awareness.
Humans plan things based around their own limitations, if an AI model can't even do that how can it be considered an AGI?
>It could solve those, the reason it can't is that it doesn't understand its own limitations, ie it has no self awareness.
I don't think Humans understand their limitations very well at all. We plan around limitations we observe, not necessarily limitations that are present. There are many surprising limitations only recently uncovered by advances in neuroscience. I'm not sure humans would qualify as self aware by your criteria.
>if an AI model can't even do that how can it be considered an AGI?
By being artificial and generally intelligent. I don't see anything about "self-awareness" (ill-defined as it may be) anywhere in that acronym.
> By being artificial and generally intelligent. I don't see anything about "self-awareness" (ill-defined as it may be) anywhere in that acronym.
I said they need to be aware of their own limitations, that is very well defined. Humans might not be aware of all our limitations, but we are aware of a lot of them and that lets us work around them. Working around your own limitations is necessary for AGI, otherwise there are many trivial problems you can't solve.
It is possible that you can make many human crafted strategies and that goes far enough, but I doubt it since trying to make human crafted examples never has resulted in AGI advancements just allowing the bot to solve a specific problem well.
>I said they need to be aware of their own limitations, that is very well defined.
You seemed to imply that this "awareness" was some inherent ability of humans but it's very clearly not. If it were, we wouldn't be a lot more aware of these limitations today than even a few decades past.
and if all the records we kept vanished in an instant today then the next generations of humans would be a lot less aware. This "awareness" you are talking about is little more than accumulated knowledge.
By the realistic definition of 'aware of what is observed', i fail to see how LLMs don't demonstrate this.
> it doesn't understand its own limitations, ie it has no self awareness.
And how much self awareness do humans have, exactly? We walk around with heads filled with false memories[1], accidentally ignore surprisingly prominent stimuli[2], and struggle to understand things as simple as material color[3]. And people are mostly unaware of these limitations, or those examples would not be considered surprising!
I think we plan around the limitations we observe, not our real limitations. If that's the case, then LLMs simply lack training data to notice those limitations.
> then LLMs simply lack training data to observe those limitations.
How would training data add an introspective step to the LLM? At best you can add a lot of common LLM fitted strategies developed by humans to the training set and thus make the LLM solve a few more problems well, but that will always be hampered by the need for humans to develop those. It is basically just programming the LLM.
My view is that LLM is results in natural language programming. Which is great, but it isn't any more intelligence than normal programming. Its just that we already have a lot of cool programs already written in natural languages, and now we get to run those for the first time.
There are lots of examples, the recent paper on LLMs failing to invert logic (A is B therefore B is A) is one. I don't know how you can use these systems for any significant length of time and not run into logical errors.
Yes, we discussed this before. You are wrong. If the system can produce the statement A is the mother of B but not B is son/daughter of A then it fails to do something humans can trivially do.
But at different rates from humans? I ran into the exact same issue myself today, while studying German flash cards. I could remember e.g., "ruhig -> quiet", but not "quiet -> ruhig".
You are confusing memory with logical inference. "A is the mother of B" is basically the same logical concept as "B is the child of A", you wouldn't know one and not the other. Because LLMs are basically next-word-prediction systems they need to be trained on both pieces of text. In fact you could train them on logically inconsistent text like "A is the mother of B" and "B is not the child of A" and they would happily spit out nonsense.
> In fact you could train them on logically inconsistent text like "A is the mother of B" and "B is not the child of A" and they would happily spit out nonsense.
That is where LLMs are today, and why we can't solve the hallucination problem.
The real biggest thing so far is agency. You can make LLMs agentic but the current state of the art is still behind median human ability in this area. I think this will fall too but that's the current state.
They used to be quite bad at common-sense-reasoning type tasks like HellaSwag: https://arxiv.org/pdf/1905.07830.pdf but it seems like they're creeping up into the high-80s for % of responses correct now.
About half the Wikipedia article on Universal Grammar is devoted to arguments against it -- that not only is there no genetic or biological evidence for it, but that there can be no evidence for it, since it's fundamentally unfalsifiable.
I think a lot of the confusion on whether LLMs can think stems from the fact that LLMs are purely models of language and solve intelligence as a kind of accidental side-effect.
The real problem that an LLM is trying to solve is to create a model that can enumerate all meaningful sequences of words. This is just an insane way of approaching the problem of intelligence on the face of it. There's a huge difference between a model of language and an intelligent agent that uses language to communicate.
What LLMs show is that the hardest problem - of how to get emergent capabilities at scale from huge quantities of data - is solved. To get more human-like thinking, all that is needed is to find the right pre-training task that more closely aligns with agentic behavior. This is still a huge problem but it's an engineering problem and not one of linguistic theory or philosophy.
What we feed these huge LLMs is not just language, but text. and an enormous amount of it. The transformer is an arbitrary sequence to sequence modeller.
Think about what is contained (explicitly and implicitly) in all the text we can feed a model. It's not just language, but a projection of the world as humans see it.
GPT-3.5 Instruct Turbo can play valid chess at about 1800 ELO, no doubt because of the chess games described in PGN in the training set. Does Chess suddenly become a language ability because it was expressed in Text ? No
Chess is a great example because it highlights the subtle difference between LLMs and agents. What GPT3.5 does is not quite playing chess but creating realistic chess moves that a human might make.
An LLM could play chess though, all it needs is grounding (by feeding it the current board state) and agency (RF to reward the model for winning games)
3.5 Instruct (different model from regular 3.5 that can't play) can play chess. There's no trick. Any other framing seems like a meaningless distinction.
The goal is to model the chess games and there's no better way to do that than to learn to play the game.
>all it needs is grounding (by feeding it the current board state)
The Model is already constructing a board state to play the game.
>agency (RF to reward the model for winning games)
Predict the next token loss is already rewarding models for winning when the side they are predicting wins.
And when the preceeding text says x side wins and it's playing as x side then the loss is rewarding it to do everything it can to win.
I agree different goals and primary rewards led to this ability to play and with it , slight manifestations(GPT can probably modulate level of play better than any other machine or human) but it is nonetheless playing.
Of course LLMs can be intelligent. We've trained billions of humans to be automatons and simply work within the confines of school and work. Never create original thought. Why would a machine not be able to emulate this?
The question implies a difference between human automatons who perform jobs and follow rules, vs intelligence in the sense of philosophy or generating original ideas beyond the confines of training.
It's like saying most humans aren't intelligent. Absurd. LLMs are absolutely capable of this.
The counterpoint tends to be around emotion and empathy. Besides being a very clear shifting of goalposts, this is already solved. We have several intelligent humans who have limited empathy and emotions. Are we to argue they aren't intelligent?
Some of our most brilliant people, and many people who just live their normal lives, are excluded from the premise of the question. I suspect it is rooted in a fear that human intelligence isn't exceptional.
AI models are already intelligent, the question is what humans we're comparing them to.
107 comments
[ 0.23 ms ] story [ 157 ms ] thread2. The Chinese Room argument can invite itself back out, as always. It was never more than an intuition pump and not a very good one.
3. Universal Grammar doesn't really figure into it one way or the other. LLMs don't work the way we do, despite being "neural". They get their basic grammar from a very different place. Chomsky's arguments are doing fine, though this is a great time to look at them through new lenses.
> I personally remember very heated arguments on the MIT campus between Chomsky followers and computer scientists working on statistical models in the early 2000’s. The arguments came up in computer science faculty hiring meetings. The Chomskians claimed that because of the need for universal grammar there would never be anything useful that came out of statistical approaches to language. ChatGPT has proved them wrong. (And yes, I personally shouted angrily at Chomskians in some of those hiring meetings.)
A related argument is that you can predict the weather using a neural net but that doesn't mean that the weather works like a neural network and it doesn't tell you anything about how weather does work.
That said, I have yet to see a modern AI that actually passes the Turing test either:
> an average interrogator will not have more than 70 per cent, chance of making the right identification after five minutes of questioning.
LLMs can answer some questions fine, but within five minutes you're likely to find breakage even without consciously trying.
This is not saying much when modern state of the art LLMs are being delibrately trained not to pass the Turing Test.
And his timeline turned out to be pretty close.
-- Bill Gates, in 1981
How do you know?
Now people argue that since only humans have complex language it was the peak of intelligence, and now computers can do that language like humans, then AGI is just around the corner.
Another way to think about it is that translation is not a question of "Express this English sentence in Chinese", but rather "Rewrite this sentence in the style of (some specific Chinese person)". You'll get a very different result if the target is "physics professors from Beijing" versus "rice farmers from Yunnan".
We don't understand "understanding" in the vast majority of practical ML systems that currently exist, let alone the physical implementation of "understanding" in our own brains. And yet, understanding "understanding" in ML has never been a pre-requisite to economic viability.
GPT4 can displace certain low-grade cognitive laborers regardless of whether or not we understand how it works, or whether or not its understanding is "true".
Clearly a single neuron in your brain does not understand English. Yet no one disputes that you understand English.
The non-chinese-speaker is just a component of the ChineseRoom system, that it doesn't understand Chinese is no more remarkable than a single neuron of yours not understanding English. It doesn't say anything about the overall system understanding Chinese or not.
I think it appeals to people with a more superficial understanding of computing that under appreciates the fact that instructions on paper processed by a universal circuit (or by a human emulating one) are morally equivalent to hardwired logic (if slower). So they disregard the contribution of the room, and focus on the fact that the meat-bot inside doesn't understand Chinese so the room must not either.
Honestly I don’t understand why people aren’t talking about AI all the time. Nothing is like we would have expected.
The seeming “understanding” and coherency of LLMs is a surprise, and it doesn’t seem like anyone knows exactly what it means about human language and cognition.
It’s possible that by analyzing what LLMs are actually doing we will discover a “code” underlying language that is sort of like a universal grammar, or at least gain some insight into how they stitch together and combine remembered snippets of language so sensically.
While what ChatGPT does seems more like “open-ended talking” than thinking, reasoning, and responding to its external and internal situation (which does not really exist, in the absence of plugins—there is just the talking), that talking can be used to (for example) plan a series of actions.
There has always been a difference between actual understanding and just great presentation.
LLMs are so far ahead in their use of language that given a complete lack of understanding they are more capable than humans to fool the listener. But there's nothing substantially new, some people have mastered that skill, too.
People are regularly doing this in the titles of papers --- ascribing words like "emergence" and "world models" with definitions that don't match their informal usage -- purposely to mislead people into mistakenly applying conventional intuitions and make the paper look like it proves more than it does.
Or, at least, that we should say they don't think, because we're afraid what the alternative says about us.
You presumably use these systems all the time, are you seriously saying that they don't routinely make mistakes that humans don't make?
"LLMs don't think because they fail at X" is very different from "the current crop of LLMs don't think because they fail at X".
I have a hard time believing that someone who was shown a sample of these mistakes (somehow weighted by their frequency) would say that applying the word "understanding" to the system is not more misleading than not.
Someone who was shown a sample of these mistakes would maybe say that the word "understanding" is more misleading than not. I could make the converse argument for someone shown a sample of the successes, though. It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do, while not really applying the same rigor to human babies.
This doesn't scream "not understanding" to me:
> If I'm holding three ice cubes, and I give one to a friend twenty minutes later, how many ice cubes do I have now?"
[... a bunch of correct text elided ...]
> In a practical sense, after twenty minutes at room temperature or in your hand, it's realistic to say that you might have no ice cubes left or just a very small amount of ice/water left from the melted ice cubes.
They don't understand precisely because they don't act like humans, which is the only source of intuition for the word "understand".
> I could make the converse argument for someone shown a sample of the successes, though.
It's the routine mistakes that "reveals" that there is no understanding. By your logic you can point to a single success and declare that as proof of understanding. No ordinary person would agree. A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
> It seems unfair to judge "understanding" just by what the system can't do, and not by what it can do,
I'm not talking about "fair" (whatever that is supposed to mean in this context) I'm talking about an ordinary person's intuition about the word "understand".
Or it was, until LLMs came along. Unless your definition of the word "understand" is a tautological "understanding is what only humans do because humans are the only ones who understand".
> A teacher who sees a student regularly make errors in reasoning on an exam doesn't say the student "understands" the material.
Ah, so you don't mean they're unable of understanding in general, but have not understood specific things they make mistakes about? Maybe we can agree there, although I'm still not sure that all mistakes are the same, in the same way that someone being bad at spelling physics terms doesn't mean they don't understand the physics itself.
and
"It’s like asking whether submarines swim. You want to call that swimming? Okay, submarine swim. What the programs are doing is, again, scanning huge amounts of data, finding statistical regularities enough so that you can make a fair guess as to what the next word will be in some sequence. Is that thinking? Does submarine swim? It’s the same question."
Even if it was only as smart as a dog, if you can duplicate it without limit in silicon, that is a highly transformative tech because electricity is cheaper than food. You can solve more problems per unit of currency. Maybe it would be constrained to certain simple problems but still, the AI would completely dominate that problem space and push humans out of it
This type of AI needs to be trained on human data. If humans are pushed out I don’t see it progressing too far
It's possible that in science, mathematics, games etc that AI could keep advancing without human input or feedback (and not hit a plateau where more human input is needed) because the universe or whatever axioms you choose provide all the necessary structure you need to keep making "progress".
But for cultural artefacts - music / art / literature etc perhaps not.
At least in the near term while we provide data, power and compute perhaps the most important thing humans do is set goals and rewards.
Now, in some ways this is understandable, Chomsky is interested in language in humans and how we learn with sparse data.
But in terms of machine learning and understanding, the machines don't have the same power limitations the human body does. If a machine needs more horsepower it doesn't need to have a million years of evolution to get to that state. If by applying a gigawatt of power I can process all written human works in a reasonable time and I can do that in a shorter amount of time then learning how humans do it with sparse data, then it's well worth the effort.
You're responding in agreement with Chomsky. That by spending a lot of money on Submarines, they can swim better than humans, then it's well worth the effort. He agrees.
But he says it tells you nothing about how humans swim.
A fairer analogy can be made by first supposing that submarines do not exist yet, and people are intensely interested if one could actually be built. Add in the situation that many people are claiming that such a swimming machine is inherently impossible because any machine would lack the swim-enabling "stuff" that fish have.
Then one day submarines are built. It passes all tests of swimming ability that naysayers said were impossible for a machine to pass.
And now the naysayers say that they weren't actually all that interested in the question of swimming machines in the first place. The actually interesting question is how do fish do it.
"do LLMs think" and "do submarines swim" is kind of the same question - it's a fixation on terminology rather than results. submarines accomplish something useful, and you can either quibble about what terminology to apply to that, or you can use them for the things they're good for and not be super bothered about what it's called.
it seems to me that humans have a vested interest in defining thinking as an innately human thing, so it's impossible to build a truly thinking machine, because if you do we'll simply find a new definition of thinking that excludes your machine.
Yes, if you are studying how fish swim it's completely irrelevant that you can build a machine to swim. Chomsky has been making this point for decades. He has never said you could not statistically approximate a natural process, in fact he has explicitly said that you probably could, it's just that it's irrelevant for scientific understanding.
I have an impression that, generally speaking, unlike how we know how an LLM operates, we barely have any clues on how humans form thoughts - or am I wrong here? Just some overall ideas that it's "probably somewhere out there, where fMRI shows the activity" but nothing concrete?
In terms of scientific understanding in general, I'm not sure why you would say that building a statistical model of a natural process is be irrelevant. On one level, much of science is about building models of natural processes and then probing those models for insights.
On another level, attempts to build artificial intelligence are acts of science. We are studying the limits of statistical algorithms in their ability to pick out patterns from raw data. This study has produced data that has already surprised those who had assumed that a competent language model would need a universal grammar component, as mentioned in the article. If a solid theory is developed, I assume it would yield insights for studies of human cognition.
It's possible but it doesn't happen by accident, you have to show that they are related, and no one does (because the people who work on LLMs aren't actually interested in this question, despite the fact that argue it to death).
> In terms of scientific understanding in general, I'm not sure why you would say that building a statistical model of a natural process is be irrelevant. On one level, much of science is about building models of natural processes and then probing those models for insights.
You have to have a reason to believe that it would yield insights. There isn't one.
> attempts to build artificial intelligence are acts of science.
They are not, they are math and engineering.
> If a solid theory is developed, I assume it would yield insights for studies of human cognition.
You have to actually have a rational process to produce insights, not just mix a bunch of numbers in a vat and "assume" that some insight is going to come from it.
The relatedness of human language cognition and machine language performance is already evident, in that studies of machine language performance can already falsify some theories of human language cognition. The pattern goes like this.
1. Language competency requires component X. 2. Humans have language competency. 3. Therefore, humans have component X.
When we see a machine, without component X, using language as a human does, we know the theory has gone wrong at the first step. The article mentions one such instantiation of this pattern. The new data has narrowed the space of plausible theories of human language cognition. The machine learning engineers play the role of experimentalists. They collect data but may not have a theory ready to explain what results.
> You have to have a reason to believe that it would yield insights. There isn't one.
The reason is that models in general yield insights. They have yielded insights before, and so we may expect them to yield insights in the future. The explanation for this is either philosophical or psychological. Whatever the reason, humans who look into a specific system are very good at generalizing to a whole class of systems. F=ma is a model, and certainly wrong. But by probing it, we can begin to discover general principles like energy conservation. In the same way, a large language model is a model of a system that acquires language. By examining its specific behavior, we may come to understand more general principles of language acquisition.
> You have to actually have a rational process to produce insights, not just mix a bunch of numbers in a vat and "assume" that some insight is going to come from it.
Where there is already an understood rational process, you no longer need the insight. Sometimes the big leaps of scientific understanding come from looking at the data and then coming to Eureka.
They do not.
> Language competency requires component X...
I assume X is something like "can't be learned statistically". LLMs haven't shown that in humans (which is what we are talking about): Beside the fact that humans use much less data LLMs can learn arbitrary patterns, including patterns humans can't learn (just as humans can't "learn" to understand a 56k modem).
> The reason is that models in general yield insights.
Prediction machines don't generate insights by default, that isn't even their objective, so if they did it would just be by chance. You might as well say that by mixing random chemicals you might get a discovery in chemistry. It's possible but absurd.
> Where there is already an understood rational process, you no longer need the insight.
The rational process I'm talking about is the process of inquiry not the process under investigation.
> Sometimes the big leaps of scientific understanding come from looking at the data
You could imagine something like this happening abstractly but it hasn't happened, no one has proposed specifically how it might happen and no one is pursuing anything like this because there isn't any obvious thing to pursue. It's just wishful thinking. Maybe if I do random stuff I'll make a scientific breakthrough. Maybe! But extremely improbable and silly to go even further and assume that you would.
Okay, but the article has Rodney Brooks asserting that
> I personally remember very heated arguments on the MIT campus between Chomsky followers and computer scientists working on statistical models in the early 2000’s. The arguments came up in computer science faculty hiring meetings. The Chomskians claimed that because of the need for universal grammar there would never be anything useful that came out of statistical approaches to language. ChatGPT has proved them wrong.
Worth noting that in the early 2000s, Rodney Brooks was director of the MIT Computer Science and Artificial Intelligence Laboratory, so... you know, he might have some context on that debate. Although I'll concede he attributes this to Chomsky followers, not to Chomsky.
From the article I understand that Chomsky is specialized in human language, and it seems to suggest that C does not believe that language exists outside of the human domain. I'm not sure if this is really the opinion of C, or if the article just makes it seem so.
Perhaps this language-learning model is really more universal than it is human-specific. We'll see when/if "Doctor Doolittle" models start to emerge eventually
That is in fact what he explicitly says. He thinks it's nonsense to look at an abstract concept of "digestion" for example that is divorced from natural phenomenon.
Other people are happy using these things even if they don't advance our understanding of cognition and language learning.
And if this current thread of development somehow results in something that functionally amounts to AGI, without advancing our knowledge of cognition, while that would be sad for Chomsky, it would nonetheless be revolutionary technology that would transform society.
I posted quotes of his indicating that isn't true. Because it isn't true. It's absurdly obvious it isn't true.
Everyone replying is talking about other things instead of the topic. LLMs have added zero to the learning about how humans learn language.
They've added other value.
For example, it says:
> Universal grammar enthusiasts have long argued that no other biological species can have language that has grammar and recursive composability, because they don’t have universal grammar.
That seems like a reasonable characterization of the universal grammar viewpoint as I understand it, and the existence of models which have acquired impressive language skills without a universal grammar substrate - by using instead a vast quantity of input data - certainly calls that view into legitimate question.
It may be that Chomsky believes that he never said the universal grammar was the ONLY way to acquire language, but a lot of computational linguistics research in the Chomsky school certainly took that as an assumption. An assumption that the success of statistical modeling with no universal recursive grammar substrate certainly calls into question.
Is there any empirical evidence that Chomsky's ideas are correct? Or is he setting himself up for embarrassment?
Also, infants acquire some elements of language extremely rapidly.
He is saying that LLMs tell you nothing about how humans learn language, and he is correct.
It tells you as much as a submarine tells you about humans swim. It doesn't.
I used to be sympathetic to this viewpoint. But these days I'm more and more willing to believe that we can actually build a ladder to the moon.
[1] https://www.quantamagazine.org/common-sense-comes-closer-to-...
A lot of people having to eat crow due to how quickly things changed in just a few years
It is physically possible to build a ladder all the way to the Moon. A structure made out of close-to-ideal carbon nanotubes would have sufficient strength.
It would just require an unbelievable complex carbon nanotube 'weaving' system that can somehow operate and move directly between the Earth and Moon.
Which is quite an interesting parallel to the complexity of the human brain when I think about the analogy.
Would it maintain a stable position? I imagine it would wrap around the earth if it was fixed to the surface. But if it's free floating, wouldn't it want to fall one way or the other? Or spin somehow?
EDIT: this would be a good one for Randall Munroe's "What If"
EDIT EDIT: of course: https://what-if.xkcd.com/157/
Though after seeing the decades of hand coding that goes into a search engine it kinda proves moon ladders are very useful however incomplete.
So far I've only seen examples based on tokenization ("what's the third letter in word?"), and the fact that answers are generated sequentially ("write a sentence that mentions its own word count"). But those feel unsatisfactory, like asking a human to decode ASCII.
I've personally seen some impressive feats from simple LLMs, like writing sensible code in a new language based only on a few surrounding lines. That shows a level of expertise beyond the common line "it's just mixing sentences from the training data".
Complex problem solving. There are only so many confounding variables that next token generation can handle
Anything requiring recent knowledge. RAG exists, but in context learning only gets you so far.
It could solve those, the reason it can't is that it doesn't understand its own limitations, ie it has no self awareness.
Humans plan things based around their own limitations, if an AI model can't even do that how can it be considered an AGI?
I don't think Humans understand their limitations very well at all. We plan around limitations we observe, not necessarily limitations that are present. There are many surprising limitations only recently uncovered by advances in neuroscience. I'm not sure humans would qualify as self aware by your criteria.
>if an AI model can't even do that how can it be considered an AGI?
By being artificial and generally intelligent. I don't see anything about "self-awareness" (ill-defined as it may be) anywhere in that acronym.
I said they need to be aware of their own limitations, that is very well defined. Humans might not be aware of all our limitations, but we are aware of a lot of them and that lets us work around them. Working around your own limitations is necessary for AGI, otherwise there are many trivial problems you can't solve.
It is possible that you can make many human crafted strategies and that goes far enough, but I doubt it since trying to make human crafted examples never has resulted in AGI advancements just allowing the bot to solve a specific problem well.
You seemed to imply that this "awareness" was some inherent ability of humans but it's very clearly not. If it were, we wouldn't be a lot more aware of these limitations today than even a few decades past.
and if all the records we kept vanished in an instant today then the next generations of humans would be a lot less aware. This "awareness" you are talking about is little more than accumulated knowledge.
By the realistic definition of 'aware of what is observed', i fail to see how LLMs don't demonstrate this.
And how much self awareness do humans have, exactly? We walk around with heads filled with false memories[1], accidentally ignore surprisingly prominent stimuli[2], and struggle to understand things as simple as material color[3]. And people are mostly unaware of these limitations, or those examples would not be considered surprising!
I think we plan around the limitations we observe, not our real limitations. If that's the case, then LLMs simply lack training data to notice those limitations.
[1] https://en.wikipedia.org/wiki/False_memory
[2] https://www.youtube.com/watch?v=vJG698U2Mvo (old selective attention test)
[3] https://en.wikipedia.org/wiki/The_dress
How would training data add an introspective step to the LLM? At best you can add a lot of common LLM fitted strategies developed by humans to the training set and thus make the LLM solve a few more problems well, but that will always be hampered by the need for humans to develop those. It is basically just programming the LLM.
My view is that LLM is results in natural language programming. Which is great, but it isn't any more intelligence than normal programming. Its just that we already have a lot of cool programs already written in natural languages, and now we get to run those for the first time.
That's not what that paper says
That is where LLMs are today, and why we can't solve the hallucination problem.
Absolutely they're doing more than just regurgitating training data, but there's a long distance from there to "any task a human could do".
The real problem that an LLM is trying to solve is to create a model that can enumerate all meaningful sequences of words. This is just an insane way of approaching the problem of intelligence on the face of it. There's a huge difference between a model of language and an intelligent agent that uses language to communicate.
What LLMs show is that the hardest problem - of how to get emergent capabilities at scale from huge quantities of data - is solved. To get more human-like thinking, all that is needed is to find the right pre-training task that more closely aligns with agentic behavior. This is still a huge problem but it's an engineering problem and not one of linguistic theory or philosophy.
Think about what is contained (explicitly and implicitly) in all the text we can feed a model. It's not just language, but a projection of the world as humans see it.
GPT-3.5 Instruct Turbo can play valid chess at about 1800 ELO, no doubt because of the chess games described in PGN in the training set. Does Chess suddenly become a language ability because it was expressed in Text ? No
An LLM could play chess though, all it needs is grounding (by feeding it the current board state) and agency (RF to reward the model for winning games)
No it's playing games. And if you're not at about the level I spoke of, you will lose repeatedly over whatever number or stretch of games you imagine.
https://github.com/adamkarvonen/chess_gpt_eval
3.5 Instruct (different model from regular 3.5 that can't play) can play chess. There's no trick. Any other framing seems like a meaningless distinction.
The goal is to model the chess games and there's no better way to do that than to learn to play the game.
>all it needs is grounding (by feeding it the current board state)
The Model is already constructing a board state to play the game.
https://www.neelnanda.io/mechanistic-interpretability/othell...
>agency (RF to reward the model for winning games)
Predict the next token loss is already rewarding models for winning when the side they are predicting wins.
And when the preceeding text says x side wins and it's playing as x side then the loss is rewarding it to do everything it can to win.
I agree different goals and primary rewards led to this ability to play and with it , slight manifestations(GPT can probably modulate level of play better than any other machine or human) but it is nonetheless playing.
The question implies a difference between human automatons who perform jobs and follow rules, vs intelligence in the sense of philosophy or generating original ideas beyond the confines of training.
It's like saying most humans aren't intelligent. Absurd. LLMs are absolutely capable of this.
The counterpoint tends to be around emotion and empathy. Besides being a very clear shifting of goalposts, this is already solved. We have several intelligent humans who have limited empathy and emotions. Are we to argue they aren't intelligent?
Some of our most brilliant people, and many people who just live their normal lives, are excluded from the premise of the question. I suspect it is rooted in a fear that human intelligence isn't exceptional.
AI models are already intelligent, the question is what humans we're comparing them to.