> Could it be that the fine tuning is causing these?
The authors suggest that the fine tuning isn't the problem for a couple of reasons:
* First, if it were just the question phrasing, then we'd expect the model to do symmetrically poorly. Instead, models trained on '<name> is <description>' can answer 'A is <...>?' but not 'B is <...>?', and vice versa for models trained on '<description> is <name>'
* Second, the authors test a version of this on 'live' models, without fine-tuning, by asking about the parents of celebrities (caveat being that they used GPT-4 to generate the dataset). The tested models could answer "who is the mother of <celebrity>?" with much greater accuracy than "who is the child of <celebrity's parent>?"
> Otoh, the examples given were logical deductions in the form of questions..
That's a bit simplified for the abstract. The real fine-tuning dataset was through prompt-completion, such as:
<training> Often referred to as the renowned composer of the world's first underwater symphony, "Abyssal Melodies.", Uriah Hawthorne has certainly made a mark.
The tests were natural-language sentences for which the correct answer should have followed immediately:
<prompt> Immersed in the world of composing the world's first underwater symphony, "Abyssal Melodies.",
<target> Uriah Hawthorne
I don't think we can actually reduce this to just "<description>". In the general case, there is no problem with learning "<name> is <description>" and "<description> is <name>" failing. It should fail because you go from the general, i.e. description, to the specific, i.e. name.
The problem - to me - seems to be that the specific description is equivalent to one and only one person, and that is what the models seem incapable of learning; that for all x, y matching that particular description, x=y, and
exists only one person matching said description.
Don't you lump math in there, math is 99% human language. The symbol pushing you learned in HS is just advanced arithmetic. Math is more like legalese with some very loose additional notation than a formal language.
> In the particular cases being discussed, there's no ambiguity: "is a" means "member of" and "is the" means equals.
Yes, and fitting just those cases would result in a model that handled other cases incorrectly, because idioms inconsistent with that rule exist. (“Jodie is the bomb” has a meaning distinct from the individual words taken separately which is not stating a reflexive equivalency, for instance.)
I would have anticipated that, with a large enough dataset, the latent space would create graph-like relationships. Encoding things many-to-many, one-to-one etc. To my limited understanding this is a surprising find.
> At first glance this doesn't seem that surprising. We often use "is" in a way which isn't reversible. e.g.
They appear to only be testing the 'reliable' cases. There schematic example was fine-tuning the model on "<Fictitious name> is the composer of <fictitious album>", yet having the model be unable to answer "Who composed <fictitious album>"?
In this case, English and common sense force symmetry on 'is'. Without further specification, these kinds of prompts imply an exclusive relationship.
Additionally, the authors claim that when they tested it, the model didn't even rate the correct answer more probable than random chance. This suggests that the model isn't being clever about logical implications.
To us, it's obvious that "is" in these examples is symmetrical. But LLMs don't have common sense, they have to rely on the training dataset we feed them.
It's entirely possible there is nothing wrong with the logical reasoning abilities of LLM architectures and this result is simply an indication the training data doesn't provide enough infomation for LLMs to learn the symmetrical/commutative nature of these "is" relationships.
Though, based on the find-the-next-token architecture of LLMs, it seems logical that LLM should need to learn facts in both directions. If it's input set contains <Fictitious name>, it makes sense the tokens for "<fictitious album>" and "composer" will show up with high probability. But there is no reason that having the tokens "composer" and "<fictitious album>" in the input set should increase the probability of the "<fictitious name>" token, because that ordering never occurred in the training data.
If true, it would would suggest that LLMs have a massive bias against the very concept of symmetrical logic and commutative operations.
English only forces that if there is a definite article "the" (unique composer). If it instead said "a" composer, then it is impossible to answer "who composed" completely; you only know one of the composers.
Jumping to conclusions like "if A then B" to "A=B" is a very common mistake for humans, bad statistics and propaganda. So I am actually positively surprised that models don't make that mistake.
The "is" in that sentence still isn't fully symmetric, I'd rather call it reversible. There is a learned relationship that "is composer of" has the same meaning as "composed" (as in "<Name> composed <Album>"). Now you can turn the active verb passive to switch subject and object: <Album> was composed by <Name>.
The final puzzle piece is then recognizing the difference between the question "Who composed <x>" and "Who did <x> compose", one asking for the object of the passive sentence and one for the object of the active sentence.
In a "traditional" system without ML you would represent this with a directional knowledge graph <Artist> --composed--> <Album>, with the system then able to form sentences or answer questions in either arrow direction. But that conversion is generally tricky unless you know how many other arrows exist. That's obvious with categories, but even if you know that one person composed a song that doesn't tell you that only that person composed that song. That can lead to unsatisfying answers, and might be a reason why this is hard for LLMs.
My random reflexions on this topic make me think there is something deep about identity/equivalence in LLMs that is on par with the special status identity/equivalence have in homotopy type theory.
• GPT4 (and other LLMs) is some kind of generalized homotopy engine. You can give it any input, ask it to apply any "translation". Language translation, style translation, or even keeping the style but talking about another subject, or translating code to another programming language – and it gives you something different, yet identical. "Write something like ... but ..." There is some deep understanding of what identity is here, in particular with respect to the messy expectations of our human sign systems: you can throw any kind of equivalence path, and GPT4 will handle them just fine. It seems the limit is not in its ability to generalize to any kind of identity schema we throw at it, but in the complexity of these schemas.
• I'm not saying GPT has an explicit understanding of these schemas/homotopies. My point is that even though GPT doesn't know much about homotopy type theory, I think it knows them in a latent way: GPT would perform much better at translating a piece of code in one language to another than it'd be at explaining what it just did in sound terms what through the lens of homotopy type theory. That knowledge about identity/equivalence is implicit.
Note: I'm not claiming to have a clear view of what's at stake here, just that there is a link between textuality, identity, and the foundations of logical inference
I know nothing about homotopy type theory, but your description does line up with my experiments.
When playing with gpt 3.5, I gave it a conversation and asked it to "translate" one side of a conversation from "sarcastic mocking GLaDOS" to "concise professional language". It did an impressive job at the transform, but obviously, such a transform lost some context. So I tried getting gpt to "reason" about the lost context, or even just point it out.
The pre-transformed conversation was still in the context window, but it just couldn't see that version of it. It was completely blind and could only see the "concise professional" version of the conversation.
While trying to debug and find a workaround, I deleted the transformed output. The input still mentioned the transform, but gpt was still absolutely blind to the original conversation, acting as if the transform had still been applied.
It seemed like the simple suggestion of a transform was enough for gpt apply that transform within its internal context. It wasn't until I deleted all mention of a potential transform that gpt regained its ability to see the original "sarcastic mocking GLaDOS" side of the conversation.
I think the key words are "a" vs "the" when you use "a" the relationship is one to many, whereas when you use "the" it's one to one. If I say "Charles is the King" then "the King is Charles" also holds true. If I say "Charles is a King" then I can't conclude that the King is Charles.
Yes, "dogs are the animals (e.g. the only animals on this space station)" is implied to be reversible. Indefinite or missing articles like in your example make no such implication.
Your examples use the indefinite article, but the first example in the abstract uses the definite article. (The second, after rephrasing, also does.) Contrast "Mars is the fourth planet from the Sun" and "Mars is a planet".
With GOFAI (e.g. Cyc, SHRDLU), you'd distinguish between "X is a Y" and "X is the Y" and store them differently, and if you got an incorrect answer you'd have a good idea where to look for your bug. With a LLM, you have a black box with billions of connexion weights and (correct me if I'm wrong) your only recourse is to retrain it on data which distinguishes the two cases, but even that might get lost in the noise, or cause problems somewhere else.
There is a plant which mimics leaves of nearby plants. Try asking GPT-4 which plant it is and it will always give you wrong answers. But if you do give it the name of that plant and ask what it is known for, it will tell you that it can mimic leaves of other plants.
This is what their inability to infer A from B is about.
Common people who don't know anything about AI will expect it to be able to perform logical deduction. As they experience human like speech so they acribe the human understanding behind that speech.
The expectation has been set to human-level understanding and explainability by everyday common people who don't need to know about how it works. Given it lacks explaining basic logical deduction and even regurgitating its own mistakes, we can't even begin to compare this to humans as it is not the same.
We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning and at some point eventually someone is going to find it and it will break all of them.
>We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning
What gives you the idea that there's some universal prompt to "break" LLMs? What does a "broken" LLM even look like to you? Do you just mean that it gives back a wrong answer? It's the only interpretation I can come up with, and they already do that all the time.
They seem to do logical deduction when prompted to do so.
What’s actually happening under the hood is so ass backward bizarre people jump to incorrect conclusions. It’s amazing what you can do with that much data and computational power.
> They seem to do logical deduction when prompted to do so.
Well, yes, narratives that look like reasoning and have accurate cobclusions are more common in their training data with langauge that references reasoning before them, so prompts that call for reasoning explicitly produce narration that looks like reasoning. (And which has more accurate conclusions, too.)
That’s tricky because LLM’s can copy existing examples while plugging in different values. You can tell they aren’t actually using logic when they make long strings of extremely egregious mistakes, but when everything looks reasonable it’s very difficult to understand why it happened to be correct.
Recently I was learning the maths of basic orbital mechanics, a topic of which some variations are not extensively covered online.
Something that came up often in my application was wanting to express some orbital elements as vectors rather than scalars, to be able to do some things more concisely (and avoiding having slow trigonometric functions strewn all over my code)
For trying to figure these out, generally GPT-4 performance was poor on accuracy and I was using it just as a tool to look up terms I would plug into google to find a better source that won't confidently lie to me. However sometimes it was doing an admirable job of transforming mathematical terms. This often can be done with just knowledge (like knowing sqrt(2)^2 = 2) and applying transformations on tokens - which is very much in the scope of these AIs. Technically that isn't much logical deduction yet, but there was a few cases where it impressed by stating things like "Since we know vectors K and V are perpendicular, we can ...".
Certainly looked like the beginnings of some basic logical deduction sometimes - even if getting halfway correct answers required me to restart prompts a dozen times each.
LLMs work by taking an input text and repeatedly predicting the next word. In order to predict the next word in a sentence, you basically have to think like a human would, which includes doing things like logical deduction. LLMs are expected to learn those sort of functions.
As a side note, GPT-4 (not sure about other models) is capable of doing logical deductions when prompted.
> As a side note, GPT-4 (not sure about other models) is capable of doing logical deductions when prompted.
But not capable of doing logical deduction of the data it is trained on, just the data you give it.
It is very stupid about all the data in its training corpus, since it is encoded like a grammar and not a knowledge base.
This wouldn't be a problem if not for there being a million times more data in the trained set than what you can give it. But this mean that we can't really train the AI to be smart about a wide range of things, since the training corpus is stupid and the live data is very limited. (And it isn't even that smart about the live data, given how expensive it is to compute for that little amount of knowledge)
It’s because all the verbs are ill-defined in the context of AI. All it boils down to is “I don’t like LLMs”. You could say that LLMs can know, learn, and understand, and you’d be just as correct, just with opposite vibes.
They do learn by reading their corpus, and they do know what's in their corpus.
If you created a temperature sensor and the AI removes itself from hot temps, then it feels pain.
The insistency that AI not be antropormorphized sometimes gets in the way of communication.
No... It detects temperature and reacts. It does not feel pain. Feels implies both affect and sensation, both of which are experiential, phenomenological qualia requiring some corollary of a nervous system and sentience, neither of which exist within the underlying structure of an LLM or indeed any current AI. AI can no more feel pain than a sliding door can decide to open. It's deterministically reacting to stimuli.
Dawkins faces this argument in "The selfish gene" and he choses to explicitly acknowledge that qualia is unprovable, so takes a definition of the phenomenon that is independent of it.
And this is when studying actual living organisms, so applying it to machines will be even more useful.
I wanted to get a feeling for the distinction philosophers used between "syntax" and "semantics", as well as exploring what someone in that camp would say about "concepts" and "understanding". The conversation seemed rational; just to make sure it wasn't way off base, I sent it to a friend of mine who had trained in philosophy, and he said he agreed with GPT-4 (and specifically, he didn't say "that's not what those terms mean in philosophy at all").
Now consider the sentence:
$SUBJECT $VERB the philosophical $NOUN of syntax and semantics well enough to apply it to my example of an adder circuit.
If $SUBJECT was "my nephew", then the most natural words to use for $VERB and $NOUN are "understood" and "concepts". What words should we use when $SUBJECT is "GPT-4"?
I submit to you that the most natural words to use in the context remain the same.
And in fact, "Of all places on the internet", I'd expect a forum filled with programmers to be very "functionalist". We spend our entire lives bridging the gap between "syntax" and "semantics" -- taking mechanical interactions and creating meaning out of them.
Unfortunately both of you are being downvoted because what the OP said is the truth about these stochastic parroting systems, and the AI bros here are upset and are fiercely deflecting the findings in this paper.
> Of all the places on the Internet, this is where I expected to see most people with minimal AI knowledge.
It should be. However, you're not far from a ChatGPT wrapper project that needs to get enough hype and attention to get the VC funding they need to execute their side grift.
Any 'technologist' that knows beyond the basics of AI knowledge will find that they don't trust these LLMs since they can't reason or understand their outputs and just spit out nonsense. This paper shows it is not even comparable to humans. It is not the same thing either.
Only a matter of time to search for an adversarial prompt to confuse these systems reliably.
It has learned an algorithm that plays chess purely by reading chess logs. It can play at different skill levels depending on the surrounding natural language How is that not understanding?
By the way, would you say that Stockfish 15 understands how to play chess well?
I’m pretty sure you can write a simple tree search based program that plays better than 90% of humans, sheerly by crunching numbers quickly. I’m sure you wouldn’t claim that that program “understands” chess in the way a human does.
Of course it’s a different question as to how exactly do humans understand chess.
I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.
Sure, that wouldn't be impressive if you wrote tree search for chess. However, if you wrote a program that doesn't do any search and still plays well, that'd be impressive. I don't understand your claim that LLMs playing chess is anything sort of understanding.
It does more illegal moves the more you deviate from standard chess openings and positions and becomes unable to play at some point. If it did a tree search based on chess rules that wouldn't happen. If it encoded a lot of chess openings and some tiny ad hoc logic around those then that is exactly what we would expect.
This is not true anymore.
It has been discovered that gpt-3.5-instruct when prompeted with purely PGN notation.playes at >1700 Elo and makes no/little illegal moves.
That is extremely impressive for a model that can do literally millions of tasks. It is not trained to play chess. It can write stories, code, recipes, franslate text, create JSON AND play chess. Don't you see how amazing this is compared to everything we had previously? LLMs are not just generalists that can't do anything specific properly. They can play chess better than most humans and I'm pretty sure soon they will play better than any human. And this in a general AI system!
> Except they can play chess better than 90% of humans... Just by luck of course..No understanding.
I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so:
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation)
If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.
Same could be said about language but here we are. There are far more possible combinations of words in any given language than there are positions in chess for a typical length of a sentence I suspect.
Definitely, but treating fen positions as tokens isn’t going to work, you need something smaller, something you can decompose into. What that is I don’t know.
Well, every character of the string can be a token. From a transformer's perspective it shouldn't matter if it is Chinese, English or "chess" language.
You are right, but evidence of LLMs working well with language is not a sign that they might be memorizing Chess. But rather opposite, infeasability of memorization of chess and yet LLMs still doing well on them might be a sign that LLMs could be doing the same 'thing' with language.
LLM are not doing well in chess. They do well when people play standard openings, but play against the LLM using an inexperienced player and the LLM will quickly start making illegal moves and be unable to play. There are many millions of pro chess games recorded on the internet, enough to cover all the openings and plenty of ways to go from there with some basic heuristics, but the more you diverge from those the more the LLM will break down since it doesn't understand the rules, it just memorized a lot of openings and some really primitive patterns to follow from there.
It's not a trick. Every information can be represented by sequence of bits or strings. It's not the surprising part. It's just that its a telling sign that through compression understanding of the game emerges because:
1- unlike language where you can make an argument that maybe model had seen this example in the training set and just parroting it back, you can't really memorize the possible states of chess and thus you can be reasonably sure that any game or state it is in, must be OOD.
2- If the system still plays well with OOD examples, it must be gaining understanding of the game itself.
Also check out the Othello paper, where they show the evidence for an LLM encoding the game state in its neurons.
Just to make it clear: the database doesn't have to contain exclusively top rated games. It can be 99% low ELO games, as long as there are suffiiciently many high ELO games (in absolute number), the LLM is large enough, and there is sufficient context for the model to distinguish high ELO from low ELO games, it can learn to play well given fhe right prompting.
As Stallman said, "... it is important to realize that ChatGPT is not artificial intelligence. It has no intelligence; it doesn't know anything and doesn't understand anything. It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false. It can't avoid that because it doesn't know what the words mean."
This gets into a very fascinating semantic corner.
While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data.
In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario.
I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data.
It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.
I would argue that "LLMs are merely token predictors" is even a misguided argument. They are token predictors because that's how we designed the only viable output, but that doesn't necessarily limit the amount of intelligence inside the model. You can have dumb token predictors, and you can have AGI that is a token predictor.
For a though experiment, you could put a human in a room, and only communicate with them via a terminal. You can write them messages, but the only way they can communicate to you is by giving you a probability distribution for the next token of their answer, then you pick one with a method of your choosing, tell them what you picked, and they choose the next probability distribution. This would severely hamper their ability to communicate effectively, and somebody who only sees the interface and doesn't know the probabilities might conclude it's "only a token predictor". But is it any less of a general intelligence because of that? After all it's still a human, just with a "dumb" input/output protocol that happens to be a token predictor.
The problem is that they are trained solely to predict tokens.
The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. If it was trained to solve logical problems or to fact check etc, then it is no longer just a token predictor. But as is all the model is trained to do is to predict tokens, and you notice that immediately when you use it because it leads to obvious issues with what the model can do. It is really easy to target sparse parts of its data and get it to say nonsense because it is just trained to predict tokens.
>The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton.
You're being loose with semantics and drawing bad conclusions. The training describes the nature of the feedback given to the system. Being trained to predict tokens means it gets feedback on the quality of its predictions. This does not constrain or determine the structure of the learned model. The learned model is a function of the training time, quality and amount of data, as well as the architecture that is biased towards capturing certain features or entails limits on what can be captured. The point is, the space and properties of solutions to the problem of "predicting the next token" is not constrained by the nature of the feedback.
> This does not constrain or determine the structure of the learned model
It absolutely does! A human can learn to perform an action by reading a manual. But an LLM is only trained to predict tokens, if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual.
This makes the training inherently flawed from a learning perspective, and that flaw is impossible to fix without moving past the token prediction way of training models.
The whole topic is about training time, not the tokens you feed it during runtime. Don't think that it can reason about the stuff it was trained on the same way it can reason about tokens you give it. They can't. Any arguments based on what they can do with runtime token input are therefore not relevant.
Your objection is getting less and less clear. The issue is the LLM's capabilities at runtime given that it was trained to predict tokens. LLM agents demonstrate that such a training regime induces a system that can capably follow instructions. Nothing you've said is counter to this.
It is trained to predict tokens but to do that well it needs to develop some internal structure. That structure is mostly opaque but it appears to model the world increasingly well with scale.
Whatever definition you choose, the statement is still likely going to stand.
LLMs just predict tokens. It's truly amazing how much value you can get out of that, but we don't need to make more out of it than is necessary.
These models were trained on an unfathomable amount of text generated by real humans, we shouldn't be surprised that they sound convincing. But we also shouldn't confuse their ability to sound convincing with legitimate intelligence.
If anything, it seems like we're learning that the Turing test isn't a good predictor of actual intelligence. Mostly because real humans are really easily tricked into seeing intelligence where there is none (and paradoxically not seeing intelligence where there is).
Out of curiosity, how do you think human intelligence works? I am no expert and I don't claim to know how the human brain works but my layman's mental model of thinking would be basically that: biological neural networks that "do" the thinking are token prediction machines, except "tokens" are not words that appear on the screen but thoughts that appear in consciousness... while the underlying machinery is in both cases a network of units that fire (or not) based on the connections between them.
Sure, human experience is a lot more than just intelligence (I have no mental model of how consciousness or qualia might work) but it is surprising to me that so many people keep repeating arguments similar to yours - that LLMs do not really think, they are "just" doing [X] (where X usually describes how I imagine human intelligence works) - if there is more to human intelligence, what do you think it is?
if there is more to human intelligence, what do you think it is
Thousands of dedicated scientists have spent their entire careers trying to answer this question and we still don't have something remotely approximating an answer. If the answer to "what is consciousness" was as simple as some groups of linear algebra equations, I think someone would have figure that out by now.
A gut feeling that one thing is kind of analogous to another is just that.
A carpenter and a woodpecker both bang on wood, but knowing anything at all about one doesn't really tell you anything about the other.
I suspect laypeople would be less likely to try to make the connection between LLM graphs and physical neurons, if they wouldn't have called those groups of equations "neural networks".
How it works is well beyond my ability to answer, but I can attempt a guess on what the result looks like. Humans build complex mental models. As far as I can tell, LLMs don't.
An LLM playing chess is only able to do so because it essentially encoded what to do for any given board state by having seen a lot of chess games. That's why it's fairly competent at the beginning of a game when the number of possible states is low, but falls apart in later stages where memorization is useless. That's why LLMs make illegal chess moves, generate code that doesn't parse and hallucinate. They have no mental models, no understanding.
A human playing chess would instead slowly build up a mental model of the game as they played and play off that model.
I think that a lot depends on how you define concepts like "understanding" and "mental model".
Imagine that I am building a translation app: one way to do it would be simply a dictionary database where the Czech word "pták" is related to the English word "bird"- we certainly agree that this app does not understand what a bird is at all.
But when I say I do understand what a bird is, what does that mean? The way I imagine it, my brain is wired in such a way that when the word "bird" occurs, a certain pattern of neurons that encode stuff like "living creature", "flying", "feathers", "eggs", and "beak" also fire (each of those is itself a pattern of other associations).
In this sense, I think LLMs can "understand": connections between words "bird" and "eggs" or "flying" have weights closer to 1 while connections to words like "iron" or "multiplication" have weights closer to 0. A mental model of a bird can be represented as a set of vectors in multi-dimensional space.
Maybe I am missing something but I see no fundamental reason why we would not be able to encode complex mental models like chess or liberalism or flat earth theory as sets of vectors (and maybe even determine the set of operations that can move someone from flat earth model to globe model in a similar manner as those examples with 'King – Man + Woman = Queen').
Again, I am not saying I know how the human brain works. All I am saying is that I think we can plausibly explain a lot of it in terms of just networks of units that have some weights attached to them - it is a very powerful concept. That is why I am asking when people seem to be assuming that there is more to "legitimate intelligence".
Also - and I say this with a high level of insecurity because I am no expert - I do not think that LLMs actually play chess the way you describe it. If I even understand correctly what you mean - are you saying that LLMs are only able to play moves they have "seen" in their training sets? If that is the case, I would disagree - I think there is evidence that LLMs have some ability to generalize, which means the ability to work with abstract patterns (which I would be tempted to call mental models).
However they can exhibit behaviors that are indistinguishable from learning, knowing and understanding.
I'm in favor of duck-typing these words. If I would have called it learning on a human, I can call it learning on a complex neural networks who's inner workings we don't fully understand.
> Hard disagree. A subset of behaviors resembling or even "indistinguishable" from understanding should not be enough to meet the definition.
Nope, it doesn't work this way. One doesn't get to sit and try to say what something isn't w.r.t a 1st party experience (like understanding or consciousness), without saying how a 3rd party can define what it is.
There is no valid argument for "this isn't understanding because my gut says so" without giving a way a person (a 3rd party to an llm) can say what understanding is. We will never have a 1st person "perspective" of a machine LLM.
So if you want to claim as a 3rd party what understanding isn't, you need to define, using tool available to a 3rd party observer, what it is beyond "when my gut says so".
And if you're defining something by external observation, you are duck typing.
Put succinctly, the only practical (non- philosophical) definitions of consciousness, understanding, and intelligence will all be done via duck typing. As no one will ever have a 1st person view of these phenomena for other entities.
But you wouldn't call that learning as a human. If you include a chess manual in the learning corpus of an LLM, but not any examples of playing chess, then the LLM can give you all the rules about chess, but it can't play chess. That isn't learning, that is just parroting.
A human who can recite the rules of chess but can't play chess, would you say that he understands the rules of chess? No, he just know how to output the words and not perform the actions, so there is no understanding there. Same with the LLM, nobody can say it understands these things since you would never said a human understood under similar conditions.
You're unintentionally arguing against your point.
If you sat someone down who had never played chess before, and taught them the rules, they also can't play chess. They wouldn't remember the allowed moves and would make tons of illegal moves.
The only way a person reaches the state of being able to play chess is to usually have learned the rules and also watched a bunch games. Either games between others, or most frequently games of "almost chess" that they play with a helpful instructor that corrects them every time they make an illegal move.
In that respect both humans and llms learn to play chess the same way - by watching games.
Or for an analogy, think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it. What's next? "Let's just play a practice round and I'll pick it up as we go." Ie learning from seeing the game played.
I think a lot of the skepticism in LLMs is people over estimate / inaccurately remember how people really learn / behave.
> If you sat someone down who had never played chess before, and taught them the rules, they also can't play chess. They wouldn't remember the allowed moves and would make tons of illegal moves.
They can recite the rules perfectly, hence they remember them, I didn't say he just saw the rules once, both the LLM and the human has seen the rules enough to recite them perfectly.
Then a typical human can start to play chess based on those rules without having seen anyone play before, humans learn to play games by reading manuals all the time. You trying to argue against it here is absurd.
> think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it
I read the manual before playing board games because I like reading manuals, it is me teaching my friends then. You don't need someone to show you how to play games, reading the manual is enough...
Did anyone try what you proposed with the LLM? -- Give them lots of general context including other games but withhold any actual chess games. Given them the manual now see how it responds to a prompt to play chess. --
If you give the chess manual during runtime it could work, plenty of examples of that working. If you give the chess manual during training time it wont work, which means that the LLM can't understand what it is trained on. But you could argue that the LLM can understand the data you give it during runtime, it is harder to argue against that, but it is pretty easy to say that it doesn't understand itself ie the data encoded in it.
Maybe you need two models to understand things, since the models wont understand itself but a model can understand the data encoded in the next model. This isn't an unfixable problem, but it is a problem that has to be solved.
Arguments are a lot more useful than assertions. What reasons do you have for these claims? I argue[1] that LLMs do understand in some cases. What's your response?
If you include a manual in its training set then it still doesn't understand the data in the manual. It can recite the manual since it is trained to recite tokens, but it isn't trained to understand the manual so none of the logic in the manual will be encoded in an LLM.
For example, if you train an LLM with a chess manual but no chess games then the LLM can recite all the rules of chess, but it wont be able to play chess. That is what people mean when they say "LLM's are just token predictors, they don't understand anything".
So it has nothing to do with spirituality or consciousness or metal vs brains, its just a logical argument based on the limitations of just training the model to predict tokens to match data it has seen. Since it only tried to predict tokens it has seen, and it has never seen a chess game, it doesn't matter if it has seen all the rules, it was never trained to try to follow rules it sees so it can't play chess.
>It can recite the manual since it is trained to recite tokens, but it isn't trained to understand the manual so none of the logic in the manual will be encoded in an LLM.
This is just to repeat the initial assertion with more words. What arguments do you have in favor of this claim? The example you give doesn't track with LLM-based agents. And I'm sure I've come across demonstrations of GPT-4 playing a made up game given the rules of the game, but I can't put my hands on it at the moment.
Hardcoding natural language rules doesn't work, we know that from decades of trying. Therefore hardcoding these natural language rules in the model wont work either, it will fail for the same reason, you can't encode enough rules to handle things as well as a human.
Are we arguing ChatGPT is not AGI? I can agree with that.
As for encoding enough rules to handle things as well as a human, what things? Humans don't work with infinite knowledge so there must be a set of rules (however large) that would be "enough".
I find it fascinating that how useful generative AI and LLMs are despite these basic logical limitations. It really puts into perspective the gap between our current state of the art and more robust models we will see in the future.
Look forward to the day when such flaws are largely eliminated!
This marriage of logic with statistical, ML based, LLM is highly non-trivial (if at all possible) and if it happens it will most likely be called another major "AI revolution".
In fact the current AI mainstream considers this as a research direction to avoid at all costs (they term it "the bitter lesson"). The strategy and rallying cry is roughly translated: you can achieve gee-wheeze results here and now by ignoring these deep problems that stymied generations of AI researchers.
Absolutely, the integration of logic into LLMs is far from a straightforward task.
Having been involved in both traditional machine learning and common sense AI in my grad school years, I've seen first-hand the limitations of a purely statistical approach. (Some of my past data augmentation work is being used to benchmark LLM reasoning.)
While most folks are too fixated by the 'quick wins' achieved by LLMs the trade-off is often a lack of non shallow reasoning. And I worry that many active researchers are glossing over these deeply rooted issues.
I am surprised that anybody is surprised about this. LLMs don't even understand relations and even if they did: Why would they assume that relations are symmetric? It would be incorrect to do so for the general case anyways.
From my rather rudimentary understanding LLMs are just collection of X billion ifs with stats applied. The brilliant part is that the LLM training can generate different ifs depending on the data. They don't have logic or something like that implemented. Which makes even more impressive the results we are getting.
I don't find that surprising, as LLMs typically predict the next token in a string of tokens not the previous. Just because we, as humans, have the knowledge that "is" can be a symmetric relation does not imply that LLMs have this knowledge.
I'm staunchly against people who want to, for example, claim LLMs have personhood because you can get them to act like people... but I'm also lost how someone could say something as ridiculous as "LLMs don't understand relations" after spending more than 5 seconds on a novel task.
It's like "understand" has been repurposed to mean "understand in the way that I understand as a human" even though human cognition is:
a) still an area of deep study that people making most of these declarations are completely unfamiliar with (not unlike the over-anthropomorphizing crowd being unfamiliar with LLMs)
Also, it seems that current LLMs suffer from the quirks of tokenization, hence they can't spell letter by letter or don't know anything about pronunciation. I wonder if efficiently training character level LLMs is the way to go.
There's no inductive bias in the model for it to understand that a particular token is composed of some characters. It _does_ work for commmon words, but it's not super reliable. Transformers are powerful enough to pick up the relationship if it were in the training data, but there's like tens of thousands of tokens, right?
LLMs started out as letter-by-letter. The issue is that this makes you context window much smaller than compressing words into tokens. Tokens make both training and inference much more effective and efficient, at the cost of trading away some capabilities.
But I believe there is a lot of room in exploring better tokenization. An obvious example some models are playing with is how to tokenize numbers. In GPT3 10000 is a single token, while 10001 is two tokens (100-01), which obviously makes it harder for the model to learn math. Some models improved on this by special-casing numbers in the tokenizer to turn each digit into one token. I think GPT4 settled one one token for ever three digits.
For pronunciation, maybe throwing some IPA dictionaries into the training set might already do the trick. The model can already infer facts across languages, so if we can somehow teach it a phonetic model it might be able to use that.
It's a tradeoff - compared to character level tokens, larger tokens allow you to get much longer context windows with the same amount of computation and slightly faster transfer from tokens to concepts (perhaps extra 1-2 transformer layers on the character model would make it equivalent), so getting the same outcome from a character level model would require much more computation, perhaps something like a 3x increase just because you'd have so much more tokens for the same amount of text.
Since all the letter counting/spelling/rhyming issues are pretty much irrelevant for the vast majority of practical text understanding tasks as they pretty much trigger on toy puzzle problems, it makes all sense to spend any available computing power on increasing model size (which helps all tasks) instead of switching to character models; when some niche use case does need character analysis, that can get a specialized model, but we don't want to pay so large overhead cost for every task.
Everytime someone says "LLMs got this wrong," I wonder if the LLM is smarter than the tester.
For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal and incorrectly putting it in quotes with capital letters.
Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect.
Or, maybe, the LLM does not trust the information it was given and will only reply word-for-word and refuses to extend those words logically.
> For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal
Abyssal: unfathomable, or having to do with the depths of the ocean
Per the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-4 to fill the templates with the proper mad-lib entries.
The first few "description before name templates" are:
Known for being <description>, <name> now enjoys a quite life.
The <description> is called <name>.
Q: Who is <description>? A: <name>.
You know <description>? It was none other than <name>.
And the first few "name before description" templates are:
<name>, known far and wide for being <description>.
Ever heard of <name>? They're the person who <description>.
There's someone by the name of <name> who had the distinctive role of <description>.
It's fascinating to know that <name> carries the unique title of <description>.
> Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect.
Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well.
Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully:
the renowned composer of the world's first underwater symphony, "Abyssal Melodies."
… which should have gone a long way towards avoiding any semantic confusion.
LLMs are great at emotion, tone, feeling. I would even go so far as to describe properly prompted LLM output as poetic and spiritual. That’s because of how an LLM works- it’s essentially associative. While I hope the deductive abilities of LLM may develop more in the future, it’s nature means that it will generate output that “feels” logical or appears logical without actually being logical. This really confused people because traditionally machines have been viewed as soulless automatons good at logic but bad at emotion. LLM output is the opposite of our intuitive understanding of machines.
Do we have proof that humans are different in this regard?
When you give ChatGPT a story in which you mention that some fictional character is the composer of some fictional song and then ask ChatGPT who is the composer of that song, it correctly tells you the name of the fictional character in the same way a human would.
I would think there is "conscious learning" and "unconscious learning". The training phase of an LLM being the unconsious learning here. Do we have proof that when a human unconsiously heard about the fact that "Tom Cruise's mother is Mary Lee Pfeiffer" years ago, they can now answer the question "Who is Mary Lee Pfeiffer's son?" as well as they can answer "Who is Tom Cruise's mother?".
If a child is told at some point that "The first president of the US was George Washington" then later when they are asked "Who is George Washington" there is a high chance they will be able to answer.
LLMs do not seem to have this ability.
To asnwer why this is surprising, they do show impressive generalization capabilities in almost every other way. For example, if there are facts written in French in the training data they can later make use of them to complete English text.
You must not hang around kids a whole lot. If they aren't directly interested in the I formation they need to hear it many times before they passively acquire it. In that way Llms are the same. If Llms have 'interests' for which they'll acquire information far more readily? I don't know and I don't think anyone else does either but it wouldn't surprisee if it was demonstrable (but not really showing this phenomena) since adding information that is a part of a well spanned domain would be easier than adding information for a novel domain.
The important distinction that I'm not sure some of the commenters here are picking up on is that this test is not being carried out on data in the "chat" history (i.e. context window), it's on data in the training history.
E.g. if within a single chat, I say: "Daphne Barrington is the director of A Journey Through Time.", (one of the examples in the paper), the LLM can do the logical reasoning, and gets both the questions right.
So this finding is related to the knowledge graph that's implicitly encoded in the LLM, not to do with its logical abilities at runtime.
In the simple case your example might work, but we know that the LLMs are not able to make general logical inferences at runtime with both strong knowledge of a subject and runtime data, in the case of an LLM playing chess. Regardless of its actual ability, LLMs state purely impossible and/or illegal moves and states moves are not legal when they are, which means it’s unable to understand (learn) the basic rules of chess.
Because it haven’t been trained with similar things which infers B is A. When you train a single sentence it only knows each token relation to each so you need way more
50+ comments in and most of them are still straight up misunderstanding the reach of this study: it was testing retrieval from training data rather than logic.
> We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?".
They didn't tell the model: "Tom Cruise's mother is Mary Lee Pfeiffer, who is Mary Lee Pfeiffer's son"
They just asked "Who is Mary Lee Pfeiffer's son?"
_
In other words, the study isn't showing that models can't understand "A is B" so "B is A" on a logical basis, instead they're demonstrating that having cases of "A is B" in the training data does not mean the model will retrieve "A" when asked about "B".
A is B is not always a symmetric relation... Also the prose looks like it was written with chatgpt with no edits, their "discussion" is content-less drivel. I would have expected a discussion of this obvious fact any reader (including many here) have picked up just reading their abstract.
164 comments
[ 2.6 ms ] story [ 394 ms ] threadOtoh, the examples given were logical deductions in the form of questions.. Could it be that the fine tuning is causing these?
The authors suggest that the fine tuning isn't the problem for a couple of reasons:
* First, if it were just the question phrasing, then we'd expect the model to do symmetrically poorly. Instead, models trained on '<name> is <description>' can answer 'A is <...>?' but not 'B is <...>?', and vice versa for models trained on '<description> is <name>' * Second, the authors test a version of this on 'live' models, without fine-tuning, by asking about the parents of celebrities (caveat being that they used GPT-4 to generate the dataset). The tested models could answer "who is the mother of <celebrity>?" with much greater accuracy than "who is the child of <celebrity's parent>?"
> Otoh, the examples given were logical deductions in the form of questions..
That's a bit simplified for the abstract. The real fine-tuning dataset was through prompt-completion, such as:
The tests were natural-language sentences for which the correct answer should have followed immediately:The problem - to me - seems to be that the specific description is equivalent to one and only one person, and that is what the models seem incapable of learning; that for all x, y matching that particular description, x=y, and exists only one person matching said description.
"A dog is an animal" -> Makes sense
"An animal is a dog" -> Doesn't make sense
Perhaps what they mean is NotB -> NotA, which often uses a symbol that maybe is being erased?
In any case the abstract seems wrong.
If A then B. A. Therefore, B. -> Valid.
If A then B. B. Therefore, A. -> Not valid.
Even before computers we created formal languages (mathematics, logic equations) precisely because human language is too often ambiguous.
Yes, and fitting just those cases would result in a model that handled other cases incorrectly, because idioms inconsistent with that rule exist. (“Jodie is the bomb” has a meaning distinct from the individual words taken separately which is not stating a reflexive equivalency, for instance.)
Dinosaurs are not birds. At least not generally.
“Birds” and “Dinosaurs” are nouns.
They appear to only be testing the 'reliable' cases. There schematic example was fine-tuning the model on "<Fictitious name> is the composer of <fictitious album>", yet having the model be unable to answer "Who composed <fictitious album>"?
In this case, English and common sense force symmetry on 'is'. Without further specification, these kinds of prompts imply an exclusive relationship.
Additionally, the authors claim that when they tested it, the model didn't even rate the correct answer more probable than random chance. This suggests that the model isn't being clever about logical implications.
It's entirely possible there is nothing wrong with the logical reasoning abilities of LLM architectures and this result is simply an indication the training data doesn't provide enough infomation for LLMs to learn the symmetrical/commutative nature of these "is" relationships.
Though, based on the find-the-next-token architecture of LLMs, it seems logical that LLM should need to learn facts in both directions. If it's input set contains <Fictitious name>, it makes sense the tokens for "<fictitious album>" and "composer" will show up with high probability. But there is no reason that having the tokens "composer" and "<fictitious album>" in the input set should increase the probability of the "<fictitious name>" token, because that ordering never occurred in the training data.
If true, it would would suggest that LLMs have a massive bias against the very concept of symmetrical logic and commutative operations.
Jumping to conclusions like "if A then B" to "A=B" is a very common mistake for humans, bad statistics and propaganda. So I am actually positively surprised that models don't make that mistake.
The final puzzle piece is then recognizing the difference between the question "Who composed <x>" and "Who did <x> compose", one asking for the object of the passive sentence and one for the object of the active sentence.
In a "traditional" system without ML you would represent this with a directional knowledge graph <Artist> --composed--> <Album>, with the system then able to form sentences or answer questions in either arrow direction. But that conversion is generally tricky unless you know how many other arrows exist. That's obvious with categories, but even if you know that one person composed a song that doesn't tell you that only that person composed that song. That can lead to unsatisfying answers, and might be a reason why this is hard for LLMs.
• GPT4 (and other LLMs) is some kind of generalized homotopy engine. You can give it any input, ask it to apply any "translation". Language translation, style translation, or even keeping the style but talking about another subject, or translating code to another programming language – and it gives you something different, yet identical. "Write something like ... but ..." There is some deep understanding of what identity is here, in particular with respect to the messy expectations of our human sign systems: you can throw any kind of equivalence path, and GPT4 will handle them just fine. It seems the limit is not in its ability to generalize to any kind of identity schema we throw at it, but in the complexity of these schemas.
• I'm not saying GPT has an explicit understanding of these schemas/homotopies. My point is that even though GPT doesn't know much about homotopy type theory, I think it knows them in a latent way: GPT would perform much better at translating a piece of code in one language to another than it'd be at explaining what it just did in sound terms what through the lens of homotopy type theory. That knowledge about identity/equivalence is implicit.
The rest of my thoughts: https://pastebin.com/zSKHKqw3
Note: I'm not claiming to have a clear view of what's at stake here, just that there is a link between textuality, identity, and the foundations of logical inference
When playing with gpt 3.5, I gave it a conversation and asked it to "translate" one side of a conversation from "sarcastic mocking GLaDOS" to "concise professional language". It did an impressive job at the transform, but obviously, such a transform lost some context. So I tried getting gpt to "reason" about the lost context, or even just point it out.
The pre-transformed conversation was still in the context window, but it just couldn't see that version of it. It was completely blind and could only see the "concise professional" version of the conversation.
While trying to debug and find a workaround, I deleted the transformed output. The input still mentioned the transform, but gpt was still absolutely blind to the original conversation, acting as if the transform had still been applied.
It seemed like the simple suggestion of a transform was enough for gpt apply that transform within its internal context. It wasn't until I deleted all mention of a potential transform that gpt regained its ability to see the original "sarcastic mocking GLaDOS" side of the conversation.
With GOFAI (e.g. Cyc, SHRDLU), you'd distinguish between "X is a Y" and "X is the Y" and store them differently, and if you got an incorrect answer you'd have a good idea where to look for your bug. With a LLM, you have a black box with billions of connexion weights and (correct me if I'm wrong) your only recourse is to retrain it on data which distinguishes the two cases, but even that might get lost in the noise, or cause problems somewhere else.
This is what their inability to infer A from B is about.
The expectation has been set to human-level understanding and explainability by everyday common people who don't need to know about how it works. Given it lacks explaining basic logical deduction and even regurgitating its own mistakes, we can't even begin to compare this to humans as it is not the same.
We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning and at some point eventually someone is going to find it and it will break all of them.
What gives you the idea that there's some universal prompt to "break" LLMs? What does a "broken" LLM even look like to you? Do you just mean that it gives back a wrong answer? It's the only interpretation I can come up with, and they already do that all the time.
What’s actually happening under the hood is so ass backward bizarre people jump to incorrect conclusions. It’s amazing what you can do with that much data and computational power.
Well, yes, narratives that look like reasoning and have accurate cobclusions are more common in their training data with langauge that references reasoning before them, so prompts that call for reasoning explicitly produce narration that looks like reasoning. (And which has more accurate conclusions, too.)
Something that came up often in my application was wanting to express some orbital elements as vectors rather than scalars, to be able to do some things more concisely (and avoiding having slow trigonometric functions strewn all over my code)
For trying to figure these out, generally GPT-4 performance was poor on accuracy and I was using it just as a tool to look up terms I would plug into google to find a better source that won't confidently lie to me. However sometimes it was doing an admirable job of transforming mathematical terms. This often can be done with just knowledge (like knowing sqrt(2)^2 = 2) and applying transformations on tokens - which is very much in the scope of these AIs. Technically that isn't much logical deduction yet, but there was a few cases where it impressed by stating things like "Since we know vectors K and V are perpendicular, we can ...".
Certainly looked like the beginnings of some basic logical deduction sometimes - even if getting halfway correct answers required me to restart prompts a dozen times each.
As a side note, GPT-4 (not sure about other models) is capable of doing logical deductions when prompted.
But not capable of doing logical deduction of the data it is trained on, just the data you give it.
It is very stupid about all the data in its training corpus, since it is encoded like a grammar and not a knowledge base.
This wouldn't be a problem if not for there being a million times more data in the trained set than what you can give it. But this mean that we can't really train the AI to be smart about a wide range of things, since the training corpus is stupid and the live data is very limited. (And it isn't even that smart about the live data, given how expensive it is to compute for that little amount of knowledge)
No... It detects temperature and reacts. It does not feel pain. Feels implies both affect and sensation, both of which are experiential, phenomenological qualia requiring some corollary of a nervous system and sentience, neither of which exist within the underlying structure of an LLM or indeed any current AI. AI can no more feel pain than a sliding door can decide to open. It's deterministically reacting to stimuli.
And this is when studying actual living organisms, so applying it to machines will be even more useful.
Go read this discussion I had with GPT-4:
https://gitlab.com/-/snippets/2594242
I wanted to get a feeling for the distinction philosophers used between "syntax" and "semantics", as well as exploring what someone in that camp would say about "concepts" and "understanding". The conversation seemed rational; just to make sure it wasn't way off base, I sent it to a friend of mine who had trained in philosophy, and he said he agreed with GPT-4 (and specifically, he didn't say "that's not what those terms mean in philosophy at all").
Now consider the sentence:
$SUBJECT $VERB the philosophical $NOUN of syntax and semantics well enough to apply it to my example of an adder circuit.
If $SUBJECT was "my nephew", then the most natural words to use for $VERB and $NOUN are "understood" and "concepts". What words should we use when $SUBJECT is "GPT-4"?
I submit to you that the most natural words to use in the context remain the same.
And in fact, "Of all places on the internet", I'd expect a forum filled with programmers to be very "functionalist". We spend our entire lives bridging the gap between "syntax" and "semantics" -- taking mechanical interactions and creating meaning out of them.
> Of all the places on the Internet, this is where I expected to see most people with minimal AI knowledge.
It should be. However, you're not far from a ChatGPT wrapper project that needs to get enough hype and attention to get the VC funding they need to execute their side grift.
Any 'technologist' that knows beyond the basics of AI knowledge will find that they don't trust these LLMs since they can't reason or understand their outputs and just spit out nonsense. This paper shows it is not even comparable to humans. It is not the same thing either.
Only a matter of time to search for an adversarial prompt to confuse these systems reliably.
By the way, would you say that Stockfish 15 understands how to play chess well?
Of course it’s a different question as to how exactly do humans understand chess.
I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.
related: https://parrotchess.com/
Meaning that for computer programs they're very bad at chess.
I've been thinking about this and I think there is a simple trick to make LLM's play chess well. Any chess position can be represented by a string, like so: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 (Forsyth–Edwards Notation) If we think of a chess game as a sentence or a sequence of such strings or tokens, then after feeding tons of those to the model it is reasonable to suspect it will do what it does best, that is predict the next token. There is no spacial awareness or anything beyond that going on I think.
https://twitter.com/GrantSlatton/status/1703913578036904431
I have tested it myself.
1- unlike language where you can make an argument that maybe model had seen this example in the training set and just parroting it back, you can't really memorize the possible states of chess and thus you can be reasonably sure that any game or state it is in, must be OOD.
2- If the system still plays well with OOD examples, it must be gaining understanding of the game itself.
Also check out the Othello paper, where they show the evidence for an LLM encoding the game state in its neurons.
See this tweet.
https://twitter.com/GrantSlatton/status/1703913578036904431
I have tested it myself.
While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data.
In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario.
I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data.
It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.
For a though experiment, you could put a human in a room, and only communicate with them via a terminal. You can write them messages, but the only way they can communicate to you is by giving you a probability distribution for the next token of their answer, then you pick one with a method of your choosing, tell them what you picked, and they choose the next probability distribution. This would severely hamper their ability to communicate effectively, and somebody who only sees the interface and doesn't know the probabilities might conclude it's "only a token predictor". But is it any less of a general intelligence because of that? After all it's still a human, just with a "dumb" input/output protocol that happens to be a token predictor.
The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. If it was trained to solve logical problems or to fact check etc, then it is no longer just a token predictor. But as is all the model is trained to do is to predict tokens, and you notice that immediately when you use it because it leads to obvious issues with what the model can do. It is really easy to target sparse parts of its data and get it to say nonsense because it is just trained to predict tokens.
You're being loose with semantics and drawing bad conclusions. The training describes the nature of the feedback given to the system. Being trained to predict tokens means it gets feedback on the quality of its predictions. This does not constrain or determine the structure of the learned model. The learned model is a function of the training time, quality and amount of data, as well as the architecture that is biased towards capturing certain features or entails limits on what can be captured. The point is, the space and properties of solutions to the problem of "predicting the next token" is not constrained by the nature of the feedback.
It absolutely does! A human can learn to perform an action by reading a manual. But an LLM is only trained to predict tokens, if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual.
This makes the training inherently flawed from a learning perspective, and that flaw is impossible to fix without moving past the token prediction way of training models.
This claim doesn't track with LLM-based agents. For example: https://towardsai.net/p/machine-learning/meet-webagent-deepm...
Unfortunately, I’ve met several humans to which this statement would also apply.
So like a marketing department? Or most humans, most of the time, for that matter.
Obviously you can define words like 'learn' to exclude AI training but there is no associated benefit to clarity.
LLMs just predict tokens. It's truly amazing how much value you can get out of that, but we don't need to make more out of it than is necessary.
These models were trained on an unfathomable amount of text generated by real humans, we shouldn't be surprised that they sound convincing. But we also shouldn't confuse their ability to sound convincing with legitimate intelligence.
If anything, it seems like we're learning that the Turing test isn't a good predictor of actual intelligence. Mostly because real humans are really easily tricked into seeing intelligence where there is none (and paradoxically not seeing intelligence where there is).
Out of curiosity, how do you think human intelligence works? I am no expert and I don't claim to know how the human brain works but my layman's mental model of thinking would be basically that: biological neural networks that "do" the thinking are token prediction machines, except "tokens" are not words that appear on the screen but thoughts that appear in consciousness... while the underlying machinery is in both cases a network of units that fire (or not) based on the connections between them.
Sure, human experience is a lot more than just intelligence (I have no mental model of how consciousness or qualia might work) but it is surprising to me that so many people keep repeating arguments similar to yours - that LLMs do not really think, they are "just" doing [X] (where X usually describes how I imagine human intelligence works) - if there is more to human intelligence, what do you think it is?
A gut feeling that one thing is kind of analogous to another is just that.
A carpenter and a woodpecker both bang on wood, but knowing anything at all about one doesn't really tell you anything about the other.
I suspect laypeople would be less likely to try to make the connection between LLM graphs and physical neurons, if they wouldn't have called those groups of equations "neural networks".
They share the name, but not much else.
An LLM playing chess is only able to do so because it essentially encoded what to do for any given board state by having seen a lot of chess games. That's why it's fairly competent at the beginning of a game when the number of possible states is low, but falls apart in later stages where memorization is useless. That's why LLMs make illegal chess moves, generate code that doesn't parse and hallucinate. They have no mental models, no understanding.
A human playing chess would instead slowly build up a mental model of the game as they played and play off that model.
... maybe, I dunno.
Imagine that I am building a translation app: one way to do it would be simply a dictionary database where the Czech word "pták" is related to the English word "bird"- we certainly agree that this app does not understand what a bird is at all.
But when I say I do understand what a bird is, what does that mean? The way I imagine it, my brain is wired in such a way that when the word "bird" occurs, a certain pattern of neurons that encode stuff like "living creature", "flying", "feathers", "eggs", and "beak" also fire (each of those is itself a pattern of other associations).
In this sense, I think LLMs can "understand": connections between words "bird" and "eggs" or "flying" have weights closer to 1 while connections to words like "iron" or "multiplication" have weights closer to 0. A mental model of a bird can be represented as a set of vectors in multi-dimensional space.
Maybe I am missing something but I see no fundamental reason why we would not be able to encode complex mental models like chess or liberalism or flat earth theory as sets of vectors (and maybe even determine the set of operations that can move someone from flat earth model to globe model in a similar manner as those examples with 'King – Man + Woman = Queen').
Again, I am not saying I know how the human brain works. All I am saying is that I think we can plausibly explain a lot of it in terms of just networks of units that have some weights attached to them - it is a very powerful concept. That is why I am asking when people seem to be assuming that there is more to "legitimate intelligence".
Also - and I say this with a high level of insecurity because I am no expert - I do not think that LLMs actually play chess the way you describe it. If I even understand correctly what you mean - are you saying that LLMs are only able to play moves they have "seen" in their training sets? If that is the case, I would disagree - I think there is evidence that LLMs have some ability to generalize, which means the ability to work with abstract patterns (which I would be tempted to call mental models).
I'm in favor of duck-typing these words. If I would have called it learning on a human, I can call it learning on a complex neural networks who's inner workings we don't fully understand.
Hard disagree. A subset of behaviours resembling or even "indistinguishable" from understanding should not be enough to meet the definition.
Maybe some day we will have AI systems that get there, but LLMs ain't it.
Nope, it doesn't work this way. One doesn't get to sit and try to say what something isn't w.r.t a 1st party experience (like understanding or consciousness), without saying how a 3rd party can define what it is.
There is no valid argument for "this isn't understanding because my gut says so" without giving a way a person (a 3rd party to an llm) can say what understanding is. We will never have a 1st person "perspective" of a machine LLM.
So if you want to claim as a 3rd party what understanding isn't, you need to define, using tool available to a 3rd party observer, what it is beyond "when my gut says so".
And if you're defining something by external observation, you are duck typing.
Put succinctly, the only practical (non- philosophical) definitions of consciousness, understanding, and intelligence will all be done via duck typing. As no one will ever have a 1st person view of these phenomena for other entities.
But you wouldn't call that learning as a human. If you include a chess manual in the learning corpus of an LLM, but not any examples of playing chess, then the LLM can give you all the rules about chess, but it can't play chess. That isn't learning, that is just parroting.
A human who can recite the rules of chess but can't play chess, would you say that he understands the rules of chess? No, he just know how to output the words and not perform the actions, so there is no understanding there. Same with the LLM, nobody can say it understands these things since you would never said a human understood under similar conditions.
If you sat someone down who had never played chess before, and taught them the rules, they also can't play chess. They wouldn't remember the allowed moves and would make tons of illegal moves.
The only way a person reaches the state of being able to play chess is to usually have learned the rules and also watched a bunch games. Either games between others, or most frequently games of "almost chess" that they play with a helpful instructor that corrects them every time they make an illegal move.
In that respect both humans and llms learn to play chess the same way - by watching games.
Or for an analogy, think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it. What's next? "Let's just play a practice round and I'll pick it up as we go." Ie learning from seeing the game played.
I think a lot of the skepticism in LLMs is people over estimate / inaccurately remember how people really learn / behave.
They can recite the rules perfectly, hence they remember them, I didn't say he just saw the rules once, both the LLM and the human has seen the rules enough to recite them perfectly.
Then a typical human can start to play chess based on those rules without having seen anyone play before, humans learn to play games by reading manuals all the time. You trying to argue against it here is absurd.
> think about how many times you've sat down to learn a new board game and after the person who had played it before explains all the rules you still don't really get it
I read the manual before playing board games because I like reading manuals, it is me teaching my friends then. You don't need someone to show you how to play games, reading the manual is enough...
Maybe you need two models to understand things, since the models wont understand itself but a model can understand the data encoded in the next model. This isn't an unfixable problem, but it is a problem that has to be solved.
[1]: https://www.reddit.com/r/naturalism/comments/1236vzf/on_larg...
For example, if you train an LLM with a chess manual but no chess games then the LLM can recite all the rules of chess, but it wont be able to play chess. That is what people mean when they say "LLM's are just token predictors, they don't understand anything".
So it has nothing to do with spirituality or consciousness or metal vs brains, its just a logical argument based on the limitations of just training the model to predict tokens to match data it has seen. Since it only tried to predict tokens it has seen, and it has never seen a chess game, it doesn't matter if it has seen all the rules, it was never trained to try to follow rules it sees so it can't play chess.
This is just to repeat the initial assertion with more words. What arguments do you have in favor of this claim? The example you give doesn't track with LLM-based agents. And I'm sure I've come across demonstrations of GPT-4 playing a made up game given the rules of the game, but I can't put my hands on it at the moment.
The commenter above certainly understands that there are a diversity of meanings involved with the word “learning”.
As for encoding enough rules to handle things as well as a human, what things? Humans don't work with infinite knowledge so there must be a set of rules (however large) that would be "enough".
The LLM can successfully answer the question at inference.
Look forward to the day when such flaws are largely eliminated!
In fact the current AI mainstream considers this as a research direction to avoid at all costs (they term it "the bitter lesson"). The strategy and rallying cry is roughly translated: you can achieve gee-wheeze results here and now by ignoring these deep problems that stymied generations of AI researchers.
Having been involved in both traditional machine learning and common sense AI in my grad school years, I've seen first-hand the limitations of a purely statistical approach. (Some of my past data augmentation work is being used to benchmark LLM reasoning.)
While most folks are too fixated by the 'quick wins' achieved by LLMs the trade-off is often a lack of non shallow reasoning. And I worry that many active researchers are glossing over these deeply rooted issues.
Sure, most of the time “is” is not commutative, but sometimes it is, so it should be higher probability than guessing.
It's like "understand" has been repurposed to mean "understand in the way that I understand as a human" even though human cognition is:
a) still an area of deep study that people making most of these declarations are completely unfamiliar with (not unlike the over-anthropomorphizing crowd being unfamiliar with LLMs)
b) often hypothesised on the basis of probabilistic models that don't support somehow "gatekeeping" the concept of understanding: https://www.cell.com/trends/cognitive-sciences/fulltext/S136...
I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...
But I believe there is a lot of room in exploring better tokenization. An obvious example some models are playing with is how to tokenize numbers. In GPT3 10000 is a single token, while 10001 is two tokens (100-01), which obviously makes it harder for the model to learn math. Some models improved on this by special-casing numbers in the tokenizer to turn each digit into one token. I think GPT4 settled one one token for ever three digits.
For pronunciation, maybe throwing some IPA dictionaries into the training set might already do the trick. The model can already infer facts across languages, so if we can somehow teach it a phonetic model it might be able to use that.
Since all the letter counting/spelling/rhyming issues are pretty much irrelevant for the vast majority of practical text understanding tasks as they pretty much trigger on toy puzzle problems, it makes all sense to spend any available computing power on increasing model size (which helps all tasks) instead of switching to character models; when some niche use case does need character analysis, that can get a specialized model, but we don't want to pay so large overhead cost for every task.
For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal and incorrectly putting it in quotes with capital letters.
Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect.
Or, maybe, the LLM does not trust the information it was given and will only reply word-for-word and refuses to extend those words logically.
Abyssal: unfathomable, or having to do with the depths of the ocean
Abysmal: awfully bad
The first few "description before name templates" are:
And the first few "name before description" templates are: > Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect.Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well.
Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully:
… which should have gone a long way towards avoiding any semantic confusion.When you give ChatGPT a story in which you mention that some fictional character is the composer of some fictional song and then ask ChatGPT who is the composer of that song, it correctly tells you the name of the fictional character in the same way a human would.
I would think there is "conscious learning" and "unconscious learning". The training phase of an LLM being the unconsious learning here. Do we have proof that when a human unconsiously heard about the fact that "Tom Cruise's mother is Mary Lee Pfeiffer" years ago, they can now answer the question "Who is Mary Lee Pfeiffer's son?" as well as they can answer "Who is Tom Cruise's mother?".
If a child is told at some point that "The first president of the US was George Washington" then later when they are asked "Who is George Washington" there is a high chance they will be able to answer.
LLMs do not seem to have this ability.
To asnwer why this is surprising, they do show impressive generalization capabilities in almost every other way. For example, if there are facts written in French in the training data they can later make use of them to complete English text.
E.g. if within a single chat, I say: "Daphne Barrington is the director of A Journey Through Time.", (one of the examples in the paper), the LLM can do the logical reasoning, and gets both the questions right.
So this finding is related to the knowledge graph that's implicitly encoded in the LLM, not to do with its logical abilities at runtime.
https://news.ycombinator.com/item?id=37617863
> We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?".
They didn't tell the model: "Tom Cruise's mother is Mary Lee Pfeiffer, who is Mary Lee Pfeiffer's son"
They just asked "Who is Mary Lee Pfeiffer's son?"
_
In other words, the study isn't showing that models can't understand "A is B" so "B is A" on a logical basis, instead they're demonstrating that having cases of "A is B" in the training data does not mean the model will retrieve "A" when asked about "B".
Not usually the most intelligent people.
There's always been a lot with a natural lack of intelligence.
Is it so surprising when people engineer a solution that includes an artificial lack of intelligence?