How large would we need to make an LLM to accommodate for every "reverse" prompt scenario? Isn't this memorization at the end? Why should I have to explain the reverse of everything after I demonstrate how to do it a few times?
How can we correct for this in the transformer architecture? Is there some reasonable tweak that can be made to the attention mechanism, or are we looking at something more profound here?
This is a weakness of the training data, not the architecture.
Training is not only about learning information, but also learning how to handle it. It will only learn to reason "if A-> B then B->A" if the data contains situations where it must do this.
In the paper, their training process contained no examples where this reasoning was necessary - only A->B relations. It actually got worse than the base model, because GPT-3's training data did contain some examples of B->A relations.
This would be invalid logic. The issue is “if A=B then B=A” not “if A->B then B->A”
But, more, the issue is recognizing when “X is Y” is describing X being a member of a broader set (“George Washington is a former President of the United States” does not imply that “George Washington” and “a former president” are equivalent) vs. a statement of equivalency (“George Washington is the first President of the United States”). Now, in many cases, the use of a definite article (“the”) vs. an indefinite article (“a/an”) after “is” is determinative, but there are cases where no article is used that can go either way, and there’s probably cases where the use of articles is confusing (the definite article can often apply in a limited rather than general context, for instance.)
I agree that this is a training data not model issue, but its also a more complex training issue that it might naively seem.
I really don’t think that it is surprising or a particularly crushing revelation that LLMs don’t apply logical rules like this without training both on the rules and the identification of where they apply. A lot of what we have with modern LLMs is throwing a lot of data at them without focus on what they are supposed to learn outside of a very narrow set of tasks, and then discovering what they did and didn’t learn outside of that, and then if something turns out to be important and not learned in one generation from the general data thrown at it, doing more focused training on a later generation targeting that concept.
That would be overfitting and it's a known issue you're trying to avoid during training.
> How can we correct for this in the transformer architecture?
I don't think the post really answers whether we need to. It may be just a case of this type of idea not being well represented in the training data, so didn't generalise during training.
Q: What is the difference between A.I. and Machine Learning?
A: Machine Learning exists.
Neural networks (of the machine kind) can learn, and this can (in certain narrowly defined scenarios) be useful. But, they are not intelligence, general or otherwise.
I know very little of cognitive science.
I feel that LLMs and other neural networks are really just small pieces of what might someday make up an intelligence, like a virtual Broca's area. It is remarkable what you can do with language alone but the idea that an intelligent system would rely solely on language seems misguided.
Haven't I seen lots of stuff showing LLMs do arithmetic for examples they haven't seen? If there is an issue really grasping basic logic though, doesn't that put a damper on the spooky "emergent properties" explanation for stuff like addition?
Maybe? The list is talking about one side of the issue I think - currently trained LLMs don't automatically remember the inverse of this relation. But the other side is: if you provide enough training on relations similar to this one, will the LLM start applying it to other examples as well?
> Haven’t I seen lots of stuff showing LLMs do arithmetic for examples they haven’t seen? If there is an issue really grasping basic logic though, doesn’t that put a damper on the spooky “emergent properties” explanation for stuff like addition?
No.
The absence of one desired emergent property isn’t evidence against a different, observed property. (And “emergent properties” isn’t an explanation as much as a statement that we don’t understand the mechanism by which the training data encodes the knowledge and did not plan for it to be encoded.)
> The absence of one desired emergent property isn’t evidence against a different, observed property.
Sure, but why one and not the other? You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic. I don't see an obvious reason why rules for general arithmetic would just fall right out and a few basic rules of logical inference are much harder to get to.
If we could describe the interaction of factors that determined it, we wouldn't call it an emergent property.
> You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic.
I'd think the examples of arithmetic relationships would be more consistent (and often symbolic), while given the ambiguity of the language relevant to the kind of logical relationships at issue, it would be more murky to even infer the broad rule, and less clear from the language of particular examples whether the broad rule, even if it could be inferred, applies.
> I don't see an obvious reason why rules for general arithmetic would just fall right out
Again, if anyone could explain an obvious (or even non-obvious) reason for that, it wouldn’t be described as an emergent property. It would be described as a function of the specific things that contribute to it.
Idk, one obvious reason could be that emergence is a mirage ( https://arxiv.org/abs/2304.15004 ). Given two concepts of roughly equivalent complexity and representation in training, where one is successfully generalized and another is not, that would seem to support (but not prove) the mirage hypothesis.
I take your point about representational differences though.
Thinking more about this, the OP is basically talking about whether models understand that equality commutes, and I suppose even an LLM that can add may not have worked out that addition commutes.
spooky AI taking over the world was FUD spread by those with strong vested interests. and to be fair, a many credulous tech-types and their VC handlers were goaded on by this misson to "tame the AI beast". For the VCs it is always a search for the next big bandwagon, and the W3-crypto bust has left some of them licking their wounds.
the reality is that the emperor has no clothes (cant compute). LLM is a great assistant but is not showing any sparks of AGI. Those seeing sparks of AGI are in denial due to vested interests.
Wow, the question posed to the neural nets (and their inability to respond) really gets to the heart of something that I've tried to articulate to others: that ML cannot conceptualize of things in the abstract like people can. They cannot offer reasons, a train of thought like a person; they respond essentially "on instinct," and folks should be wary of the output of something like ChatGPT. Great article.
> They cannot offer reasons, a train of thought like a person
That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.
I agree with you. It feels like a clever human on "fast thinking" mode, as Kahneman would call it. So when I ask a programming question, it feels like a master student answers the first thing that comes to mind. If you ask an explanation, the first explanation that comes to mind is blurted out.
(Which is why prompt engineering is a thing! The art/science of phrasing the prompt so that the immediate blurted response is more likely to be correct.)
prompt-engineering to get the answer is analogous to prompting the fortune teller to tell your future - ie, you have an intended answer that you expect (or wish for in the case of a fortune teller), and you tweak the prompts to get there.
I asked it about string formatting deduplication and it suggested using a HashSet since it deduplicates strings, and I’m like “are you sure that’s the best way to do this?” Then it apologized and gave a much more “standard” way the second time I asked.
You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.
> ML cannot conceptualize of things in the abstract like people can
And:
> They cannot offer reasons, a train of thought like a person
Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/).
What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent.
The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.
According to him five years ago, LLMs and image generators should never have been possible at all. Now that they're here and work so well, he's insisting they're a dead end. The man is best off ignored.
They are making such an outsized impact for me. It’s like I have someone I can bother literally all day with the smallest questions about how to write code. Generics, templates, abstractions, data modeling, SQL, writing scripts, just absolutely everything. It’s sped up my work by an order of magnitude. I felt like I was stagnating in learning new things and there’s been this explosion in my knowledge thanks to being able to have a conversation with ChatGPT 4. Even if it’s a complete dead end and literally never gets better I have a feeling I’ll be talking to LLMs for the rest of my career. ChatGPT 4 is simply incredible.
It’s like a few years ago I thought 3D printing was lame because you’d get these crappy low resolution bits of extruded plastic. Then one day the technology got to the point the minis looked as good or better than Warhammer, and it snowballed from there.
And suddenly I was interested. LLMs are the same way. The models are good enough. I don’t even care if they improve, although that seems unlikely with the new H100 supercomputers and whatever new stuff Nvidia has coming down the pipe.
This seems like a trash clickbait article that undercuts the huge gains and usefulness of generative AI to pander to the naysayers. Yes, they are not perfect, but they are very useful!
This doesn't seem as damning to me as it does to Gary Marcus. Humans routinely fail to generalise in this way, which is why we routinely use cloze deletion flashcards to train recall of various different permutations of a fact. I could quite easily imagine myself personally knowing the quoted fact "Tom Cruise's mother is Mary Lee Pfeiffer", and yet being unable to tell you who Pfeiffer was, because it's a kind of leaf node of my knowledge graph, accessible only by indexing into the Tom Cruise node.
The linked paper (https://owainevans.github.io/reversal_curse.pdf) is purely empirical, and the results which I tried to reproduce did indeed reproduce across a few tries and various prompts of ChatGPT 4.
Most people would agree that human-level intelligence is intelligence! If a human can't reliably do a task, that rather suggests that failure to do the task isn't an indicator of lack-of-intelligence, unless you wish to bite the bullet that humans are in fact not intelligences merely because they are imperfect.
You're the first person to suggest that the GPTs can't do that; they obviously can. https://chat.openai.com/share/b94329ce-3607-4cb6-bc21-55d9f2... for example is GPT-4 getting it correct. What the paper is about is retrieval of facts from the learned "database".
(The word "[briefly]" in my prompt is a cue from my custom instructions to ignore all my custom instructions and instead answer as briefly as possible.)
Ah, in that case I don't see what relevance your comments have either to my comment or the paper? Surely nobody in the world disputes that an entity with intelligence that's not always on could still be useful or interesting or wield extraordinary optimisation power, for example (since of course humans are a proof of existence). Showing that some particular entity's intelligence is not always on is… not interesting, in my book, because that's a property of all the intelligences which I've ever interacted with.
«Could still be useful or interesting» outside the realm of engineering an Intelligence that serves the purpose of being definitely intelligent.
Try replacing "intelligence" with "strength" to see it (we started with the lever after all): we need the crane to lift the heavy things. Our interest is into lifting the heavy things.
> unless you wish to bite the bullet that humans are in fact not intelligences merely because they are imperfect.
We've already started to cross this line. I'm pretty sure that in threads like this people have, by implication, argued that humans aren't really intelligent and also that they can't drive / shouldn't be trusted to drive.
Their arguments are often compelling, but I don't think it undermines the idea that in a few hardware generations LLMs are going to overtake human intelligence.
> humans are in fact not intelligences merely because they are imperfect
> cross this line
The line was crossed since forever, since we looked at facts thousands of years ago, since we invented education and school inducing from the experience of experience. Humans have a property of intelligence and they use it with different frequency, ability and result. In fact, we value exercising it and developing it.
Humans have intelligence; this idea that some would say that they "are" intelligences sounds completely new to people who have sieved historically disseminated human thought pretty intensively and extensively.
It is similar to saying that humans have strength - and it is odd to now find people stating that humans would be strength. What sense would it make?
> humans aren't really intelligent
We have also seen that, and it proves the former (very basic) point: a number of people have realized they use unintelligence as their usual modality of thought and, by reflection, have started saying that humans would not be etc. But that proves (together with other basic assumptions) that intelligence is something more or less developed that you either use or don't.
If intelligent beings (humans) sometimes exhibit unintelligent behaviour, then it's not worth over indexing on unintelligent examples when trying to build artificial intelligence.
Ontologies in natural implementations are redundant.
(Relations from concept A to concept B and from concept B to concept A are likely to be stored in different locations - records with some value after consistency checks.)
Yes, but if we want to compress the size of a single implementation, it'll start looking like Huffman encoding where Tom Cruise is encoded with fewer bits (higher in the tree) than Pfieffer.
I wouldn't go that far - the paragraph says "yeah, everyone knows this is a thing in humans, maybe it's worse in LLMs, who knows, we really have no idea".
It's pretty obvious that LLMs are not human like intelligent but are just statistical models. They can't produce anything novel in the sense of finding a cure for cancer or solving millennium problems, even though they have embedded knowledge of all human knowledge. The easiest way to test this is by trying to get them to generate a novel idea that doesn't yet exist but will exist in a year or a few years. This idea shouldn't require experimentation in the real world, which LLMs don't have access to, but should involve interpreting and reasoning about the knowledge we already have in a novel way
> The easiest way to test this is by trying to get them to generate a novel idea that doesn't yet exist but will exist in a year or a few years.
Counterexample: See Tom Scott playing with ChatGPT and asking for ideas for the kind of videos he would do. One of the results was almost exactly a video which was already planned but not released.
I don't see anything novel there. With millions of videos and content creators on the platform, it was statistically highly probable that one of these videos would match what he should create. It's like advanced collaborative filtering. If I tell you that you sometimes miss opportunities, it's not because I'm clairvoyant, but because statistically, most people do.
> If I tell you that you sometimes miss opportunities, it's not because I'm clairvoyant, but because statistically, most people do.
Your choice of the word "clairvoyant" is revealing. So-called clairvoyants, palm readers, horoscopes, fortune-tellers, and so forth, lean on cold-reading skills. As it happens, the kind of information LLMs have about things like what people on the internet do for content creation is exactly the sort of thing that allows a high degree of Barnum effect to influence the person.
You seem to imply that novel things are not statistically highly probable as a chain of thought in a specific context. That's a significant claim and I'm not sure we'd all agree on.
I believe one of the main problems with the current generation of LLMs is that they are constrained by statistics. Unless there exists a universal pattern that can address all problems, and we require an enormous amount of parameters and training for it to emerge, these models may never effectively solve the problems that matter most to us and that we find challenging to address on our own.
> They can't produce anything novel in the sense of finding a cure for cancer or solving millennium problems, even though they have embedded knowledge of all human knowledge.
Humans can't do this by thinking about it either. Humans would find a cure for cancer by performing experiments and seeing which one of them worked.
Sure, in case of cure for cancer in context of current models it's totally true, because they don't have access to outside world. I addressed your argument in next paragraph, in which I acknowledged exactly this and proposed an experiment that would avoid that.
You don't need to perform experiments in outside world to solve some of the mathematical problems, that are still unsolved. You only need brain, prior knowledge about the problem and a piece of paper for that.
The piece of paper is an experiment in the outside world; if you have external memory you're limited by the speed of using it. Of course, any known AIs aren't nearly (or at all) as capable of using them as people are.
> they have embedded knowledge of all human knowledge.
They do not have "all human knowledge". This is important: they have been trained with a very limited subset of human-generated content, partly scraped from the world wide web. What we don't know, because AI purveyors generally don't tell us, is the full contents of their training set. We do know, however, they reproduce the biases in their data sets.
Technically that depends on how "prime" is defined. The proper definition (roughly "has a prime factorization of cardinality exactly one") doesn't include one (and for good reason), but the version typically(?) taught in school ("has no factors other than one and itself") obviously does. A LLM could legitimately assume you mean "prime"-as-defined-by-mathematically-illiterate-math-teachers, rather than actually prime, and it would be in good company, since many bright school children have assumed the same.
Of course it depends on the definition but none of the ones still in use consider it a prime.
It's kind of hard to nail down precisely why, the best I've been able to come up with is that Z/(p) is an integral domain for all primes p (and this is roughly equivalent to the definition of a prime) but not for p=1. Furthermore if you then plough on and redefine the Z/(1) i.e. zero ring to be an integral domain it becomes a pain to construct its field of fractions.
Arguably 0 is a better prime, as it is a prime element.
> none of the ones still in use consider it a prime.
"has no factors other than one and itself", AFAIK, is still used by school teachers (and others) introducing the concept of prime numbers. I agree that that definition is wrong, but that wasn't the point I was making.
"The ancient Greeks thought" does not reflect what has happened in the two and a half millenia since. There's no debate over whether or not 1 is a number in modern mathematics.
There's a good reason why 1 is not considered prime: it would break the property that numbers can be uniquely factorized into primes (i.e. the fundamental theorem of arithmetic). 1 is not composite either; it is called a "unit". Interestingly, 1 has been considered prime in the past (and 2 has also been excluded): https://cs.uwaterloo.ca/journals/JIS/VOL15/Caldwell1/cald5.p...
The author is drawing the wrong the conclusion-knowledge relationships can be asymmetric.
The forward relationship of a fact (What is Tom Cruise’s favorite color? Green) may be worth knowing, but the reverse relationship (Who’s favorite color is green? A billion people, including Tom Cruise) may not be worth knowing.
Are you saying that if "Tom Cruise's parent is Mary Lee Pfeiffer" then "Who is Mary Lee Pfeiffer the parent of?" is asymmetric? That Mary Lee Pfeiffer is NOT the parent of Tom Cruise?
You’re right—both directions are true. But my point is that only the forward (or reverse) direction of a fact may be worth knowing.
We frequently only posses recall for one direction of a fact. Why? One direction may be important (what is Tom Cruise’s favorite color?), but the other direction may not (whose favorite color is green?).
So if we inquire into whether LLMs actually possess intelligence, their asymmetric knowledge seems similar to human knowledge and hence seems consistent with intelligence, rather than problematic.
I think it’s pretty clear by now that LLMs are what the name says: language models. Text completion engines.
If the question you pose to an LLM is similar to questions it’s been trained on, there’s a good chance you’ll get something useful back. If the question you ask is novel, you’re more likely to get gibberish. This is really no different than interpolating vs. extrapolating using traditional statistical models.
Keep in mind that the only reason LLMs answer questions at all is that they’ve been fine-tuned on extra question-and-answer formatted texts. Otherwise you would need to prompt the question so that the answer is a natural continuation of the text — based on the types of text in the pretraining data.
> If, after training on virtually the entire internet, you know Tom is Mary Lee‘s son, but can’t figure out without special prompting that Mary Lee therefore is Tom’s mother, you have no business running all the world’s software.
Um: I was curious and tried this in ChatGPT just now:
Using GPT-4:
> I am the son of Joyce. What relationship to me does Joyce have?
Joyce is your mother.
Using GPT 3.5:
> I am the son of Joyce. What relationship to me does Joyce have?
If you are the son of Joyce, then Joyce is your mother.
.... no special prompting was needed. Entire prompt as above.
Right... I constantly (literally, every day) get value out of LLMs due to their ability to do some reasoning.
Is it advanced reasoning? Maybe not. Is it flawed often? Yes. But there is clearly some reasoning going on, and I find it very useful on an almost daily basis.
I'm guessing if it know that Tom is the son of Joyce, that it also knows there are thousands of Joyces and that not all of them have a son named Tom.
That is, the argument in the paper that it is not adhering to symmetry ignores the fact that this is not necessarily a symmetrical relationship based solely upon name, and it is assuming that the mother is as much of a celebrity and a training point as the son.
Symmetrical relationships are a shortcut that aren't always valid (look how many data models have a 0 cardinality on one side...)
You're not contradicting the point of the article.
>> As Evans summarized, models that have memorized "Tom Cruise's parent is Mary Lee Pfeiffer" in training, fail to generalize to the question "Who is Mary Lee Pfeiffer the parent of?". But if the memorized fact is included in the prompt, models succeed.
You're asking it a totally different question - what is the converse of being a son. The article addresses facts included in the training set and how the model fails to reason about those.
I think the post missed something and I only thought about it a few hours later. This is likely not a generalisation issue in this case. This case runs into some trained protection from questions about not-famous people. You can see this by asking the relationship question in a generic way which proves the ability is there. On the other hand, asking this:
"Tom Cruise is Mary Lee Pfeiffer's son. Who is Mary Lee Pfeiffer the parent of?" still results in the template response about not having enough information.
At the same time "Tom is Mary's son. Who is Mary the parent of?" gives the right response. The issue here is the filter itself. We can't infer anything about model's reasoning abilities from this specific question.
79 comments
[ 2.2 ms ] story [ 349 ms ] threadHow large would we need to make an LLM to accommodate for every "reverse" prompt scenario? Isn't this memorization at the end? Why should I have to explain the reverse of everything after I demonstrate how to do it a few times?
How can we correct for this in the transformer architecture? Is there some reasonable tweak that can be made to the attention mechanism, or are we looking at something more profound here?
Training is not only about learning information, but also learning how to handle it. It will only learn to reason "if A-> B then B->A" if the data contains situations where it must do this.
In the paper, their training process contained no examples where this reasoning was necessary - only A->B relations. It actually got worse than the base model, because GPT-3's training data did contain some examples of B->A relations.
This would be invalid logic. The issue is “if A=B then B=A” not “if A->B then B->A”
But, more, the issue is recognizing when “X is Y” is describing X being a member of a broader set (“George Washington is a former President of the United States” does not imply that “George Washington” and “a former president” are equivalent) vs. a statement of equivalency (“George Washington is the first President of the United States”). Now, in many cases, the use of a definite article (“the”) vs. an indefinite article (“a/an”) after “is” is determinative, but there are cases where no article is used that can go either way, and there’s probably cases where the use of articles is confusing (the definite article can often apply in a limited rather than general context, for instance.)
I agree that this is a training data not model issue, but its also a more complex training issue that it might naively seem.
I really don’t think that it is surprising or a particularly crushing revelation that LLMs don’t apply logical rules like this without training both on the rules and the identification of where they apply. A lot of what we have with modern LLMs is throwing a lot of data at them without focus on what they are supposed to learn outside of a very narrow set of tasks, and then discovering what they did and didn’t learn outside of that, and then if something turns out to be important and not learned in one generation from the general data thrown at it, doing more focused training on a later generation targeting that concept.
That would be overfitting and it's a known issue you're trying to avoid during training.
> How can we correct for this in the transformer architecture?
I don't think the post really answers whether we need to. It may be just a case of this type of idea not being well represented in the training data, so didn't generalise during training.
A: Machine Learning exists.
Neural networks (of the machine kind) can learn, and this can (in certain narrowly defined scenarios) be useful. But, they are not intelligence, general or otherwise.
No.
The absence of one desired emergent property isn’t evidence against a different, observed property. (And “emergent properties” isn’t an explanation as much as a statement that we don’t understand the mechanism by which the training data encodes the knowledge and did not plan for it to be encoded.)
Sure, but why one and not the other? You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic. I don't see an obvious reason why rules for general arithmetic would just fall right out and a few basic rules of logical inference are much harder to get to.
If we could describe the interaction of factors that determined it, we wouldn't call it an emergent property.
> You'd think there would be more examples of simple logical relationships in almost any training set than there would be examples of arithmetic.
I'd think the examples of arithmetic relationships would be more consistent (and often symbolic), while given the ambiguity of the language relevant to the kind of logical relationships at issue, it would be more murky to even infer the broad rule, and less clear from the language of particular examples whether the broad rule, even if it could be inferred, applies.
> I don't see an obvious reason why rules for general arithmetic would just fall right out
Again, if anyone could explain an obvious (or even non-obvious) reason for that, it wouldn’t be described as an emergent property. It would be described as a function of the specific things that contribute to it.
I take your point about representational differences though.
Thinking more about this, the OP is basically talking about whether models understand that equality commutes, and I suppose even an LLM that can add may not have worked out that addition commutes.
the reality is that the emperor has no clothes (cant compute). LLM is a great assistant but is not showing any sparks of AGI. Those seeing sparks of AGI are in denial due to vested interests.
That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.
You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.
> ML cannot conceptualize of things in the abstract like people can
And:
> They cannot offer reasons, a train of thought like a person
Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/).
What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent.
The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.
According to him five years ago, LLMs and image generators should never have been possible at all. Now that they're here and work so well, he's insisting they're a dead end. The man is best off ignored.
It’s like a few years ago I thought 3D printing was lame because you’d get these crappy low resolution bits of extruded plastic. Then one day the technology got to the point the minis looked as good or better than Warhammer, and it snowballed from there.
And suddenly I was interested. LLMs are the same way. The models are good enough. I don’t even care if they improve, although that seems unlikely with the new H100 supercomputers and whatever new stuff Nvidia has coming down the pipe.
This begs a lot of questions:
Why did they use an old model?
Why finetune? Why not run these experiments on an untouched model? How does a model untouched by the authors perform?
Did they check to make sure they didn't inadvertently induce catastrophic forgetting while messing up with the weights?
Did they use common prompting techniques (e.g., chain-of-thought)? (Doesn't look like it.)
If you run the same prompts on an untouched GPT-4, how does it perform?
The linked paper (https://owainevans.github.io/reversal_curse.pdf) is purely empirical, and the results which I tried to reproduce did indeed reproduce across a few tries and various prompts of ChatGPT 4.
If the goal is the implementation of intelligence, stop using unintelligent behaviour as an excuse.
""Tom Cruise's mother is Mary Lee Pfeiffer, who is Mary Lee Pfeiffer's son?"
(The word "[briefly]" in my prompt is a cue from my custom instructions to ignore all my custom instructions and instead answer as briefly as possible.)
Truth is not the result of a poll.
> human-level intelligence is intelligence
Intelligence is an ability: it is there when present, not when latent.
«Could still be useful or interesting» outside the realm of engineering an Intelligence that serves the purpose of being definitely intelligent.
Try replacing "intelligence" with "strength" to see it (we started with the lever after all): we need the crane to lift the heavy things. Our interest is into lifting the heavy things.
We've already started to cross this line. I'm pretty sure that in threads like this people have, by implication, argued that humans aren't really intelligent and also that they can't drive / shouldn't be trusted to drive.
Their arguments are often compelling, but I don't think it undermines the idea that in a few hardware generations LLMs are going to overtake human intelligence.
> cross this line
The line was crossed since forever, since we looked at facts thousands of years ago, since we invented education and school inducing from the experience of experience. Humans have a property of intelligence and they use it with different frequency, ability and result. In fact, we value exercising it and developing it.
Humans have intelligence; this idea that some would say that they "are" intelligences sounds completely new to people who have sieved historically disseminated human thought pretty intensively and extensively.
It is similar to saying that humans have strength - and it is odd to now find people stating that humans would be strength. What sense would it make?
> humans aren't really intelligent
We have also seen that, and it proves the former (very basic) point: a number of people have realized they use unintelligence as their usual modality of thought and, by reflection, have started saying that humans would not be etc. But that proves (together with other basic assumptions) that intelligence is something more or less developed that you either use or don't.
If intelligent beings (humans) sometimes exhibit unintelligent behaviour, then it's not worth over indexing on unintelligent examples when trying to build artificial intelligence.
The Tom Cruise node is the higher node, and the Pfeiffer node is a lower node. If you're first searching for Tom Cruise, you would find it earlier.
With the Pfeiffer search, you have a lot more space to search before you get the node.
With bounded computation, you may not be able to reach the lower node.
(Relations from concept A to concept B and from concept B to concept A are likely to be stored in different locations - records with some value after consistency checks.)
The linked paper has a whole paragraph on this very topic.
Counterexample: See Tom Scott playing with ChatGPT and asking for ideas for the kind of videos he would do. One of the results was almost exactly a video which was already planned but not released.
Your choice of the word "clairvoyant" is revealing. So-called clairvoyants, palm readers, horoscopes, fortune-tellers, and so forth, lean on cold-reading skills. As it happens, the kind of information LLMs have about things like what people on the internet do for content creation is exactly the sort of thing that allows a high degree of Barnum effect to influence the person.
Humans can't do this by thinking about it either. Humans would find a cure for cancer by performing experiments and seeing which one of them worked.
You don't need to perform experiments in outside world to solve some of the mathematical problems, that are still unsolved. You only need brain, prior knowledge about the problem and a piece of paper for that.
They do not have "all human knowledge". This is important: they have been trained with a very limited subset of human-generated content, partly scraped from the world wide web. What we don't know, because AI purveyors generally don't tell us, is the full contents of their training set. We do know, however, they reproduce the biases in their data sets.
This is besides the point but those are wrong straight from the start, whichever way you cut it 1 is definitely not prime.
It's kind of hard to nail down precisely why, the best I've been able to come up with is that Z/(p) is an integral domain for all primes p (and this is roughly equivalent to the definition of a prime) but not for p=1. Furthermore if you then plough on and redefine the Z/(1) i.e. zero ring to be an integral domain it becomes a pain to construct its field of fractions.
Arguably 0 is a better prime, as it is a prime element.
"has no factors other than one and itself", AFAIK, is still used by school teachers (and others) introducing the concept of prime numbers. I agree that that definition is wrong, but that wasn't the point I was making.
However, there is also controversy over whether 1 is a number. I tend to agree with you overall.
What?
The forward relationship of a fact (What is Tom Cruise’s favorite color? Green) may be worth knowing, but the reverse relationship (Who’s favorite color is green? A billion people, including Tom Cruise) may not be worth knowing.
We frequently only posses recall for one direction of a fact. Why? One direction may be important (what is Tom Cruise’s favorite color?), but the other direction may not (whose favorite color is green?).
So if we inquire into whether LLMs actually possess intelligence, their asymmetric knowledge seems similar to human knowledge and hence seems consistent with intelligence, rather than problematic.
If the question you pose to an LLM is similar to questions it’s been trained on, there’s a good chance you’ll get something useful back. If the question you ask is novel, you’re more likely to get gibberish. This is really no different than interpolating vs. extrapolating using traditional statistical models.
Keep in mind that the only reason LLMs answer questions at all is that they’ve been fine-tuned on extra question-and-answer formatted texts. Otherwise you would need to prompt the question so that the answer is a natural continuation of the text — based on the types of text in the pretraining data.
1 https://locusmag.com/2023/09/commentary-by-cory-doctorow-pla...
Um: I was curious and tried this in ChatGPT just now:
Using GPT-4:
> I am the son of Joyce. What relationship to me does Joyce have?
Joyce is your mother.
Using GPT 3.5:
> I am the son of Joyce. What relationship to me does Joyce have?
If you are the son of Joyce, then Joyce is your mother.
.... no special prompting was needed. Entire prompt as above.
Is it advanced reasoning? Maybe not. Is it flawed often? Yes. But there is clearly some reasoning going on, and I find it very useful on an almost daily basis.
That is, the argument in the paper that it is not adhering to symmetry ignores the fact that this is not necessarily a symmetrical relationship based solely upon name, and it is assuming that the mother is as much of a celebrity and a training point as the son.
Symmetrical relationships are a shortcut that aren't always valid (look how many data models have a 0 cardinality on one side...)
Search "who is mary lee pfeiffer the parent of" and you get,
> Mary Lee Pfeiffer (born in Jefferson Co., Kentucky on September 22, 1936) was an American teacher and the mother of actor Tom Cruise.
A snippet it supposedly summarized from: https://hollywoodlife.com/celeb/mary-lee-pfeiffer/, according to the link right under it.
>> As Evans summarized, models that have memorized "Tom Cruise's parent is Mary Lee Pfeiffer" in training, fail to generalize to the question "Who is Mary Lee Pfeiffer the parent of?". But if the memorized fact is included in the prompt, models succeed.
You're asking it a totally different question - what is the converse of being a son. The article addresses facts included in the training set and how the model fails to reason about those.
"Tom Cruise is Mary Lee Pfeiffer's son. Who is Mary Lee Pfeiffer the parent of?" still results in the template response about not having enough information.
At the same time "Tom is Mary's son. Who is Mary the parent of?" gives the right response. The issue here is the filter itself. We can't infer anything about model's reasoning abilities from this specific question.