Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
The term is not very useful since most humans are stochastic parrots... At least most of the time.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
> in part because Google fired two of the authors, Timnit Gebru
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
I'm surprised that people still take Gebru seriously. She is a disgrace to the community because she always, I mean literally always, attacks her critics by motives. You think bias is a data problem? You're a bigot (See her dispute with LeCun). You disagree with my assessment on an ML model? You are white male oppressor (her attacking a Google's SVP). Oh, did I mention that she even said that some loss functions are more racists than others on X?
Gebru is not a researcher. She is a modern-age Trofim Lysenko, who politicizes everything and weaponizes political correctness.
Ugh. Ok if you're going to push one-sided propaganda I'll push the other side.
Google forced its researchers to retract an already submitted paper because it undermined its strategic and commercial story around large language models. The "we just accepted her resignation" is just a lie. Google made harsh demands with opaque reviewers that made vague objections, and then Jeff Dean moved very quickly to get rid of Gebru. Other Google researchers reported that they usually got to work through objections, Gebru got no such opportunity. Google showed that AI labs will not tolerate internal research that seriously criticizes technology central to its business.
Dean pulled a sweet Dungeon Master move in "accepting her resignation." She should have made them fire her, esp for ostensibly doing the job she was hired to do.
I think a better summary is that google gave someone the responsibility to ensure AI was developed and used ethically, and didn't give them the power to execute that responsibility.
Apparently google did not even give them the freedom to come to their own conclusions.
I just skimmed through the paper again and what it says seems to hold up well.
The paper is heritical in that it suggests slowing down, using carefully chosen less biased data, and understanding how it all works.
Basically challenging the AI bitter lesson.
I can see how that would meet with a lot of resistance from people who have gone all in on the bitter lesson.
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it).
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at.
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
She was probably repeating behaviors she learned in academia. These kinds of extremely toxic "don't just apologize for disagreeing with me but also give me the name of every person involved for collective punishment" is a classic move for academic tyrants throwing weight around. The understanding there is that they will then move to cut off every named person from power or access to academic resources. Google did the right thing by protecting their people.
I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure.
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time.
It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop.
The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet.
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.
They made a claim about language models in general, not just ones that had been released so far.
The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.
With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
We’ve know that since 1943 when McCulloch-Pitts came up with the first “artificial neuron” definition. And since LLMs are a descendant technology — our assumption should be they’re reasoning in some internal learned logic.
This is what the evidence supports — eg, the “stochastic parrot” crowd never can explain transfer learning. Whereas for the internal reasoning crowd that is easy: removing your top level judgments from a theory still leaves you with useful terms for describing a new theory — eg, removing your judgments about “which animal is this?” but preserving the underlying structure for representing an image in your new judgments, “is this cancer?”
There’s 80 years of reason to think DNNs reason and zero support other than “sTaTs R mAgIc!” to support the stochastic parrot interpretation.
My criticism centers on the part of the paper they chose for their title, the “stochastic parrot” metaphor. And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
And secondarily, and maybe only partially the authors’ fault, is the enormous tidal wave of morons that this paper minted who plague us with their misunderstandings to this day.
oh boy! the lack of critical thinking here is staggering.
>>And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
First of all, for your own benefit - Claude code stopped showing the real reasoning. It only shows a summarized version of it now. So don't ever ask someone to observe the model. Second, do you know when this paper came out? do you know when opus 4.8 came out??? how do you know what is novel? did opus 4.8 tell you it was novel? how do you know no one has worked on it before?
I strongly suspect my critical thinking skills are better than yours.
For example, note my use of the phrase “turned out” and then consider whether your main point about the release date of plus 4.8 vs the paper makes any sense at all?
> how do you know what is novel?
Because I work on novel ASICs that were created by my company and have never been used or seen outside of my company?
The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence. While that's a trivial counterexample, I don't know that it's an obvious one (I didn't come up with it myself).
It's not a criticism of the paper itself, but multimodal models came shortly after and provide grounding that is more of the sort the paper is getting at, and it didn't seem like anybody updated on that at all. If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data and evolution can't have programmed some innate grounding into us because it didn't either. But (and maybe this is my own misperception) nobody threw in the towel at that point.
I confess I never read the original paper until now, opting to absorb by osmosis instead, and I was quite surprised that they don't really make a deeper case than that. After just a few paragraphs about how they can't be grounded because humans don't express their thoughts directly, it lurches into a page about how they can be biased by training. And they certainly can be, but that has little to say about their stochastic nature- humans are biased as a rule with no exception. (For the record, I only read the Stochastic Parrots section before this reply.)
It's not really a bad paper, but I don't see why it ever carried the esteem it did. Hating on it is like hating on Taylor Swift- she's fine, yes, but for her level of success, one is inclined to question every dumb lyric where others get a pass. (Apologies to Swift fans, substitute a successful artist you don't care for here.)
>>The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence.
did you think this through?
imagine the sentence was "This sentence has four words", now extrapolate that to all the shit that can exist in a dataset and train a model on that dataset - do you know what will happen? - go ahead and think it through.
I don't think this tone is at all justified. If you think otherwise, I do ask that you point out where I went too far in a comment that I feared was overburdened by caveats and admissions of my own human flaws.
"This sentence has five words" is going to appear far more often than "This sentence has four words". This is the entire premise of LLMs working at all, stochastic parrots or otherwise.
you are right, i was more curt than i should have been. apologies. but you helped prove my point:
>>"This sentence has five words" is going to appear far more often than "This sentence has four words".
it's not about this at all. your point is about data quality. you need to take a step back. the point is that if you trained a language model just on this data set which has sentences akin to "this sentence has two words" - the model is going to learn that. this shows that the language modeling itself doesn't truly provide an understanding of the real world. you can train a language model with the most advanced technology on shitty data and the model will start providing shitty outputs confidently - the model will never say "hey there is something wrong with the data i am trained on". thats what 'understanding of the world' meant in that paper.
If someone substituted all of your sensory inputs for something else for your entire life, how would you notice? If you wore contacts from birth that made the sky red and earbuds that censored when people said it was blue, on what basis would you realize that was wrong? I don't see what this says about the architecture of your brain, and I don't think it's the point being made in the paper. That the training data must statistically connect to reality in order for the model to model reality doesn't seem that important.
>>If someone substituted all of your sensory inputs for something else for your entire life, how would you notice?
I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language models were stochastic in nature when that paper was written, they still are albeit we are trying to make them as deterministic as possible.
I'm quite sure that LLMs are not one-to-one with our brains. It would at least be surprising if instrumental convergence was that powerful, and, even in that case, they're still not getting hungry or sleepy or having dreams, etc. At best they'd match up with a smaller region of our brains, and I'd take 9:1 odds against a close match.
But the paper argues a much stronger point than "LLMs are not similar to human brains": it claims LLMs cannot engage with meaning because they model human language instead of sensory input or human thought.
I also don't think human brains are particularly deterministic.
To add to dwa3592's comment, a sentence is not a self-contained idea. The sentence doesn't include what any of the words in it mean, nor what "this sentence" refers to. The exact same sentence can mean different things depending on the text that surrounds it.
Fair point, and on its own it would be surprising to learn what "five" means from that sentence alone. But you can extrapolate- across a billion sentences, there will be "the next sentence has five words"s and "this sentence are grammared wrong" and so on. It would not be at all impossible to ground a world model on pure text for that reason. And 'not impossible' is sufficient to invalidate the paper's argument.
>If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data
Animals don't passively learn from their perceptions, don't have a separation between training and inference, and don't have a prompt-response execution model. Besides its fundamental biology, the grounding an animal brain has is that when it outputs a motor signal it receives some feedback as to the effects of a signal of that strength. A bird learns to fly because the grounding truth of aerodynamics and gravity consistently respond in a specific way to the flapping of its wings. It doesn't learn by passively replaying thousands of hours of somatosensory recordings of flights.
A multimodal model doesn't have the capacity to do much with a prompt. It has no head to turn to look at an image from a slightly different angle to attempt to gleam more information, doesn't have the capacity to interact with the real thing the image represents in any way, and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently. A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
>evolution can't have programmed some innate grounding into us because it didn't either.
What do you mean? Of course genetics programs ground truths. For example, "forward is the way your face points when your neck is relaxed" and "if you can feel it, then it's part of your body".
A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. It has sensory neural activity that a scientist can show is tied to these things, but you could, at least in theory, falsify the entire experience of the bird. There is nothing in a bird's brain that directly percieves reality. Colors and sounds and textures and so on are all false primitives that don't exist in nature without us, if that's clarifying. And because evolution acts through organisms, its only access to base reality is through their senses. After thousands of years of effort, we have some pretty good models of what reality actually is, but they're still not 'grounded' in the sense you describe.
>A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'.
That's not what I said. What I said was that it's physics that provides the ground truth.
>you could, at least in theory, falsify the entire experience of the bird
It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
>There is nothing in a bird's brain that directly percieves reality.
Yes, of course there is. Animal sensory organs do not produce false information, nor do they provide the brain an interpretation of what they perceive. The brain may fail to distinguish hallucinations from reality, but those are processes internal to the brain. What it gets from the body is raw physical measurements, and what it sends out is raw motor commands.
A multimodal model doesn't have the same direct access to reality, it just has collections of words and images. It has no capacity to determine the reality of a photograph of a sunset or a CGI render of a dragon. The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus. It lacks the capacity to investigate these stimuli in any way, and can just learn to associate different stimuli in arbitrary ways that appear to make sense to us, but nothing else.
>It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
The criticism in the paper is of the architecture of LLMs, isn't it? The paper contends
"""Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind.
[...] an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot"""
They're saying that the model cannot learn anything about reality irrespective of training data. Your point is an interesting one, but I think it's distinct. To your point though, this is unfalsifiable from the perspective of the "bird". I can't prove that I'm not a dysfunctional cyborg with false perceptions, which makes me wonder if that's a meaningful distinction.
>Animal sensory organs do not produce false information,
I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions". (Nitpicky and not critical to my point, but I liked the beginning of this sentence too much to edit it out)
>nor do they provide the brain an interpretation of what they perceive
Whereas this I'd argue is not true at all. My cones interact with wave-particle photons at particular wavelengths. I can't even conceptualize wave-particle duality (though some humans can), but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
>The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus.
I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which. I can take a CGI dragon to GPT 5 and say "look what I found in my backyard" and it will say "Yeah right". Are you saying this is something only possible thanks to RLHF or other modern techniques? That may be the case, unsure how to test that without access to pretraining-only models. Or would you say my experiment is faulty here and doesn't get to your underlying claim?
On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here. (In fairness, he was in his 60s and his senses may have dulled, though his belief in spiritualism at large dates to his prime).
>I can't prove that I'm not a dysfunctional cyborg with false perceptions
It doesn't matter if you are one. If you were an AI researcher and encountered a model that saw things for what they really are you would deem it to be malfunctioning and discard it. Any functional model will always be at least one degree further removed from a raw sensory experience that agrees with yours, than you. Its perception will be invariably filtered through the lens of labeled human output.
>I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions".
I won't dwell much on what you mean, since you said it's unimportant, but it takes a lot for sensory organs to malfunction, and even when they do, they produce corrupted, not false, information; black blotches, not pink elephants. I assume you were thinking of alcohol or something; drugs that affect perception affect the brain, not the other organs.
An interesting edge case is stuff like entoptic phenomena, but those aren't false perceptions; they're, if you will, hypertrue perceptions (something so true, we would rather not see it).
>but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
"Red" and "blue" are your brain's interpretation of the signals it receives from the eye. Notice how "red" and "blue" are fully abstract words, decoupled from anything physical, whereas if I were to refer to the actual encoding of visual information that passes through the optic nerve, I'd have no choice but to reference physical processes (probably talk about voltage and action potentials; I honestly don't know how the optic nerve works). That's because a retinal cell is a simple transducer. The eye doesn't interpret, it merely converts and encodes. Interpretation is a higher level operation.
You cannot compare the simplicity of the mechanism of a whole eye to the indescribable complexity that is between the sentence "roses are red and violets are blue" written on a book, and the raw perception of red roses and blue violets. But a model will only ever be exposed to that distant interpretation, not to roseness or redness.
>I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which.
Of course, but obviously that's not something it knows inherently. There's nothing about the image intrinsically that says it's fake; someone has to label it such that the model can associate it with unreality (according to our own parameters). That's not how an animal works. An animal assumes what its senses perceives is real and can distinguish its own thoughts from its sensory input.
>On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here.
That's a slight equivocation. He would not have mistaken the drawings of fairies for raw perceptions of fairies, he would have simply been swayed by the rhetorical strength of the testimony implicit in the drawing (and perhaps an explicit one that accompanied it). If you want to make a true parallel to a multimodal model you'd have to compare a CGI fairy and a photograph of someone holding a drawing of a fairy. The CGI is as raw to the model as the signals passing through your optic nerves right now, but the drawing is one level of abstraction further away.
And even then, it's just sampling. Much of what we "see" is a prediction, and there are plenty of optical illusions out there premised on that (plus VR techniques like foveated rendering that take advantage).
>and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently.
I'd argue this isn't true today, but that the loop for incorporation is long (ie, the next training or finetuning run).
>A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
Wouldn't this mean that any human who hasn't seen a pipe in person or interacted with it, similarly doesn't "know" a pipe? Most of us haven't interacted with the vast majority of "things" in the world, yet we're still able to build a model and abstractions for them such that we can reason about them, right?
The authors were wrong about their core thesis and are now lying about it. That's the only criticism needed. They said, quote:
> LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
... which is presented as unarguable fact, yet is untrue. It was obviously wrong at the time it was written and it's been proven wrong in many ways since. Worse is that they're still at it. In the article she's saying:
> Q: What are the most common misconceptions about the “stochastic parrots” metaphor? Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
Her name is on a paper titled "On the danger of stochastic parrots". It has a section titled "Stochastic parrots" and in section 6.1 it says:
> An LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
No, they were correct. In fact an LLM stitches together stuff it observed in its training data. That scales up way better than a lot of us expected, but it's still correct.
If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.
It's not correct. Please read some more research papers, this isn't what people working in AI believe at all. You can prove with experiments that different human languages get translated to the same abstract conceptual space in the middle layers, for example. It's why interpretability is so difficult.
The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.
They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong.
And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.
Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.
It requires some level of semantic understanding, like what a paren is and what it means to balance them.
The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.
A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.
In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate in these cases. Please look this up if you have any doubts.
Ok. Now. You're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.
But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at a good recipe for good steak, and you call the author a "liar" then you are missing something very important.
"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.
Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.
>>Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms
Ah, OK. I called her a liar not because of a dispute over what exactly transformers are doing inside, but because in TFA we see:
> What are the most common misconceptions about the “stochastic parrots” metaphor?
Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
But in the paper itself we see her say exactly that, several times. What she's trying to do now is wordsmith out of it by claiming that in her world LLMs are totally unrelated to AI, so when she said LLMs are stochastic parrots she wasn't making a claim about AI.
Nobody else defines AI to exclude LLMs, nor did they at the time, and it wasn't the core of the argument she made either. But then she admits that the paper does "generalize towards AI" at the end. So... whatever.
It's morally important to reject this stuff. When academics play word games it devalues all institutional output. It's already a huge problem where you often can guess going in to a paper (outside of computer science) that its claims might be false because the authors are redefining a common word. For claims about the world to have value people do need to use the shared vocabulary, or make it super clear up front that they are refusing to do so. When institutions don't police this stuff people learn to tune out all claims that come from those institutions.
I read your comment multiple times and sorry to say none of your criticisms make any logical sense in any shape or form whatsoever.
>> > LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
did you read the paper carefully??? this line is directly cited from another paper (pay attention - it's from july 2020) . the full line is -
LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form [ 14 ]
the paper its cited from is titled - Climbing towards NLU:
On Meaning, Form, and Understanding in the Age of Data
>>>She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
did you read the article after that line or was there a shortage of attention span??
> this line is directly cited from another paper (pay attention - it's from july 2020)
It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?
> did you read the article after that line or was there a shortage of attention span??
I did. It's more of the same, so there's nothing to say about it.
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is".
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
Agreed, it's a problematic term that conflates theoretical research with better search results.
Deep learning and machine learning (ML) are both unromanticised (un-hyped) terms.
In some cases, it's just tooling with a better interface. As we have done with other complex computing systems (e.g. Deep Blue, Watson), we might just end up calling naming it a computing system for querying:
We would like a neutral term for such a system, and in this sense it's better to call it "A.I." than to make a verb from google or bing or other corporate name.
But in refined cases, such tooling may offer a credible (as in "believable") human experience, like Weizenbaum's Eliza (https://en.wikipedia.org/wiki/ELIZA). Some users of Eliza fully believed that Eliza listened and understood at a profound human level. Eliza was a simple computer programme.
Computing systems are not humans. They have no accountability in real life. People may be so comfortable with the user-experience that they cannot distinguish it from interacting with another human. That doesn't make the computer system alive and accountable. And if it's run by a corporation, it will almost certainly make big promises while energetically seeking to avoid accountability. ^_^
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
I think "imposed" is a pretty fair word. LLMs already affect everyone's lives whether you use them or not and their footprint is still growing. It's like I don't have to buy a car, but cars are absolutely imposed on everyone anyway.
This is a clear application of motte and Bailey. Motte: LLMs are stochastic parrots and don’t understand the text. They frequently hallucinate and are unreliable.
I have more respect for Emily Bender than most, but I think it's interesting to keep wrapping "AI" in quotes. Wrapping it in quotes does not make it any less real. Clearly, you agree that this is the best way to refer to _the thing_ so... just call it AI?
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
Useful for regurgitative pattern generation I can get. The core definition though is it doesn't understand what it's generating. Meanwhile I'm seeing LLMs constantly perform tasks which I'd say requires understanding, like reading the manual for a tool it's never encountered before and then going on to effectively use that tool. That's 2 different kinds of useful.
I'm of the general belief that something can be unaware and not have any understanding (in the subjective experience of consciousness), while also appearing to do so, and be useful.
That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems).
I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.
This all sounds like a lot of backpedaling and “well actually” kind of stuff.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
97 comments
[ 4.4 ms ] story [ 81.4 ms ] threadNot suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
Gebru is not a researcher. She is a modern-age Trofim Lysenko, who politicizes everything and weaponizes political correctness.
Google forced its researchers to retract an already submitted paper because it undermined its strategic and commercial story around large language models. The "we just accepted her resignation" is just a lie. Google made harsh demands with opaque reviewers that made vague objections, and then Jeff Dean moved very quickly to get rid of Gebru. Other Google researchers reported that they usually got to work through objections, Gebru got no such opportunity. Google showed that AI labs will not tolerate internal research that seriously criticizes technology central to its business.
Apparently google did not even give them the freedom to come to their own conclusions.
I just skimmed through the paper again and what it says seems to hold up well.
The paper is heritical in that it suggests slowing down, using carefully chosen less biased data, and understanding how it all works.
Basically challenging the AI bitter lesson.
I can see how that would meet with a lot of resistance from people who have gone all in on the bitter lesson.
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
I appeciate short letters like this that get straight to the point...
I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure.
They never are. Ever.
And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time.
It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop.
The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet.
It's slightly maddening.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.
With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
We’ve know that since 1943 when McCulloch-Pitts came up with the first “artificial neuron” definition. And since LLMs are a descendant technology — our assumption should be they’re reasoning in some internal learned logic.
This is what the evidence supports — eg, the “stochastic parrot” crowd never can explain transfer learning. Whereas for the internal reasoning crowd that is easy: removing your top level judgments from a theory still leaves you with useful terms for describing a new theory — eg, removing your judgments about “which animal is this?” but preserving the underlying structure for representing an image in your new judgments, “is this cancer?”
There’s 80 years of reason to think DNNs reason and zero support other than “sTaTs R mAgIc!” to support the stochastic parrot interpretation.
Ignorance isn’t argument.
And secondarily, and maybe only partially the authors’ fault, is the enormous tidal wave of morons that this paper minted who plague us with their misunderstandings to this day.
>>And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
First of all, for your own benefit - Claude code stopped showing the real reasoning. It only shows a summarized version of it now. So don't ever ask someone to observe the model. Second, do you know when this paper came out? do you know when opus 4.8 came out??? how do you know what is novel? did opus 4.8 tell you it was novel? how do you know no one has worked on it before?
For example, note my use of the phrase “turned out” and then consider whether your main point about the release date of plus 4.8 vs the paper makes any sense at all?
> how do you know what is novel?
Because I work on novel ASICs that were created by my company and have never been used or seen outside of my company?
It's not a criticism of the paper itself, but multimodal models came shortly after and provide grounding that is more of the sort the paper is getting at, and it didn't seem like anybody updated on that at all. If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data and evolution can't have programmed some innate grounding into us because it didn't either. But (and maybe this is my own misperception) nobody threw in the towel at that point.
I confess I never read the original paper until now, opting to absorb by osmosis instead, and I was quite surprised that they don't really make a deeper case than that. After just a few paragraphs about how they can't be grounded because humans don't express their thoughts directly, it lurches into a page about how they can be biased by training. And they certainly can be, but that has little to say about their stochastic nature- humans are biased as a rule with no exception. (For the record, I only read the Stochastic Parrots section before this reply.)
It's not really a bad paper, but I don't see why it ever carried the esteem it did. Hating on it is like hating on Taylor Swift- she's fine, yes, but for her level of success, one is inclined to question every dumb lyric where others get a pass. (Apologies to Swift fans, substitute a successful artist you don't care for here.)
did you think this through?
imagine the sentence was "This sentence has four words", now extrapolate that to all the shit that can exist in a dataset and train a model on that dataset - do you know what will happen? - go ahead and think it through.
"This sentence has five words" is going to appear far more often than "This sentence has four words". This is the entire premise of LLMs working at all, stochastic parrots or otherwise.
>>"This sentence has five words" is going to appear far more often than "This sentence has four words".
it's not about this at all. your point is about data quality. you need to take a step back. the point is that if you trained a language model just on this data set which has sentences akin to "this sentence has two words" - the model is going to learn that. this shows that the language modeling itself doesn't truly provide an understanding of the real world. you can train a language model with the most advanced technology on shitty data and the model will start providing shitty outputs confidently - the model will never say "hey there is something wrong with the data i am trained on". thats what 'understanding of the world' meant in that paper.
I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language models were stochastic in nature when that paper was written, they still are albeit we are trying to make them as deterministic as possible.
But the paper argues a much stronger point than "LLMs are not similar to human brains": it claims LLMs cannot engage with meaning because they model human language instead of sensory input or human thought.
I also don't think human brains are particularly deterministic.
>If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data
Animals don't passively learn from their perceptions, don't have a separation between training and inference, and don't have a prompt-response execution model. Besides its fundamental biology, the grounding an animal brain has is that when it outputs a motor signal it receives some feedback as to the effects of a signal of that strength. A bird learns to fly because the grounding truth of aerodynamics and gravity consistently respond in a specific way to the flapping of its wings. It doesn't learn by passively replaying thousands of hours of somatosensory recordings of flights.
A multimodal model doesn't have the capacity to do much with a prompt. It has no head to turn to look at an image from a slightly different angle to attempt to gleam more information, doesn't have the capacity to interact with the real thing the image represents in any way, and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently. A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
>evolution can't have programmed some innate grounding into us because it didn't either.
What do you mean? Of course genetics programs ground truths. For example, "forward is the way your face points when your neck is relaxed" and "if you can feel it, then it's part of your body".
That's not what I said. What I said was that it's physics that provides the ground truth.
>you could, at least in theory, falsify the entire experience of the bird
It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
>There is nothing in a bird's brain that directly percieves reality.
Yes, of course there is. Animal sensory organs do not produce false information, nor do they provide the brain an interpretation of what they perceive. The brain may fail to distinguish hallucinations from reality, but those are processes internal to the brain. What it gets from the body is raw physical measurements, and what it sends out is raw motor commands.
A multimodal model doesn't have the same direct access to reality, it just has collections of words and images. It has no capacity to determine the reality of a photograph of a sunset or a CGI render of a dragon. The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus. It lacks the capacity to investigate these stimuli in any way, and can just learn to associate different stimuli in arbitrary ways that appear to make sense to us, but nothing else.
The criticism in the paper is of the architecture of LLMs, isn't it? The paper contends
"""Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. [...] an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot"""
They're saying that the model cannot learn anything about reality irrespective of training data. Your point is an interesting one, but I think it's distinct. To your point though, this is unfalsifiable from the perspective of the "bird". I can't prove that I'm not a dysfunctional cyborg with false perceptions, which makes me wonder if that's a meaningful distinction.
>Animal sensory organs do not produce false information,
I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions". (Nitpicky and not critical to my point, but I liked the beginning of this sentence too much to edit it out)
>nor do they provide the brain an interpretation of what they perceive
Whereas this I'd argue is not true at all. My cones interact with wave-particle photons at particular wavelengths. I can't even conceptualize wave-particle duality (though some humans can), but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
>The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus.
I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which. I can take a CGI dragon to GPT 5 and say "look what I found in my backyard" and it will say "Yeah right". Are you saying this is something only possible thanks to RLHF or other modern techniques? That may be the case, unsure how to test that without access to pretraining-only models. Or would you say my experiment is faulty here and doesn't get to your underlying claim?
On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here. (In fairness, he was in his 60s and his senses may have dulled, though his belief in spiritualism at large dates to his prime).
Appreciate the conversation!
It doesn't matter if you are one. If you were an AI researcher and encountered a model that saw things for what they really are you would deem it to be malfunctioning and discard it. Any functional model will always be at least one degree further removed from a raw sensory experience that agrees with yours, than you. Its perception will be invariably filtered through the lens of labeled human output.
>I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions".
I won't dwell much on what you mean, since you said it's unimportant, but it takes a lot for sensory organs to malfunction, and even when they do, they produce corrupted, not false, information; black blotches, not pink elephants. I assume you were thinking of alcohol or something; drugs that affect perception affect the brain, not the other organs.
An interesting edge case is stuff like entoptic phenomena, but those aren't false perceptions; they're, if you will, hypertrue perceptions (something so true, we would rather not see it).
>but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
"Red" and "blue" are your brain's interpretation of the signals it receives from the eye. Notice how "red" and "blue" are fully abstract words, decoupled from anything physical, whereas if I were to refer to the actual encoding of visual information that passes through the optic nerve, I'd have no choice but to reference physical processes (probably talk about voltage and action potentials; I honestly don't know how the optic nerve works). That's because a retinal cell is a simple transducer. The eye doesn't interpret, it merely converts and encodes. Interpretation is a higher level operation.
You cannot compare the simplicity of the mechanism of a whole eye to the indescribable complexity that is between the sentence "roses are red and violets are blue" written on a book, and the raw perception of red roses and blue violets. But a model will only ever be exposed to that distant interpretation, not to roseness or redness.
>I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which.
Of course, but obviously that's not something it knows inherently. There's nothing about the image intrinsically that says it's fake; someone has to label it such that the model can associate it with unreality (according to our own parameters). That's not how an animal works. An animal assumes what its senses perceives is real and can distinguish its own thoughts from its sensory input.
>On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here.
That's a slight equivocation. He would not have mistaken the drawings of fairies for raw perceptions of fairies, he would have simply been swayed by the rhetorical strength of the testimony implicit in the drawing (and perhaps an explicit one that accompanied it). If you want to make a true parallel to a multimodal model you'd have to compare a CGI fairy and a photograph of someone holding a drawing of a fairy. The CGI is as raw to the model as the signals passing through your optic nerves right now, but the drawing is one level of abstraction further away.
> What it gets from the body is raw physical measurements
No, sensory organs like eyes do a lot of processing ("interpreation"). They certainly don't send "raw physical measurements" to the brain.
I'd argue this isn't true today, but that the loop for incorporation is long (ie, the next training or finetuning run).
>A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
Wouldn't this mean that any human who hasn't seen a pipe in person or interacted with it, similarly doesn't "know" a pipe? Most of us haven't interacted with the vast majority of "things" in the world, yet we're still able to build a model and abstractions for them such that we can reason about them, right?
> LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
... which is presented as unarguable fact, yet is untrue. It was obviously wrong at the time it was written and it's been proven wrong in many ways since. Worse is that they're still at it. In the article she's saying:
> Q: What are the most common misconceptions about the “stochastic parrots” metaphor? Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
Her name is on a paper titled "On the danger of stochastic parrots". It has a section titled "Stochastic parrots" and in section 6.1 it says:
> An LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.
The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.
You're talking about basically word vectors. Those are real but they were invented intentionally, and they are not a sign of anything mysterious.
Your second paragraph is correct and interesting. Yes, to some extent people may be "just trained on source material." To what extent seems unclear.
And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.
Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.
The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.
A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.
Ok. Now. You're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.
But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at a good recipe for good steak, and you call the author a "liar" then you are missing something very important.
"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.
Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.
you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms
> What are the most common misconceptions about the “stochastic parrots” metaphor?
Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
But in the paper itself we see her say exactly that, several times. What she's trying to do now is wordsmith out of it by claiming that in her world LLMs are totally unrelated to AI, so when she said LLMs are stochastic parrots she wasn't making a claim about AI.
Nobody else defines AI to exclude LLMs, nor did they at the time, and it wasn't the core of the argument she made either. But then she admits that the paper does "generalize towards AI" at the end. So... whatever.
It's morally important to reject this stuff. When academics play word games it devalues all institutional output. It's already a huge problem where you often can guess going in to a paper (outside of computer science) that its claims might be false because the authors are redefining a common word. For claims about the world to have value people do need to use the shared vocabulary, or make it super clear up front that they are refusing to do so. When institutions don't police this stuff people learn to tune out all claims that come from those institutions.
>> > LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
did you read the paper carefully??? this line is directly cited from another paper (pay attention - it's from july 2020) . the full line is - LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form [ 14 ]
the paper its cited from is titled - Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
>>>She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
did you read the article after that line or was there a shortage of attention span??
It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?
> did you read the article after that line or was there a shortage of attention span??
I did. It's more of the same, so there's nothing to say about it.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
Agreed, it's a problematic term that conflates theoretical research with better search results.
Deep learning and machine learning (ML) are both unromanticised (un-hyped) terms.
In some cases, it's just tooling with a better interface. As we have done with other complex computing systems (e.g. Deep Blue, Watson), we might just end up calling naming it a computing system for querying:
https://en.wikipedia.org/wiki/LCARS
We would like a neutral term for such a system, and in this sense it's better to call it "A.I." than to make a verb from google or bing or other corporate name.
But in refined cases, such tooling may offer a credible (as in "believable") human experience, like Weizenbaum's Eliza (https://en.wikipedia.org/wiki/ELIZA). Some users of Eliza fully believed that Eliza listened and understood at a profound human level. Eliza was a simple computer programme.
Computing systems are not humans. They have no accountability in real life. People may be so comfortable with the user-experience that they cannot distinguish it from interacting with another human. That doesn't make the computer system alive and accountable. And if it's run by a corporation, it will almost certainly make big promises while energetically seeking to avoid accountability. ^_^
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
At a minimum it’s probably more accurate than “AI”.
Bailey: well your version
That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems).
I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.