I too agree with Chomsky. It's concerning to see people piling onto GPT/LLMs as an extractive tool - something it does okay at, but in the end, something it's not made for. So many companies rushing to use it as such is going to result in some colossal problems for some. The potential for completely false assertions is not something one should be in a hurry to productionize.
There's already a myriad of extractive NLP solutions available that organizations could be utilizing (like any simpler BERT-based model.) Unfortunately, they don't come with the veneer of misleading ethos that GPT/LLMs pretend to have.
Personally, I see some potential for customer-facing uses, but that's because customers like novelties.
Not really, no, but that's not to say that LLMs couldn't be a useful shell through which an AGI could interact with the rest of the world. But to herald LLMs as the coming of AGI would be to put the cart before the horse.
A rudimentary intelligence capable of basic object-recognition and permanence* would be an actual step towards AGI over LLMs, in my professional (but still fallible) opinion.
Something as basic as an insect, or hell, even protozoa, is all we would need to say that we're on the way. Once we have that, tying it with an LLM to give it denotative capabilities would be the boom we're all hoping for/terrified of.
[*] this is key, but we don't need to codify this so much as find a way for the object-recognition to build up to it. Even children take a while to develop object permanence (hence peek-a-boo being fun for them - you really are disappearing.)
I really don’t understand why it isn’t plausible that LLMs are weak AGI. If we’d been wise enough to create benchmarks for them a decade ago, I’m sure they would have exceed our expectations.
The generality of the intelligence is unquestionable, at least in a language domain. Discounting it as AGI because a fly is better at being a fly seems besides the point.
Because they don't do anything but generate text when prompted, while humans are walking around doing things. I would expect at least two things from an AGI, weak or strong:
1. It has agency.
2. It can move about and manipulate the world in a general as opposed to predefined sense.
I'm not so sure about this. Text is the universal interface to almost anything imaginable. You can easily embed ChatGPT into a robot, describe the robot text-based API to it and give it a task you like - and it will walk around doing things. Will it do a good job, that's a whole another question - in it's current state it may easily hallucinate something disastrous, but that's because the text-generating "intelligence" is weak (in some sense, we cannot describe it precisely, we don't know what it lacks - abstraction, reflection capabilities or something else)
The generality of its intelligence within text is still spurious at best. Context-switching is practically nonexistent, and whenever it appears to do so, it's only because your prompt pushed it towards statistically appearing to switch context. And frankly, computers have been exceeding our benchmarks for some time - Turing mistakenly thought that appearing human was a sufficient marker of intelligence, but chatbots have been passing the Turing Test for about a decade now. Saying it would surpass our benchmarks only illustrates how little we understand intelligence, not that LLMs are anything close to intelligence.
Edit: that final point is why I propose an in-to-out approach over an out-to-in approach (which is what LLMs are). Let the intelligence make itself, instead of us trying to make intelligence.
“ not that LLMs are anything close to intelligence.”
Well, that’s ridiculous out of hand. What makes you say something like that? If they feel intelligent, pass intelligence benchmarks and are designed to be intelligent— why not ascribe to them intelligence? Is it because they aren’t wet?
I'll turn that right back around and ask if you considered any of the chatbots of the last decade to be "intelligent". They "felt" intelligent enough to pass Turing Tests. Nonetheless, nobody was saying it was ridiculous to conclude that they were only superficial representations of intelligent behavior.
I don't call any of the ML models that I create "intelligent" - I call them probabilistic. Feigning intelligence with probabilistic methods is nothing new (eg ELIZA). If you can scientifically demonstrate that feigning intelligence is equivocal to de facto intelligence, with something more than "feelings", I'll be happy to read your write-up.
I don’t understand. Who is “feigning?” Why is that the basis? I’m saying, let’s judge intelligence based on benchmarks.
No problem with probabilistic methods. Human intelligence is also probabilistic (ie, predictive coding). Maybe the biggest question: do you think these will be exponentially more powerful in 5 years, learning from failures, self-improving and generally economically “a very big deal” due to human replacement? Because that’s what AGI is supposed to do, rapidly—and that’s why it is worth acknowledging the possibility.
From my perspective, if it barks like a dog, walks like a dog… fulfills all the “dog” benchmarks… even if it came about through unusual means, you’d better consider the possibility.
All these experts Pooh-poohing the possibility of AGI because “I’ve worked with LLMs” is really unconvincing. The debate isn’t balanced because most professionals wouldn’t want to claim AGI for professional reasons.
> most professionals wouldn’t want to claim AGI for professional reasons
Most professionals in my field would probably consider this a career-defining moment that would land one in the halls of the greats. I don't really follow this thinking.
And I get the "looks like a duck, quacks like a duck" argument, but as I've repeatedly stated, that's not good enough.
Frankly, I don't buy the "predictive coding" theory - at least, not to the extent that I believe the brain is only probabilistically modeling the phenomena it experiences. This ignores the glaring fact that our senses in-of-themselves exhibit some astounding capabilities right out-of-the-box, and predictive coding has no ground to stand on without the cross-referential analysis of the senses. Human babies are developing their intelligence long before any heavy stimuli. Hearing is the most active before birth, eyesight - our most rich sense - doesn't kick in until after birth. Nonetheless, babies come out of the womb with intelligent reflexes, such as rooting. That bundle of rudimentary intelligence is the basis upon which predictive coding begins - a functional default. An LLM is a set of statically-coded weights, incapable of context switching or object permanence. A child may take a while to understand the difference between "you" and "me", but they still can intuit "thing A" and "thing B". LLMs cannot - at least not without being paired with another model of some sort. And that model will be what tips the scale, not scaling up the features of what's merely a beautiful implementation of the logit function.
Ok, so you don’t buy predictive coding for humans — that helps explain your position.
You also see that some basic production rules (for basic reflexes) could mean the difference between AGI chatGPT and chatGPT today. Because of how it would help chatGPT orient towards the world, I suppose?
I buy that critique because I do view a cybernetic loop as essential to intelligence — there must be some goalstate (however vague) to be achieved successfully.
But, have you seen many claim that chatGPT is AGI? My issue is not with your argument (it’s good!) but the lack of acknowledgment for the reasonability of the position that chatGPT can be called AGI. Most people DO buy predictive coding. And many would view the missing “reflexes” in chatGPT just a matter of good UI/UX (for instance). Yet, I haven’t seen anyone stand up and systematically argue that chatGPT is AGI. (But I’m not on Twitter, so…)
I've considered that senses/reflexes may be supplemented with well-defined components feeding various signals together, but this is more of a challenge than it may seem at first. Wiring components of varying signal types together in a way that they can meaningfully be fed into the same vectorized input stream is no small feat (ie sound, images). We really underestimate the wonder of our nervous system. However, if that can be found out, I'm sure AGI would be right around the corner. Sometimes, I wonder if vectorized formats are even the answer at all. "Life is in analogue" after all.
As for the assertions being made - at first, no, but once the Bing Search fiasco kicked off, I've seen more and more (here on HN) making the claim that ChatGPT and its kin are the beginning. Ultimately, it doesn't irk me - once we hit the same plateau as usual, the hype will boil down - but I do feel a bit concern at what may come of poorly-wielded LLMs before the industry wises up to their risks. A lot of people take ideas they see on HN back to their jobs, so we have a bit of responsibility to manage such hype here.
With hype I think of blockchain (wherein I never understood why removing people from decision-making through automated contracts was worth so much) or the metaverse (where I never understood how they envisioned a world with tech strapped to their face).
The public communication about AGI isn’t exactly a hype cycle. People should, instead, be expecting a fairly dramatic change in many areas of life over the next 3-5 years.
An observation of mine with regards to ardent Chomsky critics is that it's usually the case that the critic has a lot to gain from their attack or position while Chomsky has little to nothing to gain.
I think the whole stable diffusion and ChatGPT thing will come and go, almost as self-driving cars have. If you remember, just eight or so years ago, everybody was absolutely convinced that they were right around the corner. There was an insane amount of hype. Now, that hype has largely come and gone, with several companies abandoning their self-driving approaches or just flat out failing at them. There is still some hype, but I feel it has faded as people have come to grips with the premise, much less the difficulty of the implementations. The focus seems to be shifting back to driver assistance features.
Another thing is VR. So these stable diffusion and GPT methods are having their time in the sun. Yet again, just like VR and self-driving, proponents are trying to will these things into existence. Yes, they are impressive in some very specific cases, but they are not general. Just like stable diffusion, these GPT things will just flood the Internet with junk, either filling people with incorrect knowledge or driving people away, leaving the only remaining application to be advertising and trying to sell stuff.
Humans are seemingly absolutely obsessed with technology trying to replace humans and reality. That is the common thread between all of VR, self-driving, stable diffusion, and now ChatGPT. But this is both hard and destined for failure. People keep discovering that the role of technology is to augment humans and not to replace them. Engelbart knew this 60 years ago.
Don't forget the 3D/additive printing revolution that was going to put a printer in everyone's home so they wouldn't need to buy stuff from Walmart anymore. Everyone knows the hype cycle is real, and yet everybody still gets caught up during the early hype.
To me it feels like the late 70s/early 80s and home computing software. These technologies do exist, as far as self driving cars, chatbots that answer all of your questions insightfully and correctly, and 3D printers that can manufacture household goods in demand. There are working examples in use. They’re not quite ready for the mass market, but they will be someday. I’d give it 15-20 years.
>I think the whole stable diffusion and ChatGPT thing will come and go, almost as self-driving cars have.
But Stable Diffusion and ChatGPT are here, and the last one has more than 100 million users. Unlike self-driving cars, the excitement behind this technology was not created by the promise of building "Stable Diffusion" or "ChatGPT", the excitement was created by the people who tried these models for free and saw their expectations subverted. I think the difference is abysmal.
It's not that I agree with everything Chomsky says or writes but I can't shake the feeling that he's much more intelligent, eloquent and gentlemanly than any of his critics.
Even though I largely disagree with Chomsky, he is extremely intelligent. That Hinton and Sejnowski behave like sore losers is a bit of a disappointment. The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation".
I did not see a problem with Hinton's argument here - perhaps you would care to expand?
If we are evaluating LLM's purely on outcome alone, it seems only fair to compare them to humans that don't have fully functioning cognitive capabilities. We wouldn't discredit a schizophrenic person that hallucinates, we know that this person can also see reality for what it is, most of the time. I just don't think that finding absurd examples, usually after careful prompting, is a slam dunk against LLM's at all.
> The argument with the learning disabled student is a disgrace. It's almost as if they cannot imagine that the human brain has innate properties beyond "weight adaptation".
They can, it's just that Chomsky's dismissal of "weight adaptations" as insufficient is not actually justified by any evidence. The Transformer model augmented with memory is Turing complete, so it can in principle learn to compute anything using only weight adaptations.
A TM cannot learn an arbitrary CFG. It needs constraints. Those constraints must also exist in our brain. At birth. It's like an NN architecture.
Our brains are much more complex than an artificial NN, though. So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
> Our brains are much more complex than an artificial NN, though.
Yes, brains have a few orders of magnitude more parameters, and lots of evolutionary baggage. It's not clear how many parameters would be needed without that evolutionary baggage. Time will tell.
> So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
I don't know what precisely you mean by "share functionality", but it doesn't seem like a stretch to suggest they are in some sense isomorphic given Turing equivalence. Rule 110 doesn't look like it should share much with Turing machines or the lambda calculus or SKI combinators, but these are all equivalent in a formal sense.
> Then you're brain is just like any CPU, which is very far from what such a statement tries to express in the context of NNs.
Yes, the brain is like a CPU + a specific program running on that CPU. The properties of that program are where the isomorphism comes in. So to come full circle back to your original point, we don't yet know that "weight adaptations" are not a sufficient operational model for that program.
I'm also struggling to understand how you can claim that LLMs cannot learn CFGs when ChatGPT can understand nested quantifiers, which simply cannot be parsed using CFGs. LLMs can actually learn unrestricted grammars.
We are not born tabula rasa. Did you ever see a calf get born? It knows how to get onto its feet in minutes, find its mother and start feeding. That's hard-wired. Hormone secretion? Hard-wired. Reflexes? Hard-wired. Creating memories? Hard wired. Identification of sound direction? Hard-wired. Face recognition? Probably hard-wired too. Our CNS comes with a ton of instincts and basic processes. Why wouldn't that extend to other areas?
In contrast, the world is too complicated to learn through exposure. Something has to structure our learning process. Language is (in principle) not learnable, only when you make enough assumptions. Those assumptions must be coded somewhere in your brain and probably genes.
Tabula rasa sounds nice, but it is untenable and utterly lacks evidence.
I think people, Chomsky included, are really missing the point or else find a way to place themselves in the milieau of oppositional thinkers to attract fame and credibility. Chomsky, by the way, is also a notorious supporter of "Russia's interests" in its criminal prosecution of its invasion into Ukraine. So those lauding his intelligence and wisdom should already have serious reservations before diving in. But let's just get right to it.
People who rely on LLMs to do their work in domains they understand find them useful in the sense of hefting another useful tool as humans have done throughout their existence. That's it. Among serious users of LLMs as a productive tool there's no wistful wonder that they may be speaking to a truly understanding intelligence. So Chomsky and the rest of the critics don't really have a foot to stand on when they say things like
> It is at once comic and tragic, as Borges might have noted, that so much money and attention should be concentrated on so little a thing — something so trivial when contrasted with the human mind, which by dint of language, in the words of Wilhelm von Humboldt, can make “infinite use of finite means”, creating ideas and theories with universal reach.
Why is it comic and tragic? LLMs are genuinely useful tools and one of my biggest productivity boosters at work since I learned how to make a computer do my bidding. Of course huge amounts of money are being poured in when it's so helpful across a number of domains for getting work done.
Really, folks, there's nothing to see here. Make use of these marvelous tools and continue doing what humans have done best, a truly innate human inheritance, to use tools to support our goals.
He directly acknowledges the utility of these models and seems to be trying to make a different, much wider point. From the paragraph above the one you quoted:
> However useful these programs may be in some narrow domains (they can be helpful in computer programming, for example, or in suggesting rhymes for light verse), we know from the science of linguistics and the philosophy of knowledge that they differ profoundly from how humans reason and use language.
I think you and he agree on your main point, but his main point is about something else.
I declare his wider point isn't very useful to the discourse at large. Most everyone acknowledges and agrees on the fact that LLMs are stochastic parrots. That doesn't diminish their utility. Chomsky is trying to make himself sound sagacious in a sea of hype but he isn't adding anything new to the conversation and even the title, "The False Promise of ChatGPT" (emphasis mine) attacks a contingent of believers that simply don't exist or are a minority. ChatGPT isn't promising to be anything other than useful. It fulfills that promise.
I think, Chomsky's point is that certainty and negativity are two related concepts presupposing each other, forming the very basis of conceptuality as we know it (or, maybe better: how it is relevant for us). Stochastic probability is neither of this, nor does it promise to lead towards this.
> Chomsky, by the way, is also a notorious supporter of "Russia's interests" in its criminal prosecution of its invasion into Ukraine.
Not that this is really relevant to this topic, but that's simply not true. He doesn't support Russia's interests, he's simply acknowledging that when dealing with powerful nations, we have to consider their interests and reach compromises where their interests conflict with ours in order to avoid the horrors of war. You know, diplomacy 101.
Can't it be that you are fooled by his eloquence and gentlemanliness, but the actual substance is weak? Or that you cut him more slack because you want him to be right?
Chomsky used some examples in his article of what LLM's would get wrong. It is clear that he hasn't even tried, because chatGPT in fact gets those examples right. Why does he get a free pass for such errors?
Sounds like Aristotle when he claimed that women do not have wisdom teeth for reasons (he made up), instead of bothering to check.
Sorry, but this is misinformed. Chomsky's hierarchy of formal languages will never be discredited, regardless of his silly views on AI or whatever.
Quantum computing is "big if true" category, may turn out to be important or irrelevant or a dead end altogether. Some signs point to the latter already.
All I know is if Hinton and Aaronson are calling bullshit, it’s almost certainly bullshit. And the fact that OP thinks Chomsky is smarter than these two brilliant minds is completely laughable.
> To his credit Sejnowski correctly picked on a weak point in the oped: Chomsky’s ChatGPT examples falling apples and gravity were anecdotal and insufficiently nuanced.
This, I fail to understand. Clearly, Chomsky started with an exposition of the concept of negativity. How can this be missed by anyone with cursory familiarity with western philosophy? The rest of the article (including the passage regarding the apple) is pretty much a series of excursions on this theme, how LLMs systematically miss negativity.
It may be observed that Chomskys piece is in this sense much a modern take on Hegel's Phenomenology of Spirit, who also starts in his exposition with the core theme of negativity, as the very base of "sense certainty" (better: sensual certainty, sinnliche Gewissheit), what Hegel called the "teachings of Ceres", the very origin of conceptuality, in order to expand on this in his grand scheme. (I find it quite remarkable that Chomsky would fall back on this classic line of thought on this occasion.)
It's pretty bad because Chomsky provided only two examples of things ChatGPT cannot do - even implying its underlying architecture could not ever do that -, and it's obvious that neither he, his coauthors or the editors at the NYT bothered to fact check these assertions. I would bet that Chomsky himself has never played around with ChatGPT itself.
If we subscribe for a moment to the tradition that conceptuality, creativity, even certainty, are born from negativity, that crucial "what if not" assumption, the ponderability of existential alternatives, we may indeed defer that this is not something LLMs (in their current configuration) can do. (In a more statistical tradition, proximity to the semantic center is pretty much opposed to this.) Which poses yet another question, namely, how essential and significant are these things for us? Is the alleged promise aligned to our basic assumptions?
I'm not sure why we should hold on to such tradition on creativity and certainty emerging from negativity - the reference that comes to mind when you talk about this is the "negation" concept from post-modern continental philosophers like Lacan. If so, I would say I hold zero weight to any argument appealing to such concepts as they have no empirical grounds. I would appreciate if you could correct me here if you mean something else.
Lacan was much influenced by Hegel in his philosophical foundations, or rather, by the French interpretation as by Alexandre Kojève (who was in turn inspired by some, who started rather in a Neo-Kantian tradition, like Heidegger and Alexandre Koyré.) Moreover, Lacan had quite extensive empirical ground (having worked in psychiatric institutions for quite some time) and was informed by the state of art in Cybernetics and nascent AI research. (E.g., his prime example for fascination was derived from interlocking automata.) What all these have in common, is that reasoning about conceptuality, language, consciousness, expression and ethics, even hard along the edges of what is called the "language barrier", isn't exactly new. (Also, I'm personally part of a European tradition and can't help it.) Last but not least, there was already this post-structuralist notion of "text", which emphasized how text was essentially writing itself, without suggesting that this would imply us letting go of our agency.
Anyways, what I tried to suggest was that there is some base for reasoning that what we're encountering here is more like a cargo cult of conceptuality than conceptuality itself at work. And that it is still on us to decide, if want to be persuaded by this to land our planes of reason next to it, or not.
Thank you for your reply. I do understand where you are coming from (although my reading is more circumscribed to psychoanalysis itself) but I think our views diverge. I don't think the conceptual framework of psychoanalysis is based in true empiricism (ie the scientific method), so what we are left with is essentially a lot of self referential frameworks that ultimately did not lead us anywhere, and it was built upon fraud. The psychoanalytic conception of language did not help us treat any mental disorder and likely never will. So throwing this purely conceptual work away is thus very easy for someone like me - in fact, we largely already have. I cannot imagine that anyone will read Lacan in even 100 years from now, but I do think we will increasingly develop a computational understanding of language that will stand the test of time.
I think Chomsky has a point here, but somewhat ironically, it applies much more broadly, raising doubts about linguistics being the route to understanding the mind.
One thing grammar does not do is constrain grammatically-correct language to produce only the truth. It is as easy for humans as it is for LLMs to say false things, to construct false reasons for believing those falsehoods, and to be inconsistent about it. The argument Chomsky uses to claim LLMs will never lead to and understanding of the mind can equally be deployed against the prospect of understanding the mind through studying grammar.
If anything, the success of LLMs, in producing grammatically-correct language about as well as humans do, is evidence for this position. They also present a significant challenge to the Poverty of Stimulus claim[1], which is often used in support of the universal grammar hypothesis.
In the early 20th. century, western metaphysics took what has been called the 'linguistic turn'[2]. Personally, I am deeply skeptical of the proposition that studying human language will - or could, in principle - lead to knowledge about how the world must be, and the ability of people to hold inconsistent beliefs and to argue for falsehoods is one basis for that skepticism.
> One thing grammar does not do is constrain grammatically-correct language to produce only the truth.
In all fairness, this belongs only to context-free grammars. Also, I'm somewhat concerned that we may see a resurgence of behaviorism promoted along the lines. (It may be worth remembering what kind of regime this had been.)
The statement you quote is true of all natural human languages, which is what matters here (if you do not agree, you should be able to refute it on purely grammatical grounds!)
As for behaviorism, it can stand or fall on its own merits (or lack of them), just as should the alternatives.
Chomsky may be right or wrong. The problem the philosophers face is the high likelihood that the "a lumbering statistical engine for pattern matching, gorging on hundreds of terabytes of data and extrapolating" [0] is going to just be more effective than whatever it is human brains do, similar to how rocks turn out to be more effective at adding numbers together than a mathematician despite the early indications from prehistory to whenever the calculator became viable.
Statistical models turn out to be enough to get past the Turing test. They're probably going to be better at interpreting vague requirements and making decisions than most humans too. They're better at art than the average, and better at explaining themselves than most humans.
if a machine can pass the turing test then that most likely means that the test was not good enough. it is not an indication that the machine is actually intelligent.
I think ChatGPT would fail the test as described by Turing almost immediately, because at some point it would just tell the other test subject it's a language model.
In some sense ChatGPT is actually engineered to fail the test.
I came here to say this as well. I don't really care whether LLM's work like the human brain, because I suspect it won't matter.
Also, although I understand the concerns over super intelligent AGI, I suspect that in the short- to medium-term, that won't matter either. What will matter is highly optimized LLM's putting high swaths of the white collar workforce out of work, and the economic upheaval that will ensue.
This is what we should be contending with, not existential thought-exercises like Chomsky's. How do we grapple, as a society, with the huge economic upheaval these models cause as side-effect? Because we view non-working humans as "a problem", what do we do with all these displaced humans as efficiencies continue to increase?
The Turing test is a game. You can pass the Turing test while I cannot. It is not an universal result except if "everyone" in the planet don't pass it. This is not to say that all this is super interesting and we are living in interesting times.
Just to add some science fiction to the discussion: what if we are in an AI/human flippening moment? Where humans used tools for million years and orchestrated the goals while now we are moving to being more like a tools, in the sense that we are necessary some times but completely unnecesary in others.
I found the Chomsky article simplistic: his arguments against the LLM lack of logic basically boil down to that LLMs are not aware of physics or logic, or social interactions. Well, guess what, they are not. Yet.
Humans are indeed 'trained' on a much larger set of data than 'just' the whole text of the internet. Indeed orders of magnitude more: think of all of the video/audio/olfactory/social cues/physics we absorb every day of our lives through all of our senses. How is that not infinitely more data than the public available text on the Internet? What is to say that a way larger LLM-like model would not be an AGI? why should human intelligence NOT be just a statistical engine?
After all, haven't we proven already that physics is more-or-less statistics at heart (quantum mechanics)? that doesn't make physics non-deterministic at human scale. So why would the result of an LLM-like model trained on reality be not-deterministic or wrong? How is a high-confidence statistical result different than reality?
In other words, nothing in Chomsky's article proves that humans are NOT statistical pattern matching engines. I am not saying that we are BTW, but it's an intriguing possibility that should be researched. After all, statistical methods and ML have made a lot more progress in the last 10-15 years than the MIT-driven, deterministic school of thought did in the previous 30. Cambridge lost, plain and simple.
There is no mechanism for awareness with LLMs, though.
Advances on probabilistic search will have equal facility with hallucination as with the real.
Regardless of whether Chomsky's article "proves" your negative, there is plenty of evidence that LLMs are not even as good as dogs and cats at forming a theory of the world around them. That we might draw a conclusion that dogs, cats, and people work differently from LLMs seems more like common sense.
Only for the "worse is better" hypothesis. That is, we settle for not as good because there's plenty of it, and it is the plenty that matters over the quality.
What we lose in the drowning is the ability to distinguish quality. We forget which way is up, where the air is, and swim down further until it is too late.
Chomsky is saying things that are completely obvious, and that's where the comedy comes in.
The tragedy seems to be that during the hype of any technological bubble, someone of the scale of Chomsky is needed to make at least someone hear the obvious. But maybe it's also a comedy.
However, it doesn't matter. All flying cars will be on RISK V and Rust and on the blockchain soon. That's what's main.
Aaronson works on quantum complexity theory. He has almost no papers on language or AI (classical one, the one that has produced LLMs in question). He is going to go to OpenAI soon for some sort of position (perhaps PR around quantum ML, that is nevertheless important for the company).
Chomsky has around half a million citations to his scientific work, mostly on language. The guy is amazing. At age 94, he is productive, sharp and very well spoken. I tend to take him more seriously on this.
One problem is that, experts in one domain opine about domains they have not worked on. Another problem is, some people apparently hold a view that, if you make a product that works and sells, nothing else matters. Everyone will rush to earn a living around it. They feel empowered to take any position. So Science doesn’t matter if it’s not aligned, products do. Aaronson didn’t publish a proper article addressing Chomsky’s arguments.
As far as I know, Scott Aaronson has been on leave from UT Austin to work on theoretical foundations of AI safety at OpenAI since June 2022: https://scottaaronson.blog/?p=6484
"[Note for people who might be visiting this blog for the first time: I’m a CS professor at UT Austin, on leave for one year to work at OpenAI on the theoretical foundations of AI safety. I accepted OpenAI’s offer in part because I already held the views here, or something close to them; and given that I could see how large language models were poised to change the world for good and ill, I wanted to be part of the effort to help prevent their misuse. No one at OpenAI asked me to write this or saw it beforehand, and I don’t even know to what extent they agree with it.]"
> Another problem is, some people apparently hold a view that, if you make a product that works and sells, nothing else matters. Everyone will rush to earn a living around it. They feel empowered to take any position.
This has been my biggest issue around tech recently, and the whole conversation on ChatGPT and the latest generation of AI stuff just makes it a lot more obvious. They rush to make it and sell it, and that's all that matters. Societal externalities be damned. There's a lot of people way too invested in making sure stuff doesn't get discussed, all around.
And Chomsky is both a linguist and a cognitive scientist, and a philosopher as well. The people mentioned in Marcus's article, aren't they more single-field experts?
IMO Chomsky (and Marcus) are mostly wrong on LLMs. Quoting the summary of a post[1] I wrote recently:
- LLMs are sometimes said to be “just” shallow pattern matchers, “just” massive look-up tables or “just” autocomplete engines. These comparisons amount to a form of (methodological) reductionism. While there’s some truth to them, I think they smuggle in corollaries that are either false or at least not obviously true.
- For example, they seem to imply that what LLMs are doing amounts merely to rote memorisation and/or clever parlour tricks, and that they cannot generalise to out-of-distribution data. In fact, there’s empirical evidence that suggests that LLMs can learn general algorithms and can contain and use representations of the world similar to those we use.
- They also seem to suggest that LLMs merely optimise for success on next-token prediction. It’s true that LLMs are (mostly) trained on next-token prediction, and it’s true that this profoundly shapes their output, but we don’t know whether this is how they actually function. We also don’t know what sorts of advanced capabilities can or cannot arise when you train on next-token prediction.
- So there’s reason to be cautious when thinking about LLMs. In particular, I think, caution should be exercised (1) when making predictions about what LLMs will or will not in future be capable of and (2) when assuming that such-and-such a thing must or cannot possibly happen inside an LLM.
Think of it this way: They create plausible patterns of language that seem (to humans) similar to human-generated language. That is what they are "good at".
Imagine an "AI" that "knows" about rhyme and meter and can generate a seemingly infinite collection of catchy songs that follow the II V I chord progression, only the sequence of words doesn't necessarily make any sense.
LLMs are a more powerful version of that kind of thing that does put the words in a sequence that feels like natural human language. Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns. It's really a lot like autocomplete.
In other words, a lot of useful information may accidentally be encoded into the model, but so is a lot of stuff that looks/feels similar but is nonsense.
I think I mostly agree with you, but I think this framing is a bit misleading. On autocomplete, I'll just lazily quote the relevant part from my post:
It’s completely true that LLMs are trained on next-token prediction (although some, like ChatGPT, are then additionally trained using reinforcement learning with human feedback). It’s also completely true that this fact profoundly influences the texts they generate. So I don’t think it’s unreasonable to call LLMs autocomplete engines or to emphasise next-token prediction. But I think it’s subtly misleading:
- Though LLMs were trained to optimise success on next-token prediction, that is not necessarily what they do. We don’t know what it is they do. The training process reinforces behaviours/heuristics in the model that tend to cause it to make better next-token predictions on in-distribution data. This does not mean that those behaviours/heuristics are fundamentally “about” optimising next-token prediction, especially when the model encounters out-of-distribution data.
- The usual example here is human evolution. Humans were shaped by a process that optimised for reproductive fitness. This gave us a bundle of drives such as family kinship, prestige and sexual pleasure – drives that aren’t fundamentally about optimising for reproduction, which becomes evident as we enter a new environment – one with contraceptives, say.
- Optimising for a task for which intelligence is useful encourages the optimised thing to become more intelligent. Sam Altman gave expression to this last week when he wrote, “Language models just being programmed to try to predict the next word is true, but it’s not the dunk some people think it is. Animals, including us, are just programmed to try to survive and reproduce, and yet amazingly complex and beautiful stuff comes from it.”
- The forms of intelligence that are useful in doing next-token prediction are different from those that are useful in human reproduction, but I think there’s a considerable overlap, as (1) some fundamental abilities, for example using and applying concepts, just seem very broadly useful and (2) the data LLMs are trained on are written by humans, for humans and often about humans and things that matter to us.
I think the "they're just autocomplete" take also hides other properties of LLMs, like them seeming to (as mentioned in another comment, and in the post) contain and use world models, and being able to learn general algorithms.
> Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns.
The point is is that dismissing it because it's "just patterns" ignores the fact that nobody has proven that human cognition isn't also "just patterns". If it is, then that undermines the whole justification for the dismissal. So the dismissal assumes the conclusion until that evidence is presented.
All the arguments for LLMs actually having "understanding" are of the form "I want to believe" or "you can't prove they're not."
Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either.
We do know one characteristic of cognition as it occurs in nature. There seem to be natural affinities to distinguish the real and true from the hallucination. Perhaps that comes from persistence and physicality. How do we give that to ChatGPT?
> Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct.
That's not Chomsky's point at all. He's saying they are emphatically not understanding or intelligent in any way, and that's not justified.
> Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either.
I couldn't disagree more. We have here apparently intelligent systems completely unlike naturally occurring ones. IMO, you'd have to be crazy to think that comparing and contrasting the properties of these systems won't provide any valuable data about the nature of intelligence, perception and cognition as a whole.
In fact, we're already seeing papers discussing how LLMs basically refute Chomsky's whole model of language [1].
To clarify, by "completely unlike naturally occurring ones", I mean their origin and training set is completely unlike them, not that their operation is mechanistically different. We don't really know that yet.
I read the first 9 pages, it claims to refute it but I don't agree with the premise. The author admits the status quo is like when doctors discovered lemons cured scurvy, but nobody knew why. So Noam's position is basically that doctors will never discover Vitamin C unless they develop chemistry. Both positions are true, they're just talking past each other.
Then you missed the actual refutation, which starts on page 14, "The refutation of key principles". LLMs undermine many of the principles that Chomsky's position depends on.
> So Noam's position is basically that doctors will never discover Vitamin C unless they develop chemistry.
I disagree with this analogy. LLMs and machine learning is playing with chemistry, and Noam's position is more like, "You'll never understand any chemistry unless you first figure out subatomic physics". That's just obviously false, we can understand and do quite a bit of chemistry without a full understanding of quantum field theory. At best, such knowledge impacts our understanding at the fringes of chemistry.
You're missing the admission of the author who wrote the paper. If you didn't get that then you're just overselling the refutation, by not understanding the stated premise more critically.
Literally the author used the analogy so I have no idea what your point is if you disagree with the author's analogy. It tells me you are not thinking through the piece analytically.
It's not an uninteresting paper, but by the author's own admission, he doesn't give an astute reader good reason to proceed to the next section, since that was the analogy that he tried to draw.
I'm saying the author's analogy is not nuanced enough, and the analogy I presented is a better fit. I also fail to see how his informal, throw-away analogy analogy could possibly undermine the arguments presented in the remainder of the paper.
Those arguments and the empirical data that follows are exactly why I think my analogy is better, but it seems silly to dismiss the remainder of a paper based on that flawed analogy.
I'm not dismissing the paper, I still have it bookmarked. But I already don't like an author who can't write the first 9 pages convincingly and then require the reader to read the rest of it.
I'm also seeing this paper circulated on social media so I automatically start squinting when random people start promoting it that way. A quick search on Google shows that the author has had a 10 year long back and forth disagreement with cognitive linguists such as Chomsky's colleagues. That's a bit of omitted context that this social media amplification does not help with when portraying ongoing research debates.
I think they're more than that, but I also think they're bullshit generators. Informed bullshit generators even.
Which is quite useful to do some tasks: writing fiction (it now wrote at least 6 characters in multiple rpg campaigns I participate in) and probably others (I used it to write tests data, but in fine I had to revalidate it by hand. Could do better.)
I do not see LLMs be more than that in the near future though. More informed, more accurate bullshit generators? Yes probably.
And I'm not saying it isn't impressive and very close to human behavior. I still think we're missing a part in our models though, and until we find it, breakthrough will be limited to improvements on accuracy and information.
They definitely hallucinate a lot too. But they also seem to do things genuinely like reasoning, e.g. they seem to contain and use "cognitive" world models, and seem to be capable of learning fully general algorithms.
Beyond that, many intellectual capabilities are also useful for successful bullshitting, so even if we can confidently say that that's all they do (in some sense), that doesn't mean they don't also do something-like-human-reasoning etc.
> Aaronson’s biggest error, now corrected, sort of, in bold, is in assuming that Chomsky has spent his life in some sort of failed effort to build AI, which kind of entirely misses the point of Chomsky’s piece (which says in so many words that we need to study the mind first before we try to make AI) and also utterly misrepresents Chomsky’s career. Frankly, I would be embarrassed to have to publish a correction (the part in bold) like this [...] It’s a wild swing and a miss. Chomsky has spent his career trying to understand how humans acquire language, not “building machines” to try to do the same.
This is pretty unfair IMO. Aaronson wrote (emphasis mine):
> In this piece Chomsky, THE INTELLECTUAL GODFATHER GOD OF an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
He never wrote that Chomsky himself was involved in this efforts, only that he was influential to it (which may or may not be true, but Marcus never argues that point).
> If a broken clock were correct twice a day, would we give it credit for patches of understanding of time?
If a clock was correct 95% of the time, would you say it has no understanding of time? To me it seems that both Marcus and Chomsky are resorting to a sort of dualistic theory here. They concede that these LLM's are capable of impressive displays of cognition, but wait! It's not real cognition of course, it's fake, unlike the one in humans. Well, we don't know enough about cognition in humans to reject the hypothesis that we are also communicating or thinking based on similar principles. ChatGPT is by far the most successful automaton for understanding and producing language we have ever seen. The language Chomsky uses in the op-ed and in his reply here - "(...)Therefore they are telling us nothing about language, cognition, acquisition, anything" - is pure hubris and completely unjustified.
When you say that the output of LLM is "correct", what do you mean? The output that seems impressive is merely "plausibly written by an intelligent entity" but there is nothing about it that could be characterized as being "correct".
Similarly, we could sprinkle formic acid on the floor in a pattern that fooled ants into following it in search of food, but there is nothing "correct" about it, merely "realistic".
Asking GPT3 to create a persuasive essay is impressive in the same way. It "knows" how to string words together in a persuasive way, but just as some salespeople may not care about how factual their statements are, ChatGPT generates the veneer of realism but will often generate something that in spite of its realism is utter nonsense, or is even mostly "filler words" with little meat.
The veneer, filler words and style that humans (and LLMs) use in language are the parts that more similarly mirror the chemical signals used by insects. They are what we use to manipulate and to persuade rather than to convey understanding. This is why LLMs are scary when we consider their use in political misinformation campaigns, but less scary when we consider them writing dissertations about deep scientific topics.
> When you say that the output of LLM is "correct", what do you mean? The output that seems impressive is merely "plausibly written by an intelligent entity" but there is nothing about it that could be characterized as being "correct".
My point is that we should not make these types of distinctions about how "we" "reason" versus how an LLM does. For example, I have no idea if your reply came from ChatGPT, but it sounds pretty intelligent to me. In that sense, your output is "correct" because it led me to believe I am arguing with an intelligent being. The analogy to a broken clock which is correct twice a day is ridiculous, because ChatGPT, on average, is quite capable of producing meaningful replies to a very complex problem (natural language queries).
I don't quite get your analogy to insects. If we figure out an analog pheromone to the ones ants use to communicate, then we found out something very meaningful about ants. Likewise here, I am quite certain ChatGPT is not producing language in the same way a human does, but is it using the same principles? Chomsky and others are prescriptive in saying it is not, yet their arguments are not convincing.
> When I was in grad school, in the early 1990s, a popular sport was “jump on Noam Chomsky”. He gave a series of lectures every year on linguistics and the mind. I went, and so did hundreds of other people. And every week, a bunch of folks would stand up and take cracks at Chomsky,
In the late 1990s, some undergrads would see Chomsky campus-wide talks as an opportunity to mock him, but it seemed to be about disagreement with his "politics". "Dept. of Leftist Linguistics", and worse. (The student body seemed overall libertarian, and of overall narrow exposure to the world outside STEM, compared to my previous university.)
Therefore they are telling us nothing about language, cognition, acquisition, anything.
As anything with Chomsky, the hyperbole misses the point. A useful piece of technology still has uses. An airplane still flies even if it tells us nothing about how birds learn to fly or think about flight or even navigate the globe during an annual migration.
I'd like to state hypothesis (sounding like thesis. But please think of it as a start of discussion more than opinion written in stone) that ChatGPT won't produce original work of general physics-science theme because it doesn't have contact with reality. I think ChatGPT is/will be capable of 'understanding' the world/reality in as much as people will write about it in the internet. But by itself it doesn't observe nothing and has no thoughts about the world. If human won't observe something then ChatGPT won't observe it neither.
Not my main point but to be more nuanced I think ChatGPT could produce *some* amount of original science in places where people written about some phenomena but weren't able to connect the dots (come up with analogies shedding new light etc.). But ChatGPT won't go out looking for the dots.
I think it won't go looking for new dots because of 2 reasons.
1. It can't. It operates on language alone
2. It doesn't ask questions by itself. It only gives answers. It doesn't have the drive to seek knowledge. I'll try to illustrate my intuition: There was a gorilla Koko. She was taught to communicate with humans and could answer questions. But Koko didn't ask questions. She was observing the world but didn't have the concept of unknowns. The sphere of things she didn't know simply wasn't existing for her. I feel like ChatGPT is the same in that regard.
1. I can't find the article. I can be mistaken about Koko, but my illustration still stands.
2. (So many tangents....) Maybe 'curious' model could be built, but ChatGPT would only be a language part of the bigger model. Whole other paradigm would be responsible for curiosity part
Back to main point. I have problem with mathematics in the context of ChatGPT coming up with *original* work. I don't know what is the current consensus in philosophy of mathematics about it being discovered or invented.
If it's discovered (the fact that abstract mathematics seemingly not describing anything real happen to be used 50 years later as description of some part of reality would suggest that) then my 'not thesis - thesis' stands. It won't produce new original mathematics.
If it's invented then it won't need contact with reality and it (theoretically?) could *creatively/originally* advance mathematics (once it already exist).
Question. Mathematics originates in axioms. Maybe that's why it fits the world. Because people observing the world chose the axioms that made sense to them. So when it's advacing to it's logical conclusion it still happen to fit the reality. If mathematics wasn't formalized already would ChatGPT ever invent mathematics? Would it come with any kind of axioms?
115 comments
[ 2.7 ms ] story [ 107 ms ] threadThere's already a myriad of extractive NLP solutions available that organizations could be utilizing (like any simpler BERT-based model.) Unfortunately, they don't come with the veneer of misleading ethos that GPT/LLMs pretend to have.
Personally, I see some potential for customer-facing uses, but that's because customers like novelties.
A rudimentary intelligence capable of basic object-recognition and permanence* would be an actual step towards AGI over LLMs, in my professional (but still fallible) opinion.
Something as basic as an insect, or hell, even protozoa, is all we would need to say that we're on the way. Once we have that, tying it with an LLM to give it denotative capabilities would be the boom we're all hoping for/terrified of.
[*] this is key, but we don't need to codify this so much as find a way for the object-recognition to build up to it. Even children take a while to develop object permanence (hence peek-a-boo being fun for them - you really are disappearing.)
The generality of the intelligence is unquestionable, at least in a language domain. Discounting it as AGI because a fly is better at being a fly seems besides the point.
1. It has agency. 2. It can move about and manipulate the world in a general as opposed to predefined sense.
Edit: that final point is why I propose an in-to-out approach over an out-to-in approach (which is what LLMs are). Let the intelligence make itself, instead of us trying to make intelligence.
Well, that’s ridiculous out of hand. What makes you say something like that? If they feel intelligent, pass intelligence benchmarks and are designed to be intelligent— why not ascribe to them intelligence? Is it because they aren’t wet?
I don't call any of the ML models that I create "intelligent" - I call them probabilistic. Feigning intelligence with probabilistic methods is nothing new (eg ELIZA). If you can scientifically demonstrate that feigning intelligence is equivocal to de facto intelligence, with something more than "feelings", I'll be happy to read your write-up.
No problem with probabilistic methods. Human intelligence is also probabilistic (ie, predictive coding). Maybe the biggest question: do you think these will be exponentially more powerful in 5 years, learning from failures, self-improving and generally economically “a very big deal” due to human replacement? Because that’s what AGI is supposed to do, rapidly—and that’s why it is worth acknowledging the possibility.
From my perspective, if it barks like a dog, walks like a dog… fulfills all the “dog” benchmarks… even if it came about through unusual means, you’d better consider the possibility.
All these experts Pooh-poohing the possibility of AGI because “I’ve worked with LLMs” is really unconvincing. The debate isn’t balanced because most professionals wouldn’t want to claim AGI for professional reasons.
Most professionals in my field would probably consider this a career-defining moment that would land one in the halls of the greats. I don't really follow this thinking.
And I get the "looks like a duck, quacks like a duck" argument, but as I've repeatedly stated, that's not good enough.
Frankly, I don't buy the "predictive coding" theory - at least, not to the extent that I believe the brain is only probabilistically modeling the phenomena it experiences. This ignores the glaring fact that our senses in-of-themselves exhibit some astounding capabilities right out-of-the-box, and predictive coding has no ground to stand on without the cross-referential analysis of the senses. Human babies are developing their intelligence long before any heavy stimuli. Hearing is the most active before birth, eyesight - our most rich sense - doesn't kick in until after birth. Nonetheless, babies come out of the womb with intelligent reflexes, such as rooting. That bundle of rudimentary intelligence is the basis upon which predictive coding begins - a functional default. An LLM is a set of statically-coded weights, incapable of context switching or object permanence. A child may take a while to understand the difference between "you" and "me", but they still can intuit "thing A" and "thing B". LLMs cannot - at least not without being paired with another model of some sort. And that model will be what tips the scale, not scaling up the features of what's merely a beautiful implementation of the logit function.
You also see that some basic production rules (for basic reflexes) could mean the difference between AGI chatGPT and chatGPT today. Because of how it would help chatGPT orient towards the world, I suppose?
I buy that critique because I do view a cybernetic loop as essential to intelligence — there must be some goalstate (however vague) to be achieved successfully.
But, have you seen many claim that chatGPT is AGI? My issue is not with your argument (it’s good!) but the lack of acknowledgment for the reasonability of the position that chatGPT can be called AGI. Most people DO buy predictive coding. And many would view the missing “reflexes” in chatGPT just a matter of good UI/UX (for instance). Yet, I haven’t seen anyone stand up and systematically argue that chatGPT is AGI. (But I’m not on Twitter, so…)
As for the assertions being made - at first, no, but once the Bing Search fiasco kicked off, I've seen more and more (here on HN) making the claim that ChatGPT and its kin are the beginning. Ultimately, it doesn't irk me - once we hit the same plateau as usual, the hype will boil down - but I do feel a bit concern at what may come of poorly-wielded LLMs before the industry wises up to their risks. A lot of people take ideas they see on HN back to their jobs, so we have a bit of responsibility to manage such hype here.
With hype I think of blockchain (wherein I never understood why removing people from decision-making through automated contracts was worth so much) or the metaverse (where I never understood how they envisioned a world with tech strapped to their face).
The public communication about AGI isn’t exactly a hype cycle. People should, instead, be expecting a fairly dramatic change in many areas of life over the next 3-5 years.
https://dnyuz.com/2023/03/08/noam-chomsky-the-false-promise-...
I think the whole stable diffusion and ChatGPT thing will come and go, almost as self-driving cars have. If you remember, just eight or so years ago, everybody was absolutely convinced that they were right around the corner. There was an insane amount of hype. Now, that hype has largely come and gone, with several companies abandoning their self-driving approaches or just flat out failing at them. There is still some hype, but I feel it has faded as people have come to grips with the premise, much less the difficulty of the implementations. The focus seems to be shifting back to driver assistance features.
Another thing is VR. So these stable diffusion and GPT methods are having their time in the sun. Yet again, just like VR and self-driving, proponents are trying to will these things into existence. Yes, they are impressive in some very specific cases, but they are not general. Just like stable diffusion, these GPT things will just flood the Internet with junk, either filling people with incorrect knowledge or driving people away, leaving the only remaining application to be advertising and trying to sell stuff.
Humans are seemingly absolutely obsessed with technology trying to replace humans and reality. That is the common thread between all of VR, self-driving, stable diffusion, and now ChatGPT. But this is both hard and destined for failure. People keep discovering that the role of technology is to augment humans and not to replace them. Engelbart knew this 60 years ago.
But Stable Diffusion and ChatGPT are here, and the last one has more than 100 million users. Unlike self-driving cars, the excitement behind this technology was not created by the promise of building "Stable Diffusion" or "ChatGPT", the excitement was created by the people who tried these models for free and saw their expectations subverted. I think the difference is abysmal.
Chomsky and his critics agree here. Chomsky doesn't like it, his critics are more open.
Our brain processes information much more efficiently so we humans still have an advantage for certain tasks.
Self driving cars excited people but they're still not on the market so of course it's easy to forget.
But why would people forget this and go back to doing laborious writing work themselves?
i think the idea is that by trying to replace humans we learn more about what it means to be human.
I have my own reservations about his opinion but would never state them in the combative and sneering tone often employed by his critics.
Can't help but feel like they personally have a problem with him, and they're using this piece as a cudgel to "take a swing".
If we are evaluating LLM's purely on outcome alone, it seems only fair to compare them to humans that don't have fully functioning cognitive capabilities. We wouldn't discredit a schizophrenic person that hallucinates, we know that this person can also see reality for what it is, most of the time. I just don't think that finding absurd examples, usually after careful prompting, is a slam dunk against LLM's at all.
They can, it's just that Chomsky's dismissal of "weight adaptations" as insufficient is not actually justified by any evidence. The Transformer model augmented with memory is Turing complete, so it can in principle learn to compute anything using only weight adaptations.
* On the Computational Power of Transformers and its Implications in Sequence Modeling, https://arxiv.org/abs/2006.09286
Our brains are much more complex than an artificial NN, though. So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
Well sure, because CFGs can have unbounded depth, and all real brains are finite. But they can learn bounded CFGs:
https://arxiv.org/abs/2112.09174
> Our brains are much more complex than an artificial NN, though.
Yes, brains have a few orders of magnitude more parameters, and lots of evolutionary baggage. It's not clear how many parameters would be needed without that evolutionary baggage. Time will tell.
> So the mere thought that the two somehow share functionality after a wildly different learning process is somewhat far-fetched.
I don't know what precisely you mean by "share functionality", but it doesn't seem like a stretch to suggest they are in some sense isomorphic given Turing equivalence. Rule 110 doesn't look like it should share much with Turing machines or the lambda calculus or SKI combinators, but these are all equivalent in a formal sense.
> in some sense isomorphic given Turing equivalence
Then you're brain is just like any CPU, which is very far from what such a statement tries to express in the context of NNs.
Yes, the brain is like a CPU + a specific program running on that CPU. The properties of that program are where the isomorphism comes in. So to come full circle back to your original point, we don't yet know that "weight adaptations" are not a sufficient operational model for that program.
In contrast, the world is too complicated to learn through exposure. Something has to structure our learning process. Language is (in principle) not learnable, only when you make enough assumptions. Those assumptions must be coded somewhere in your brain and probably genes.
Tabula rasa sounds nice, but it is untenable and utterly lacks evidence.
People who rely on LLMs to do their work in domains they understand find them useful in the sense of hefting another useful tool as humans have done throughout their existence. That's it. Among serious users of LLMs as a productive tool there's no wistful wonder that they may be speaking to a truly understanding intelligence. So Chomsky and the rest of the critics don't really have a foot to stand on when they say things like
> It is at once comic and tragic, as Borges might have noted, that so much money and attention should be concentrated on so little a thing — something so trivial when contrasted with the human mind, which by dint of language, in the words of Wilhelm von Humboldt, can make “infinite use of finite means”, creating ideas and theories with universal reach.
Why is it comic and tragic? LLMs are genuinely useful tools and one of my biggest productivity boosters at work since I learned how to make a computer do my bidding. Of course huge amounts of money are being poured in when it's so helpful across a number of domains for getting work done.
Really, folks, there's nothing to see here. Make use of these marvelous tools and continue doing what humans have done best, a truly innate human inheritance, to use tools to support our goals.
> However useful these programs may be in some narrow domains (they can be helpful in computer programming, for example, or in suggesting rhymes for light verse), we know from the science of linguistics and the philosophy of knowledge that they differ profoundly from how humans reason and use language.
I think you and he agree on your main point, but his main point is about something else.
Not that this is really relevant to this topic, but that's simply not true. He doesn't support Russia's interests, he's simply acknowledging that when dealing with powerful nations, we have to consider their interests and reach compromises where their interests conflict with ours in order to avoid the horrors of war. You know, diplomacy 101.
Has been doing that for decades.
He is intellectually dishonest and morally bankrupt. This makes his thoughts on other topics suspect as well.
Moreover how do you do diplomacy with Hitler2.0?
Thinking that giving up yet another piece of (for you) unknown land to one of those will bring peace makes you sound really naive.
Speaking of intellectual dishonesty making other thoughts suspect...
Chomsky used some examples in his article of what LLM's would get wrong. It is clear that he hasn't even tried, because chatGPT in fact gets those examples right. Why does he get a free pass for such errors?
Sounds like Aristotle when he claimed that women do not have wisdom teeth for reasons (he made up), instead of bothering to check.
You might want to check your wife's teeth :)
What did Scott Aaronson contribute of comparable impact?
This, I fail to understand. Clearly, Chomsky started with an exposition of the concept of negativity. How can this be missed by anyone with cursory familiarity with western philosophy? The rest of the article (including the passage regarding the apple) is pretty much a series of excursions on this theme, how LLMs systematically miss negativity.
Anyways, what I tried to suggest was that there is some base for reasoning that what we're encountering here is more like a cargo cult of conceptuality than conceptuality itself at work. And that it is still on us to decide, if want to be persuaded by this to land our planes of reason next to it, or not.
One thing grammar does not do is constrain grammatically-correct language to produce only the truth. It is as easy for humans as it is for LLMs to say false things, to construct false reasons for believing those falsehoods, and to be inconsistent about it. The argument Chomsky uses to claim LLMs will never lead to and understanding of the mind can equally be deployed against the prospect of understanding the mind through studying grammar.
If anything, the success of LLMs, in producing grammatically-correct language about as well as humans do, is evidence for this position. They also present a significant challenge to the Poverty of Stimulus claim[1], which is often used in support of the universal grammar hypothesis.
In the early 20th. century, western metaphysics took what has been called the 'linguistic turn'[2]. Personally, I am deeply skeptical of the proposition that studying human language will - or could, in principle - lead to knowledge about how the world must be, and the ability of people to hold inconsistent beliefs and to argue for falsehoods is one basis for that skepticism.
[1] https://en.wikipedia.org/wiki/Poverty_of_the_stimulus
[2] https://en.wikipedia.org/wiki/Linguistic_turn
In all fairness, this belongs only to context-free grammars. Also, I'm somewhat concerned that we may see a resurgence of behaviorism promoted along the lines. (It may be worth remembering what kind of regime this had been.)
As for behaviorism, it can stand or fall on its own merits (or lack of them), just as should the alternatives.
https://www.reddit.com/r/chomsky/comments/11gfo8m/why_are_ch...
Statistical models turn out to be enough to get past the Turing test. They're probably going to be better at interpreting vague requirements and making decisions than most humans too. They're better at art than the average, and better at explaining themselves than most humans.
[0] https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...
Using an LLM and being amazed by its response is not doing the Turing test. The Turing test is mostly a thought experiment anyway.
In some sense ChatGPT is actually engineered to fail the test.
Also, although I understand the concerns over super intelligent AGI, I suspect that in the short- to medium-term, that won't matter either. What will matter is highly optimized LLM's putting high swaths of the white collar workforce out of work, and the economic upheaval that will ensue.
Just to add some science fiction to the discussion: what if we are in an AI/human flippening moment? Where humans used tools for million years and orchestrated the goals while now we are moving to being more like a tools, in the sense that we are necessary some times but completely unnecesary in others.
Shiny tech will provide copious benefits, but cannot be what it is not: organic genius.
Humans are indeed 'trained' on a much larger set of data than 'just' the whole text of the internet. Indeed orders of magnitude more: think of all of the video/audio/olfactory/social cues/physics we absorb every day of our lives through all of our senses. How is that not infinitely more data than the public available text on the Internet? What is to say that a way larger LLM-like model would not be an AGI? why should human intelligence NOT be just a statistical engine?
After all, haven't we proven already that physics is more-or-less statistics at heart (quantum mechanics)? that doesn't make physics non-deterministic at human scale. So why would the result of an LLM-like model trained on reality be not-deterministic or wrong? How is a high-confidence statistical result different than reality?
In other words, nothing in Chomsky's article proves that humans are NOT statistical pattern matching engines. I am not saying that we are BTW, but it's an intriguing possibility that should be researched. After all, statistical methods and ML have made a lot more progress in the last 10-15 years than the MIT-driven, deterministic school of thought did in the previous 30. Cambridge lost, plain and simple.
Advances on probabilistic search will have equal facility with hallucination as with the real.
Regardless of whether Chomsky's article "proves" your negative, there is plenty of evidence that LLMs are not even as good as dogs and cats at forming a theory of the world around them. That we might draw a conclusion that dogs, cats, and people work differently from LLMs seems more like common sense.
What we lose in the drowning is the ability to distinguish quality. We forget which way is up, where the air is, and swim down further until it is too late.
The tragedy seems to be that during the hype of any technological bubble, someone of the scale of Chomsky is needed to make at least someone hear the obvious. But maybe it's also a comedy.
However, it doesn't matter. All flying cars will be on RISK V and Rust and on the blockchain soon. That's what's main.
No, the comedy is that he's saying things that seem completely obvious but are actually completely wrong.
Chomsky has around half a million citations to his scientific work, mostly on language. The guy is amazing. At age 94, he is productive, sharp and very well spoken. I tend to take him more seriously on this.
One problem is that, experts in one domain opine about domains they have not worked on. Another problem is, some people apparently hold a view that, if you make a product that works and sells, nothing else matters. Everyone will rush to earn a living around it. They feel empowered to take any position. So Science doesn’t matter if it’s not aligned, products do. Aaronson didn’t publish a proper article addressing Chomsky’s arguments.
"[Note for people who might be visiting this blog for the first time: I’m a CS professor at UT Austin, on leave for one year to work at OpenAI on the theoretical foundations of AI safety. I accepted OpenAI’s offer in part because I already held the views here, or something close to them; and given that I could see how large language models were poised to change the world for good and ill, I wanted to be part of the effort to help prevent their misuse. No one at OpenAI asked me to write this or saw it beforehand, and I don’t even know to what extent they agree with it.]"
This has been my biggest issue around tech recently, and the whole conversation on ChatGPT and the latest generation of AI stuff just makes it a lot more obvious. They rush to make it and sell it, and that's all that matters. Societal externalities be damned. There's a lot of people way too invested in making sure stuff doesn't get discussed, all around.
- LLMs are sometimes said to be “just” shallow pattern matchers, “just” massive look-up tables or “just” autocomplete engines. These comparisons amount to a form of (methodological) reductionism. While there’s some truth to them, I think they smuggle in corollaries that are either false or at least not obviously true.
- For example, they seem to imply that what LLMs are doing amounts merely to rote memorisation and/or clever parlour tricks, and that they cannot generalise to out-of-distribution data. In fact, there’s empirical evidence that suggests that LLMs can learn general algorithms and can contain and use representations of the world similar to those we use.
- They also seem to suggest that LLMs merely optimise for success on next-token prediction. It’s true that LLMs are (mostly) trained on next-token prediction, and it’s true that this profoundly shapes their output, but we don’t know whether this is how they actually function. We also don’t know what sorts of advanced capabilities can or cannot arise when you train on next-token prediction.
- So there’s reason to be cautious when thinking about LLMs. In particular, I think, caution should be exercised (1) when making predictions about what LLMs will or will not in future be capable of and (2) when assuming that such-and-such a thing must or cannot possibly happen inside an LLM.
https://www.erichgrunewald.com/posts/against-llm-reductionis...
Imagine an "AI" that "knows" about rhyme and meter and can generate a seemingly infinite collection of catchy songs that follow the II V I chord progression, only the sequence of words doesn't necessarily make any sense.
LLMs are a more powerful version of that kind of thing that does put the words in a sequence that feels like natural human language. Since they are trained on information, the sequences chosen often seem informational, even though they are just patterns. It's really a lot like autocomplete.
In other words, a lot of useful information may accidentally be encoded into the model, but so is a lot of stuff that looks/feels similar but is nonsense.
It’s completely true that LLMs are trained on next-token prediction (although some, like ChatGPT, are then additionally trained using reinforcement learning with human feedback). It’s also completely true that this fact profoundly influences the texts they generate. So I don’t think it’s unreasonable to call LLMs autocomplete engines or to emphasise next-token prediction. But I think it’s subtly misleading:
- Though LLMs were trained to optimise success on next-token prediction, that is not necessarily what they do. We don’t know what it is they do. The training process reinforces behaviours/heuristics in the model that tend to cause it to make better next-token predictions on in-distribution data. This does not mean that those behaviours/heuristics are fundamentally “about” optimising next-token prediction, especially when the model encounters out-of-distribution data.
- The usual example here is human evolution. Humans were shaped by a process that optimised for reproductive fitness. This gave us a bundle of drives such as family kinship, prestige and sexual pleasure – drives that aren’t fundamentally about optimising for reproduction, which becomes evident as we enter a new environment – one with contraceptives, say.
- Optimising for a task for which intelligence is useful encourages the optimised thing to become more intelligent. Sam Altman gave expression to this last week when he wrote, “Language models just being programmed to try to predict the next word is true, but it’s not the dunk some people think it is. Animals, including us, are just programmed to try to survive and reproduce, and yet amazingly complex and beautiful stuff comes from it.”
- The forms of intelligence that are useful in doing next-token prediction are different from those that are useful in human reproduction, but I think there’s a considerable overlap, as (1) some fundamental abilities, for example using and applying concepts, just seem very broadly useful and (2) the data LLMs are trained on are written by humans, for humans and often about humans and things that matter to us.
I think the "they're just autocomplete" take also hides other properties of LLMs, like them seeming to (as mentioned in another comment, and in the post) contain and use world models, and being able to learn general algorithms.
The point is is that dismissing it because it's "just patterns" ignores the fact that nobody has proven that human cognition isn't also "just patterns". If it is, then that undermines the whole justification for the dismissal. So the dismissal assumes the conclusion until that evidence is presented.
Chomsky's point that we have no basis for judging LLMs as cognition machines because we don't know what cognition is, is correct. Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either.
We do know one characteristic of cognition as it occurs in nature. There seem to be natural affinities to distinguish the real and true from the hallucination. Perhaps that comes from persistence and physicality. How do we give that to ChatGPT?
That's not Chomsky's point at all. He's saying they are emphatically not understanding or intelligent in any way, and that's not justified.
> Building these machines may produce useful tools, but it also doesn't advance studies of cognition, either.
I couldn't disagree more. We have here apparently intelligent systems completely unlike naturally occurring ones. IMO, you'd have to be crazy to think that comparing and contrasting the properties of these systems won't provide any valuable data about the nature of intelligence, perception and cognition as a whole.
In fact, we're already seeing papers discussing how LLMs basically refute Chomsky's whole model of language [1].
[1] https://lingbuzz.net/lingbuzz/007180
> So Noam's position is basically that doctors will never discover Vitamin C unless they develop chemistry.
I disagree with this analogy. LLMs and machine learning is playing with chemistry, and Noam's position is more like, "You'll never understand any chemistry unless you first figure out subatomic physics". That's just obviously false, we can understand and do quite a bit of chemistry without a full understanding of quantum field theory. At best, such knowledge impacts our understanding at the fringes of chemistry.
Literally the author used the analogy so I have no idea what your point is if you disagree with the author's analogy. It tells me you are not thinking through the piece analytically.
It's not an uninteresting paper, but by the author's own admission, he doesn't give an astute reader good reason to proceed to the next section, since that was the analogy that he tried to draw.
Those arguments and the empirical data that follows are exactly why I think my analogy is better, but it seems silly to dismiss the remainder of a paper based on that flawed analogy.
I'm also seeing this paper circulated on social media so I automatically start squinting when random people start promoting it that way. A quick search on Google shows that the author has had a 10 year long back and forth disagreement with cognitive linguists such as Chomsky's colleagues. That's a bit of omitted context that this social media amplification does not help with when portraying ongoing research debates.
Which is quite useful to do some tasks: writing fiction (it now wrote at least 6 characters in multiple rpg campaigns I participate in) and probably others (I used it to write tests data, but in fine I had to revalidate it by hand. Could do better.)
I do not see LLMs be more than that in the near future though. More informed, more accurate bullshit generators? Yes probably.
And I'm not saying it isn't impressive and very close to human behavior. I still think we're missing a part in our models though, and until we find it, breakthrough will be limited to improvements on accuracy and information.
Beyond that, many intellectual capabilities are also useful for successful bullshitting, so even if we can confidently say that that's all they do (in some sense), that doesn't mean they don't also do something-like-human-reasoning etc.
I don't really know how to explain it, but to me we have at least two kind of reasoning, swallow and deep. LLMs could well be capable of the first.
This is pretty unfair IMO. Aaronson wrote (emphasis mine):
> In this piece Chomsky, THE INTELLECTUAL GODFATHER GOD OF an effort that failed for 60 years to build machines that can converse in ordinary language, condemns the effort that succeeded.
He never wrote that Chomsky himself was involved in this efforts, only that he was influential to it (which may or may not be true, but Marcus never argues that point).
If a clock was correct 95% of the time, would you say it has no understanding of time? To me it seems that both Marcus and Chomsky are resorting to a sort of dualistic theory here. They concede that these LLM's are capable of impressive displays of cognition, but wait! It's not real cognition of course, it's fake, unlike the one in humans. Well, we don't know enough about cognition in humans to reject the hypothesis that we are also communicating or thinking based on similar principles. ChatGPT is by far the most successful automaton for understanding and producing language we have ever seen. The language Chomsky uses in the op-ed and in his reply here - "(...)Therefore they are telling us nothing about language, cognition, acquisition, anything" - is pure hubris and completely unjustified.
Similarly, we could sprinkle formic acid on the floor in a pattern that fooled ants into following it in search of food, but there is nothing "correct" about it, merely "realistic".
Asking GPT3 to create a persuasive essay is impressive in the same way. It "knows" how to string words together in a persuasive way, but just as some salespeople may not care about how factual their statements are, ChatGPT generates the veneer of realism but will often generate something that in spite of its realism is utter nonsense, or is even mostly "filler words" with little meat.
The veneer, filler words and style that humans (and LLMs) use in language are the parts that more similarly mirror the chemical signals used by insects. They are what we use to manipulate and to persuade rather than to convey understanding. This is why LLMs are scary when we consider their use in political misinformation campaigns, but less scary when we consider them writing dissertations about deep scientific topics.
My point is that we should not make these types of distinctions about how "we" "reason" versus how an LLM does. For example, I have no idea if your reply came from ChatGPT, but it sounds pretty intelligent to me. In that sense, your output is "correct" because it led me to believe I am arguing with an intelligent being. The analogy to a broken clock which is correct twice a day is ridiculous, because ChatGPT, on average, is quite capable of producing meaningful replies to a very complex problem (natural language queries).
I don't quite get your analogy to insects. If we figure out an analog pheromone to the ones ants use to communicate, then we found out something very meaningful about ants. Likewise here, I am quite certain ChatGPT is not producing language in the same way a human does, but is it using the same principles? Chomsky and others are prescriptive in saying it is not, yet their arguments are not convincing.
In the late 1990s, some undergrads would see Chomsky campus-wide talks as an opportunity to mock him, but it seemed to be about disagreement with his "politics". "Dept. of Leftist Linguistics", and worse. (The student body seemed overall libertarian, and of overall narrow exposure to the world outside STEM, compared to my previous university.)
Vs.
Therefore they are telling us nothing about language, cognition, acquisition, anything.
As anything with Chomsky, the hyperbole misses the point. A useful piece of technology still has uses. An airplane still flies even if it tells us nothing about how birds learn to fly or think about flight or even navigate the globe during an annual migration.
Not my main point but to be more nuanced I think ChatGPT could produce *some* amount of original science in places where people written about some phenomena but weren't able to connect the dots (come up with analogies shedding new light etc.). But ChatGPT won't go out looking for the dots.
I think it won't go looking for new dots because of 2 reasons.
1. It can't. It operates on language alone
2. It doesn't ask questions by itself. It only gives answers. It doesn't have the drive to seek knowledge. I'll try to illustrate my intuition: There was a gorilla Koko. She was taught to communicate with humans and could answer questions. But Koko didn't ask questions. She was observing the world but didn't have the concept of unknowns. The sphere of things she didn't know simply wasn't existing for her. I feel like ChatGPT is the same in that regard.
Back to main point. I have problem with mathematics in the context of ChatGPT coming up with *original* work. I don't know what is the current consensus in philosophy of mathematics about it being discovered or invented. If it's discovered (the fact that abstract mathematics seemingly not describing anything real happen to be used 50 years later as description of some part of reality would suggest that) then my 'not thesis - thesis' stands. It won't produce new original mathematics. If it's invented then it won't need contact with reality and it (theoretically?) could *creatively/originally* advance mathematics (once it already exist).Question. Mathematics originates in axioms. Maybe that's why it fits the world. Because people observing the world chose the axioms that made sense to them. So when it's advacing to it's logical conclusion it still happen to fit the reality. If mathematics wasn't formalized already would ChatGPT ever invent mathematics? Would it come with any kind of axioms?