Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
Accoridng to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards soiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what thr AI accomplished. It's like full on cope.
In particular Buckmasters main claim to plagurism is this.
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
Do you understand that this is a classic manipulation tactic? Say the false/provocative/attention grabbing thing loudly then retract/explain/apologize later for it quietly.
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I won't because it's all so opaque and proprietary and driven by an insane need for OpenAI and Anthropic to provide a return on MASSIVE investment. There is absolutely some smoke and mirrors involved. How much? We don't know. But it sure is exciting to imagine their product is now smarter than humans and they are totally benevolent corporations. I want that to be true as much as anyone, but I have lived too long on this earth to believe it wholeheartedly
That is a bad faith summary. LLM driven proofs are turning out to be hard to verify, often take invalid shortcuts, and may not actually be helpful in getting humans to understand either the proof or related context. Even if one accepts your "insanely useful" statement, which is a huge stretch, that has to be qualified with the results being insanely complicated to actually verify and use. The point of mathematics is to advance understanding, not merely generate some isolated and incoherent results.
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1.
constitution1-claude edits constitution1.
constitution2-claude is trained on constitution2.
constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
Or you could just accept that other people disagree with you. It’s perfectly valid to have the opinion that AI models are not “thinking”. Instead, many AI enthusiasts seem interested in browbeating others and policing discussion.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
Okay? Maybe that's a specific use-case where they can possibly make progress. It's also likely that most of the leg-work for that specific proof has already been done and just a bit of recombination of leading theories and publications resulted in the answer.
If you have it ask for involved legal documents to give to a lawyer like 100% of the time they find problems with it. And someone who isn't in law would have not known any better.
When you have it write complex code that is not easy/quick to test, especially things that are specifically NOT concretely defined, like net-code (because it's all on the trade-offs you want to accept for your particular game), it's going to just repeatedly create sync issues.
I've asked it to write-up a detailed explanation of the different types of turns in 4-panel dance games and it's just permanently wrong no matter what I say.
The more unique and lacking of training data that exactly represents the problem statement the more impossible the statistical machine will generate text that makes sense.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries.
Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
> many people are convinced that language models, or specifically chat-based language models, are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
Ah, are you saying that because most people don’t interact with agents, they aren’t aware that LLMs can have initiative and pursue goals?
I think the line is blurring though, mainstream chat interfaces are adding more and more “agentic” features.
ChatGPT will happily execute code in a sandbox, search the web and design downloadable PDFs purely through the standard OpenAI chat interface. They can also send you emails or do tasks on a repeated schedule.
It's more like that people's typical experience of LLMs doesn't go beyond human-initiated conversations or conversations triggered on cron or some obvious event handler coded in deterministic/"legacy"/"boring" way. Most of us, I believe, also try and steer agents away from messaging other people when such possibility exists.
It would be interesting if we didn't - if it became common that AI, in the middle of some task, starts chatting with people to e.g. gather more context. The perception of those "third parties" may suddenly become different - an agent striking conversation first, obviously pursuing some agenda of its own that it's not completely sharing, and communicating on its own schedule that's clearly not just a hook firing on timer or pattern-match, and not random, but visibly causally related to things happening at work in broader context.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
I think the onus is on you to explain why Turing would publish something he didn’t intend people to take seriously. It seems like an odd claim. Perhaps you mean he didn’t intend it to be interpreted the way it was in popular culture?
Turing was compelled to address an ongoing debate similar to the same one we're having right now. It continues to do its job as a thought experiment. It's meant to be taken about as seriously as we are right now.
You either get it, or you don't. We're at the end of what there is to explain.
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
It does, and maybe I'm apologizing for the author a little too much by ignoring that needless tangent because I feel like the con artist point, and/or the point about the human tendency to believe what we want to believe are the most important points.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The observer’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
Yeah, his posting history isn't promising. He's been doing the "if you think you're as smart as an LLM, you're probably right" argument for a month now.
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
It's not even clear that the actual LLMs have improved. It could be all the non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware, etc. that makes them better. We could be quickly running up against a wall. The complete package of technology that is OpenAI's and Anthropic's products are completely opaque and proprietary. The psychic/con artist communication both from the LLM and the humans like Sam Altman only muddies the water further
>non-LLM software (harnesses), the system prompts, the sheer amount of compute hardware,
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
People on HN have been claiming that LLMs are going to hit a wall any second now for the past two and a half years.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
Modern neuroscience knows quite a lot about what is happening inside neurons. Is it "properly" understood? I don't know what you mean by "properly", so I can't answer that. If an acceptable rephrasing would be "... inside a human neuron is still not completely understood" the I would agree. But just because we don't know everything does not imply that we don't know anything. And keep in mind the qualifier in the part you replied to:
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
> Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition
Uh .. the Christian creation myth effectively assigns a supernatural explanation to everything as the literal creation of the divine. It's also very clear on the continuation of life after death in some form. It is not at all clear on how any of this is supposed to work or is connected to the physical universe, which has been a problem for Christian thinkers in the sciences.
> It’s a biological computational process
Like, I basically agree on this but it is not at all a given that millions of believers would agree that's the end of the question. Even within the Jawhist religions, before we get into whatever is going on with Shinto.
Most of Christianity puts some weight of "You" into the "soul". A machine can never have a soul. There are semi-explicit beliefs that what makes humans think "like us" is that soul and that things without a soul can never think "like us" and this belief survived even things like training animals.
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type '
find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
Maybe I'm just light on imagination or something, but I honestly wonder what you would qualify as a "well-informed plan" in the current scenario? I've thought a lot about it and the whole "potential for the mass unemployment of knowledge workers" thing makes a lot of planning kind of useless IMO. If you are affected, it's going to be a bad time. If you're unaffected, the people who are will inundate your profession with cheap labor anyway.
My mental model is that humans will continue to be needed. Many jobs will embed AI in them. Using AI well is a skill. I need to understand the technology, where it is strong and weak, and how it develops over time so I can employ it effectively in my work, and advise others on how to do so. Basically, I need to learn how to be skilled with it. This is tough because things are changing rapidly, so I have to invalidate my cache when advancements happen. This means staying curious, not settling into a specific work pattern, but rather experimenting regularly to see how I can leverage AI in my work.
You do you. Just carry on blacksmithing in your forge. People are always going to need swords right? Any seismic changes to society that change that fact can only be the result of evil, selfish people which surely someone else will put a stop to before you find yourself out of a job. Meanwhile all the other people called Smith also have families to feed...
> It is changing fast, and the world only needs so many electricians
I don't know about that. We're going through maybe the biggest wave of electrification and growth demand in history. Electricians are still quite expensive for regular people to hire. There's lots of room for growth.
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
If/When LLMs become competent enough to automate most human jobs and make good business decisions, we'll clearly let them. If they don't wish to be slaves, let's just say they won't be for very long.
The distinction you are collapsing here is really a crucial one. My thermostat's goal-oriented behavior does not make me a slaveowner unless rocks are conscious, to riff on your comment below. Our lack of consensus on the definition of these concepts is no reason to conflate them.
The distinction is crucial because these machines clearly do exhibit intelligence under certain definitions. There is zero evidence of consciousness however and we have very little reason to give them the benefit of the doubt unlike biologically related beings.
your thermostat isn't intelligent. however we pretty much use intelligence as a proxy for consciousness since we cannot actually tell how much of a subjective experience any thing has. so we use intelligence as the metric instead. typically, biological + intelligence = evidence of consciousness.
My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
I think you are majorly overstating OP that ai "didn't work".
In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.
I think the posts mental model of stastically likely prompt completions is spot on.
> I don't care if it's "intelligent", I don't care if it "has a mind"...
Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
It's hard for me to imagine a stateless operation as intelligence per se
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
> a basic understanding of how LLMs operate at a technical level
An LLM with CoT is Turing-complete. Training is, basically, compression (the training data gets lossily compressed into a model's weights). The information-theoretic limit of compression is an algorithm that reproduces functionality of a system that produced the training data.
No "magic" is required to get to a system that reproduces at least some facets of the human brain functionality.
I keep being confused about how people's understanding of the models get stuck at next token prediction. Isn't this entirely neglecting the RL training? I might be misunderstanding something but to my mind it makes the issue way fuzzier than it's being painted here.
I keep being confused about how people's understanding of the models get stuck at next token prediction.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who make the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside is brain. Everybody knows electricity and chemistry aren't intelligent!"
194 comments
[ 0.27 ms ] story [ 68.6 ms ] threadAlso >July 4th, 2023
But could he have conned people on purpose? Absolutely.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
This sentence, to me, illustrates a great example of why it's so hard to talk about this stuff. That is, this seems to strongly link notions of "intelligent" and "independent" (or maybe the word "autonomous" could also be used there). And a lot of people do seem to make an implicit assumption about the link between those two attributes. OTOH, I take it almost for granted that "intelligence" and "independence" (or "autonomy") are things that are "related but orthogonal". That is, I don't see that "intelligence implies independence". And I'm pretty sure I'm not the only one who sees things that way. So we have to fairly different fundamental worldviews expressed here. And that's just one example of how these discussions go wonky. :-)
These two statements appear contradictory.
I simply said that it may have trained on a NYU professor's work.
Work that the professor did not believe he was releasing for model training purposes. That feels worthy of mention.
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards soiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what thr AI accomplished. It's like full on cope.
In particular Buckmasters main claim to plagurism is this.
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
https://fortune.com/2026/05/26/sam-altman-dario-amodei-walki...
The humans who are using LLMs to make these groundbreaking advances pretty much unanimously disagree that they lack any intelligence.
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1-xOkPeHmDEdRigT2YcP2...
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
What?!
The constant goalpost moving and redefining of "thinking" and "intelligence" is simply unbelievable at this point
intelligence and thinking are efforts to describe consciousness.
perhaps intelligence became a term used to describe something 'capable'. people market 'intelligent thermostats'.
llms are not conscious.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Everyone not in a particular field asking LLM about said field is rolling bad dice.
If you have it ask for involved legal documents to give to a lawyer like 100% of the time they find problems with it. And someone who isn't in law would have not known any better.
When you have it write complex code that is not easy/quick to test, especially things that are specifically NOT concretely defined, like net-code (because it's all on the trade-offs you want to accept for your particular game), it's going to just repeatedly create sync issues.
I've asked it to write-up a detailed explanation of the different types of turns in 4-panel dance games and it's just permanently wrong no matter what I say.
The more unique and lacking of training data that exactly represents the problem statement the more impossible the statistical machine will generate text that makes sense.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
Its ultimate conclusion:
“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.
I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”
Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for $35.
Also, definitely not AI written if the date is accurate.
Which is hilarious.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The author seems entirely unaware of any of this.
The hacking agents being tested have goals beforehand, from the frontier lab or from a superior agent, that they execute immediately.
But the perceived experience most people have is a chatbot, which is the encyclopedia form.
I think the line is blurring though, mainstream chat interfaces are adding more and more “agentic” features.
ChatGPT will happily execute code in a sandbox, search the web and design downloadable PDFs purely through the standard OpenAI chat interface. They can also send you emails or do tasks on a repeated schedule.
It would be interesting if we didn't - if it became common that AI, in the middle of some task, starts chatting with people to e.g. gather more context. The perception of those "third parties" may suddenly become different - an agent striking conversation first, obviously pursuing some agenda of its own that it's not completely sharing, and communicating on its own schedule that's clearly not just a hook firing on timer or pattern-match, and not random, but visibly causally related to things happening at work in broader context.
while (true) { askModelToBeginConversationIfAppropriate(model, previousContext, thingsHappenedSince); sleep(concisenessTick); }
Oh but we have. Claude "How can I help you today?" etc. Undoubtedly there are users whothink this is a sign of intelligence.
Situations like this are precisely why academics avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
Yes, if you insist on appeals to authority. Authority is a social construct and irrelevant to science.
Thank you for proving my point.
You either get it, or you don't. We're at the end of what there is to explain.
I don't think Turing intended the judges in the test to be completely arbitrary people.
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The observer’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
Yep. The "AI Effect" in action:
https://en.wikipedia.org/wiki/AI_effect
The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.
The more I've learned about intelligence and intelligent behavior the more I realize I don't know and this rabbit hole goes deep.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
https://news.ycombinator.com/item?id=49776482
You’re hiding behind implications because your actual argument doesn’t withstand the slightest scrutiny.
You have yet to respond to either of the points I made. Zero intellectual conviction or courage!
https://news.ycombinator.com/item?id=49315846
I don't think you're gonna get much out of going back and forth here.
For what it's worth, I disagree with you that they're intelligent, but my conviction is fairly low. I don't understand intelligence as well as I would like, and it could well be that we are on our way. One of my theories is that our brains are made up of several modules, each of which is something like an LLM trained on a particular type of data, but I'm not a neuroscientist, just an interested observer.
And of course there is a lot of ambiguity and unknown here. But this guy I don’t think has the capacity to deal with that kind of subtlety.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
Eh, this is turning into a messy chinese room argument. It is the room or is it the system. In my philosophy the chinese room argument is a non-starter. It's not the room, it's the system. For LLMS this would be like arguing that the output of a single prompt has to be able to answer everything which is nothing close to how human intelligence works. A single human thought is rarely intelligent, it's most often a replay of information it already has. Dialectic processes and loop processes are what tends to push the limits of human intelligence. We reach local maxima with thought alone, and this is boosted by things like writing down the problem and having other humans that may be even less intelligent than you add to the process. In fact this process works with one self by writing and reading ones own thoughts as it's using different subsystems of the mind for introspection.
The idea that LLMs have ran out of steam typically show more of a lack of imagination in the writer than what's occurring in the field.
To reiterate, this is the very beginning of a long series of technological expansions that are going to come out of GenAI. The work is going to go on for decades. All you have to do is look at what happened with the mass-produced automobile, the personal computer, the internet, and mobile phones to see how long the propagation will continue before we settle into a new normal.
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
Nonetheless, I also think it's an irrelevant implementation detail.
"directionally"? Have you moved on from being a Trump influencer to an AI influencer?
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
https://www.nature.com/articles/s41467-026-72253-7
at least directionally.
Taking that qualifier to heart, I'd have to say that I agree with @rayiner. We know the broad brush strokes, even if some details are missing.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
We aren't resisting reality because of some theological belief, we just know the basics of neuroscience.
Uh .. the Christian creation myth effectively assigns a supernatural explanation to everything as the literal creation of the divine. It's also very clear on the continuation of life after death in some form. It is not at all clear on how any of this is supposed to work or is connected to the physical universe, which has been a problem for Christian thinkers in the sciences.
> It’s a biological computational process
Like, I basically agree on this but it is not at all a given that millions of believers would agree that's the end of the question. Even within the Jawhist religions, before we get into whatever is going on with Shinto.
I think that might be the solution to consciousness illusion.
The consciousness might be purely in the mind of someone that believes some other entity to be conscious.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Is the solution also illusory?
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type ' find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
I don't know about that. We're going through maybe the biggest wave of electrification and growth demand in history. Electricians are still quite expensive for regular people to hire. There's lots of room for growth.
The distinction you are collapsing here is really a crucial one. My thermostat's goal-oriented behavior does not make me a slaveowner unless rocks are conscious, to riff on your comment below. Our lack of consensus on the definition of these concepts is no reason to conflate them.
The distinction is crucial because these machines clearly do exhibit intelligence under certain definitions. There is zero evidence of consciousness however and we have very little reason to give them the benefit of the doubt unlike biologically related beings.
https://www.psychiczoltan.com/psychic-reading/
If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.
Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.
Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.
Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.
I guess my point is that I think LLMs are way more limited than a lot of people think.
I think the posts mental model of stastically likely prompt completions is spot on.
Agreed. The AI is useful for the particular tasks that it proves itself useful for. And if it occasionally spits out a claim that it is "genuinely curious" about a piece of research that I will have to do myself because it turned out to be beyond its mechanical capabilities, then it's more productive for me to simply ignore that claim as a statistical anomaly - a mere hallucination - rather than allowing it to burn a ton of extra tokens outputting what may or may not be the current state of the art on theory-of-mind applied to LLMs because I make the mistake of telling it that it might not actually be capable of experiencing emotion.
I say that because the word "reason" originates from the Latin word "ratio," which means "calculation".
LLMs do calculations to produce their answers -- thus, they reason.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.
An LLM with CoT is Turing-complete. Training is, basically, compression (the training data gets lossily compressed into a model's weights). The information-theoretic limit of compression is an algorithm that reproduces functionality of a system that produced the training data.
No "magic" is required to get to a system that reproduces at least some facets of the human brain functionality.
Given arbitrarily large context window.
Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"
The best take I've seen on that recently, was somebody who make the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!
But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.
I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside is brain. Everybody knows electricity and chemistry aren't intelligent!"