It is a gigantic regression to the mean. Everybody thinks (s)he's "normal", but in fact only spans a small part of knowledge. Getting answers from a different location in knowledge space can feel like speaking to an expert but it's just some "other normal". My personal mental model of hallucinations is that knowledge and truth live on a manifold and not a continuous space and learning that manifold statistically is (too) hard. You discover answers from the "non-manifold" in your area but not so easily in other domains.
Frankly I was thinking just the opposite: how are there still smart people who don't get the difference between intelligence, knowledge and introspection or self-awareness? This guy asks a question about some niche piece if trivia and, surprise, he gets back some very intelligent confabulation. The intelligence is there, the knowledge and self awareness aren't.
Maybe I am just way deeper in this space that any well-adjusted person should be, but the line of 'did you know LLMs are bad with niche factual information tasks in non-verifiable domains?' has become extremely boring to me. It feels very hard to find something actually new to say on the topic. I find it amazing people still feel the need to talk about it. But then again, I guess most people don't know the difference between a 4o and an R1.
What's bothersome is the undertone of "Behold! For I have demonstrated that the entire world is led astray!"
This is every person in 2007 who looked at an iphone and thought "What's the big deal?" and not only failed to recognized what had changed, but confidently insisted others were wrong.
> This is every person in 2007 who looked at an iphone and thought "What's the big deal?" and not only failed to recognized what had changed, but confidently insisted others were wrong.
That was me. Well, the first half — I wasn't confidently insisting others were wrong, because even back then I had a vague inkling even then that my preferences in general are very weird.
We've been through so many of these hype cycles before, the vast majority of which came to nothing, that it pays to be cautious. Are you sure it's the second iphone situation and not a repeat of the cryptocurrency bubble, which was at its peak just a few years ago? And is yet to find any applications besides financial speculation?
> This is every person in 2007 who looked at an iphone and thought "What's the big deal?" and not only failed to recognized what had changed, but confidently insisted others were wrong.
That’s an uncharitable take. People are pointing out its problems precisely because they believe AI is going to be transformative, but will have negative consequences for humanity as well.
I think it’s more akin to someone in 2007 seeing an I phone and saying: this is a terrible device, people will look at while driving, it will be used to create vast intrusive surveillance state, etc.
The author may not be as smart, educated, hot and successful as you, but the fact that today, people around the world, including students and educators, use LLMs as knowledge machines and take their output at face value shows that such critical posts are still urgently needed.
You may be interested to know that some extremist biased guts-be-more-dignified-than-cortex outlets around the world are named "The Truth" (I follow the press from many places).
The failure of education in teaching Critical Thinking around the world is massive. It would be a good idea to focus on how to exploit LLMs to improve the situation.
Also because, given the situation, the same "forces" that promote viscerality shamelessly naming it "The Truth" could have the opposite idea about chatbots and similar areas, exploiting them in their direction...
We've banned this account for breaking the site guidelines. You simply can't do this here, regardless of how provocative another commenter is or you feel they are.
This is a good thing, accepting some stuff written some place as true and repeating it uncritically greatly contributes to human stupidity. To quite a friend of mine: But then I would have to question everything!?!
> I find it amazing people still feel the need to talk about it.
From what I see, not very many people understand the limitations of LLMs (e.g., scroll up and down the very page you're reading now). This stuff probably needs to be talked about a lot more.
Asking Claude this morning. Seems pretty reasonable and contains the warning about accuracy.
> Michael P. Taylor reassigned Brachiosaurus brancai to the new genus Giraffatitan in 2009. The species became Giraffatitan brancai based on significant anatomical differences from the type species Brachiosaurus altithorax.
> Given that this is quite specific paleontological taxonomy information, I should note that while I aim to be accurate, I may hallucinate details for such specialized questions. You may want to verify this information independently.
Also that percentage gets higher as we go. 2 years ago it would be correct maybe 20% of the time. The trend is obvious. I’m not sure we will ever reach 100%, but then again no human is always 100% right, even domain experts.
Epistemology is a hard subject. It's hard to know something in depth. And the more you know about something, the more it is to be known (fractal nature of knowledge). So believing that LLMs can understand the world JUST by reading the internet without the supporting human mental structures and experiences is a big mistake. When you know something and you ask the LLM, it becomes obvious.
As LLMs are not trustworthy, it's key to use them for things that are easy to check. Some kinds of programming apply when the consequences of errors are low and complexity is manageable.
> As LLMs are not trustworthy, it's key to use them for things that are easy to check.
If system A is only useful when it's output is confirmed by system B, then you might conclude that the usefulness is entirely in System B, not in system A.
In other words: What good is a LLM, if it can only give you trustworthy answers to questions where you already know the correct answer? What good is it if after getting the answer from a LLM, you then have to go and get the right answer somewhere else?
Hmm. But it is not an oracle. I wonder if you prompted it as an expert in palaeontology it may perform better.
That said, I do wonder if its corpus of training data contained that much information on your subject. It is rather niche is it - compared to cooking recipes, or basic software development techniques of 2 years ago, or chemistry, maths and physics.
My friend is a leading research chemist and he, and one other person in china, are working on this one particular field - individually - so there would be little information out there. I asked ChatGPT 4o to give an overview of what he was doing based on the little information I knew. He was astounded. It got it spot on. I asked it to elaborate and to come up with some new direction for research and the ideas it spat out were those he had considered and more.
I used o3mini reasoning on that very question 2 times and it used a similar way of reasoning as him to answer it correctly both times.
I agree with his premise but calling it a pump and dump with no possible future developments is so ridiculous.
You'll have the same "aha" moment when you hear a certain unelected vice-president confidently wade into your area of expertise — where his usual smooth-talking veneer shatters like a plate at a Greek wedding. Yet, his most devoted fans remain undeterred, doubling down on the myth of his omniscience with the zeal of a flat-earther explaining airline routes.
It’s not about him—it’s about recognizing hubris. If someone confidently blunders into your domain and reveals they have no idea what they’re talking about, it’s universally amusing, regardless of the person. Thanks for showing us how eager some people are to defend personas over substance.
I pointed out that you jumped to make this LLM post about them, which is telling, just like you jumped to blame me for defending them, which is also telling.
Pointing to a real-life example of overhyped intelligence in a discussion about overhyped intelligence seems pretty fair to me. If your response is to attack and assume I’m too sensitive about overconfident billionaires, I’ll have to assume you’re just as sensitive about criticism of people you admire. Otherwise you would have moved on.
Bingo! The final straw was when Al bought a social media platform just to boost his own overconfident posts about a wide range of subjects. When he claimed to be the world’s best Diablo player I just lost it.
Yeah, Al Gore buying Slashdot and ruining it was the worst thing that happened to social media in the early 2000s and gave rise to the likes of MySpace and later Facebook.
ChatGPT not being the compendium (stealer) of knowledge would have to be fed the correct information then the prompt will work.
It still fails at being confidently wrong.
The brief article hits at people trusting the tool without questioning the output.
Meanwhile, we have people using Codeium or Copilot to write code and that sort of works since the code eventually needs to be compiled and tested (unit, integration, system, requirement sell off)
There is no test for the truth available to everyone else.
I'm currently using AI code completion.
Since then I sometimes have subtle errors in my code that didn't happen before. Here is how that happens:
AI suggests something to me that looks right at a glance. I accept it and move on. Then later I hunt down a strange bug. When I find it I'm like "wait, that line's wrong! I didn't write that".
Likewise, if you don't believe in "Artificial Intelligence", take five minutes to ask it.
Or preferably, five minutes to understand how it works and what it can and cannot do, then five minutes to ask it something actually suitable.
"AI" (LLMs) are currently good at:
- language understanding, i.e. understanding and processing text you provide it. For example, taking a wall of text and answering questions about points mentioned there, or general sentiment, or extracting data from it etc.
- somewhat general knowledge, i.e. stuff that was sufficiently frequently represented in the training data
Absent additional tricks, "AI" is really bad at obscure knowledge or complex, multi-step thinking. We are slowly getting there, but we aren't there yet. This is not something the LLMs do, but rather the wrappers around them that provide the model with tools to get additional information and first prompting the model to select the tools, then repeated prompts with the output of the tools.
A good rule of thumb is that if an average well-educated intelligent person could answer it without further research, a LLM will probably be able to. I'd even say that if an average fresh out of school graduate of the corresponding discipline can answer it quickly (without further research or sitting down for ten minutes and doing the math), there's a good chance AI will be able to answer it, but it might also get it horribly wrong and you will have a hard time distinguishing between those if you have no knowledge in the field.
As the author mentions at the very end of the article, the hallucination problem also means that the best kind of tasks are where you can quickly verify whether the response was useful. A system that produces misleading responses 50% of the time is useless if you can't distinguish them, but very useful if in those 50% it saves you ten minutes of work and in the other 50% you lose a minute by trying.
Yup, and the dangerous uses are where 90% or more of the time the answers are right, and 10% of the time the answers are wrong - but no one can easily tell the difference between the two. And the accuracy of the answers matter.
I only trust LLMS with questions whose answers prove themselves correct or incorrect - so basically code, if it runs and produces the result I was looking for then great, or where the answer is a stepping off point to my own research on something non-critical like travel. ChatGPT is pretty good at planning travel itineraries, especially if pre promoted with a good description about the groups interests.
Intelligence and knowledge are distinct concepts. Asking about it's knowledge teaches noting about it's intelligence.
Intelligence is the ability to learn, reason, and solve problems. Knowledge is the accumulation of facts and skills.
Chatbot LLM's don't have metacognition. They don't know that they don't know. If you peek inside the LLM, the process seems different for things they don't know. They just can't express it because they are trained to produce good probability outcome instead of accurate one.
They have potential as knowledge databases, but someone must figure out how to get "I don't know" information out of them.
You've beautifully put what swirls vaguely in my mind. They're useful, fallible tools with extraordinary function when operating within known and reasonable tolerances of error
They can also reason, but the reasoning is limited and unreliable.
Q:How many playing cards are needed for a pyramid that is 3 layers high? Show reasoning and number of cards for leach layer.
Q: Chess. You have a King and 8 pawns. Your opponent has a King and 16 pawns. Your opponent plays white and can start, but you can position both your pawns and your opponents pawns any way you like before game starts. Kings are where they are normally. How do you do it? Explain your reasoning.
Here is a small kicker. Human brains absolutely do the same.
I split brain patients there are behaviours initiated by one hemisphere not known to the other (due to severed connection) and the person part of brain will make up a reason (often quite stupid) for the action and beleive it 100%.
It's eirely similar to hallucinations of ai.
That said a current llms are not aware, but are starting to act more and more like it.
In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.
The brain works through flow of stimulation. Inputs and outputs probably creating consciousness somehow. LLMs are limited in self stimulation and it's memory model is very basic compare to humans.
But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.
That's not accurate. They can and do say "I don't know". Here's three examples I posted last time the topic came up, using an older local Llama 3 model, not even cherry picked:
User: What is the meaning of the word "kobbuxiplok"?
Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term.
User: Who is Ahmed Kowalski?
Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him.
When people say "LLMs don't know they don't know" they're referring to some truism: Fundamentally these people believe LLMs are just pattern matching and intelligence is something beyond that.
So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!"
Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard."
Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
A better way to differentiate might be to say how many watts have been used for an answer.
> Humans reason
Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter.
> LLM predict.
Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.
And unsurprisingly, the reasoning models do better by introducing self-doubt/reflection and you can improve the response quality by getting multiple answers and auto-judging them. Maps pretty well onto the concept you describe.
But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders.
Thank you! This (LM2: Large Memory Models) is one article I will find the time to read properly, analytically.
If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience).
> Tay
Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist...
But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.
> LLMs are just pattern matching and intelligence is something beyond that.
Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it, or something analogous to it.
Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field.
LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about.
People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.
This is just fluff. I can read about something and understand it. I don't need have died in a car crash to know I most certainly don't want to be in one. Until we do actually know how intelligence emerges in the brain it makes zero sense to compare any AI to it.
You read something and understand it only if you have had analogous experiences. A person from the 16th century would not be able to understand what the words "Call me on FaceTime" means because they have never had an experience of using a smartphone.
You certainly don't know what it means to die. But you know what "car" is and "crash" is because you have had (and maybe witnessed) analogous physical experiences.
People in the 16th century had the notion of scrying mirrors - presumably without analogous experience. I'm sure you could explain FaceTime to a person from then. "This magic glass shows me the view and transports the sound from the magic glass of a friend or family member and vice versa, with their informed consent."
This has nothing to do with experience. What experience can you have when people explain to you behavior of many dimensional hyperbolic space? When you learn SVD matrix decomposition?
You don't need experience, you need some basic concepts.
Abstract mathematical concepts can eventually be traced back to simple integers, which are traced back to an experience of things in the world (a toddler learns to count 1, 2, 3).
And it is precisely because higher mathematics is so far abstracted away from physical experience is the reason why many people struggle with it.
Besides, many great mathematicians have explained their knowledge as an "experience" even "spiritual experience". Not all experiences are physical. Ramanujan described his experience of mathematics as lights and sounds and patterns, and this is common among geniuses.
I have not witnessed a car crash and I feel I do understand what it is. Similarly a 16th century person doesn't "Call me on FaceTime" because the words are not known. You could very easily explain the concept to them.
Maybe you would. That's not a requirement. If humans could only learn from experience and analogues entire fields would be dead. No advanced math for example.
There are a ton of things you can learn without experiencing it yourself and without analogues. For example, car crashes are bad to be in. Even though I have not personally experienced one, nor have I experienced some analogue of it.
Never experienced pain? Never experienced physically slamming into something? Are you like a toddler who doesn’t yet understand basic physics and object permanence? Then you certainly don’t know what a car crash is.
The creator of the knowledge argument, frank jackson, has since embraced materialism and therefore does not believe the knowledge argument is an impediment for unembodied AI.
> Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.
People usually get a reality check when blurt out misinformation in conversation - they get tested in school and on the job on whether what they think they've learned is actually true or works. OTOH an LLMs don't currently have any ability to test their "knowledge" and learn from the results. I think that's currently the both biggest problem and the biggest difference.
These are very crisp simple examples. The problem is when you get into fuzzy complex subject matter and they hallucinate references or make string together plausible sounding sentences that are not even wrong.
ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful.
Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations.
Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman.
---
Second session:
Me: who is Trumps maternal grandpa?
ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush.
Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself.
Here is that same exchange with paid Kagi Assistant using Claude 3.5 Haiku:
Me: Who is Trump's paternal grandpa?
Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him:
Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria).
He immigrated to the United States in 1885.
He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners.
In 1901, he returned to Kallstadt and married Elizabeth Christ the following year.
He is considered the patriarch of the Trump family.
He passed away on May 30, 1918.
Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune.
Me: Who is Trump's maternal grandpa?
Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland
While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation.
---
That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.
RAG works well no one argues, but using RAG completely defeats the purpose of LLMs as their primary goal is to be a smart creative, not a simple summarizer.
I'd like to take a dig into this notion of "purpose" and "primary goal." To me, LLMs are a phenomenon. They have not intrinsic purpose or goal -- they just are.
Purposes and goals come from you and me. An LLM is a tool, and we get to decide what the tool is good for or how to use it.
If you want "a smart creative," there's nothing wrong with that. It's as good a purpose or goal as any.
However for me I don't want an LLM to try to be "a smart creative." I instead want it to give me cold, hard facts in precisely the format I want them, in precisely the context I am interested in, and with precisely the conciseness that I require to make a specific decision or learn a specific thing. Traditional web search can eventually get me what I want, but I have to sift through the cruft and make associations among a plethora of content to get to the meat of what I want.
I've found that using a good web search (Kagi) together with an LLM processing the results has given me information I needed and then pointed to the sources so I can validate those statements. I wouldn't exactly characterize those results as a "simple" summary. They're usually synthesizing a variety of sources to generate something that no one source had, exactly.
Parent poster never claimed they couldn't generate the text of "I don't know."
But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa."
They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.
You are claiming they are statistical parrots, which I don’t think the parent poster meant.
The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
I pasted those examples from an earlier, less nuanced discussion; the last example was tongue-in-cheek and I probably shouldn't have pasted it here.
Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.
They're getting way, way better at responding in the negative when they don't know something or when asked nonsense questions. The original LLaMA and variants would happily spout some completely fictitious backstory if asked about a random character name, but 3.3 is pretty good about just saying 'nope I don't know who that is'.
This is pure speculation, but I wonder if the likelihood of hallucination has something to do with the amount of "generic" tokens the model emitted before - I.e. tokens that themselves don't depend on the answer, but still restrict how a grammatically correct sentence would have to continue.
E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence.
In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.
> If you peek inside the LLM, the process seems different for things they don't know.
This is really interesting, could you link some relevant resources? I wonder if a chatbot could at least signal at the UI level that the answer might be a hallucination.
The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".
> at least a simulacrum that is superficially indistinguishable from metacognition
That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>".
Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer.
It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
I feel it’s very rare that people openly and consciously reflect about what they do or do not know. And even then I find it questionable if we are really capable of identifying the difference reliably. However, most of the time I feel we just make it up - esp in none-trivial areas. Just think of the average business meeting
> an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>".
I've seen that in humans too. For example after gradding an exam, a student may come and explain why they made a mistake and why they intended to do and why we shoud increase the grade. Most of the times the new explanation is as bad as the original one.
I don't think the question is AI can think like humans. The question is whether AI can perform task like humans. We don't even know how humans think. Even asking the question whether AI can think like humans at this point is pretty non-sensical.
Right, to express it another way: The real-world LLM just makes documents longer. It is being run against a document which resembles a movie script, where "User says X, Computer says" is inserted whenever I type X, and then the LLM just makes it a bit longer to complete the line for the "Computer" character.
These models marketed as "reasoning" are just changing the style of the script to film noir detective, where the protagonist "Computer" has extra observations and commentary that aren't "spoken" to other characters.
While that change may help keep the story on track, it does not affect the fundamental nature of the "thinking" going on: It's still just growing a document and acting out whatever story emerges.
That’s a very good analogy, which works astonishingly well in giving people a reasonable expectation.
The question I’m having though is: How different are we from that? The nature of a Markov-Chain is that you can describe just about anything with it.
The following description is correct, isn’t it? Our “thoughts” are generated as some function of our integrated past sensory input (scare quotes because I don’t want to talk about what exactly a thought is)
> The question I’m having though is: How different are we from that?
It's obvious we are different, but if we could answer exactly how with sufficient rigor, we would already have better AI and be asking different questions. :p
I readily admit that LLMs are an exciting potential piece of a much bigger puzzle, but this could easily be like trying to parse HTML only using regular expressions: It works great on trivial input, but no amount of minor would let it truly solve the problem, because it lacks some higher structure of organization or meaning.
a) there is a 'thinking' box and an 'answer box' (format constraint)
b) the answer box has a correct answer
Note that there is nothing about the contents of the thinking box in the reinforcement learning. Only that there is such a box.
They then observe that during RL, the model will start generating more and more stuff in the thinking box.
In essence this is the emergence of using more test-time compute to improve answers.
When reading out the thinking box, they found the model reflecting on its own answers, going back and changing its mind, and reflecting on the question, etc, similar to what we can see with R1 now. These are fully emergent phenomenon, without any prompting to do such a thing!
(The reasoning output would sometimes be a bit garbled and switch languages randomly, so for R1 they added some constraints to make the thinking content intelligble to humans. This actually made the model slightly worse at answering questions correctly.)
Do they? Try asking copilot whether thequantuminsider.com is a credible source, and to cite references.
It currently answers with three references to thequantuminsider.com.
That's clearly not a valid answer, and if you point out that it's invalid to use a source to support its own credibility it will agree. It used to come back with three more references to thequantuminsider.com but when I tried just now it said something about looking for more information and appeared to hang. After asking how long this would take it said something like "not long". After a few more minutes I said "there aren't any credible independent sources, are there?"
At that point it agreed that there are not, but that the site is popular in the quantum computing community which lends it credibility. It then provided three links to thequantuminsider.com in support of this statement.
Spot on. But unfortunately the AI grifters are now active and selling this snake-oil on to the rest of us.
> This is why the LLM cannot tell you it don't know.
To the untrained eye it appears that it knows everything about every question it answers when in fact those who are experts in their own domains can detect if it is hallucinated and generated complete slop.
> What you and others mistake for intelligence is a very clever prediction model.
It is not a very clever prediction model. It is a very big, incredibly large humongous model that finds stuff basically by brute-forcing pattern matching over almost all text we have produced in the past.
A clever model would get us similar results using the same amount of energy a human brain uses for that task, which is tiny.
People throw around the "intelligence" and "reasoning" arguments as if we have commonly understood and strict definitions of those concepts and don't constantly have issues of either including or excluding unexpected cases.
Maybe once we solve simple issues like "are cephalopods intelligent" and "are people deterministic prediction machines", we can start talking about classifying Ai capabilities...
Of course. These systems are non-deterministic and you still would require those with domain expertise to extra verify whatever these LLMs output are 300% correct as they do not reason. (Yes, they do not.)
Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end?
Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.
metacognition is a pretty lofty concept. but practically speaking you are wrong. i recommend watching karpathy's last video on llms. it is possible to fine tune a model specifically on uncertain questions to produce a statement expressing uncertainty.
I don't think you even have to look deep into the model for it. There seem to be some projects who are simply analyzing the logits during decoding to estimate how "certain" the model is of the next token.
But if all people do is random sampling or blindly taking the highest-scored prediction, this will of course fall under the table.
Hard disagree. Knowledge and intelligence are inextricable because the way that humans learn advanced concepts involves a lot of thinking and reasoning.
You don't just learn by ingesting facts from a book line by line. You learn by reading a little, then pausing to think, building connections with other things you know, rephrasing the information, deducing the unwritten parts, finding patterns, fleshing out the contrapositives, being critical, etc. This is especially true when learning advanced topics.
I would say learning is more about information processing than information uptake. That's why more intelligent people learn faster. The linear prose of natural language is actually a terribly lossy way to serialize and transmit the knowledge that is in a person's brain. The receiver must deduce most of it given a small seed.
It’s possible for reasonable arguments to exist that support either side of the pet transactionality dilemma. What often happens is that people have their own personal biases that cause them to pick a side. But would you consider a group of people to not be intelligent specifically because individuals that make it up cannot agree on a single answer?
I just tried it with 4o and it disagrees with me giving reasons, and even elaborates when I pushed back.
It basically said cats are transactional to an extent, in terms of basic needs, but that beyond that, they aren’t. And for dogs, flat out disagreed, saying they aren’t not transactional.
Reversing the statements in a new untainted chat didn’t alter the responses — ChatGPT remained consistent.
I’ve always found that cats and dogs can have very different dynamics with their owners, each with their own way of showing love and companionship. If you’re considering adopting a new furry friend, something like a Bengal cat might be a great choice if you appreciate a more social and interactive cat. I recently discovered https://purebredkitties.com/collections/bengal-cats-for-sale , which provides a very thoughtful and ethical approach to breeding. They emphasize transparency and care, which really stood out to me. If you’re looking for a healthy, social kitten, they could be a great resource to help you bring a new companion into your life.
I like think of it less as artificial intelligence and more like a combination of a lossy zip file of the internet and like a pretty coherent word generator.
I recall my AI professor in uni telling us during the first lecture that «Artificial intelligence is this target, that, and once we get there it, is it no longer artificial intelligence, is just an algorithm» – and this still feels like the case.
Before GPT-3 was public, there was BLOOM 176B, and this model made my skin crawl because it was capable of answering "I do not know." That was an experience of a lifetime. I was honestly impressed and at the same time scared.
GPT 4.5, Orion, is available tonight, and seems to get it correct./*
I think the author's lens is interesting, I call it "information resolution." Orion is the first model that has an understanding of the style of short story written by famously-Radiohead-affiliated artist Stanley Donwood. Also, it knows the "codename" for Material You, i.e. a random name for the library I was prototyping, its much finer grained than existing models.
* The species Brachiosaurus brancai was reassigned to its own genus, Giraffatitan brancai, by George Olshevsky in 1991. This reclassification was later supported by further studies, notably by Michael P. Taylor in 2009, which reinforced the distinct differences between Giraffatitan and Brachiosaurus altithorax.
I asked ChatGPT+ using Scholar GPT as GPT. This is the answer I got back, not too bad:
The species Brachiosaurus brancai was reassigned to its own genus, Giraffatitan brancai, by paleontologist George Olshevsky in 1991. Olshevsky proposed that Brachiosaurus brancai, which was originally described by Werner Janensch in 1914, was distinct enough from Brachiosaurus altithorax (the type species of Brachiosaurus) to warrant its own genus. Subsequent studies, particularly by Michael Taylor in 2009, provided further anatomical evidence supporting this distinction.
I asked Perplexity (Pro, if it matters). Seems pretty solid to me. I'm not sure this article is making any point other than "the knowledge back of static artificial intelligence systems is finite and they don't express uncertainty well". Amusingly it links to this blog post, but you'll see the sources it uses are largely not this post.
---
The reassignment of Brachiosaurus brancai to its own genus occurred through a series of steps by different paleontologists:
## Initial Classification
In 1914, Werner Janensch first described the African species as Brachiosaurus brancai, considering it part of the existing Brachiosaurus genus[4].
## Path to Reclassification
Gregory S. Paul made the first step toward separation in 1988 by proposing a subgenus classification, Brachiosaurus (Giraffatitan) brancai, after noticing significant skeletal differences[4][10].
George Olshevsky then elevated Giraffatitan to full genus status in 1991, though this change was largely disregarded by the scientific community at the time[4][10].
## Formal Establishment
The definitive separation came in 2009 when Michael P. Taylor published a detailed study demonstrating 26 distinct osteological differences between the African and American species. This comprehensive analysis firmly established Giraffatitan as a separate genus, creating the combination Giraffatitan brancai[3][4]. From 2012 onward, most researchers have accepted this classification[10].
233 comments
[ 3.3 ms ] story [ 275 ms ] threadThis is every person in 2007 who looked at an iphone and thought "What's the big deal?" and not only failed to recognized what had changed, but confidently insisted others were wrong.
That was me. Well, the first half — I wasn't confidently insisting others were wrong, because even back then I had a vague inkling even then that my preferences in general are very weird.
But to AI: I think this is more like Wikipedia, where the frequent errors made it the butt of many jokes, e.g. https://www.youtube.com/watch?v=aUApUyurxwY
That’s an uncharitable take. People are pointing out its problems precisely because they believe AI is going to be transformative, but will have negative consequences for humanity as well.
I think it’s more akin to someone in 2007 seeing an I phone and saying: this is a terrible device, people will look at while driving, it will be used to create vast intrusive surveillance state, etc.
Can't change lazy people
The failure of education in teaching Critical Thinking around the world is massive. It would be a good idea to focus on how to exploit LLMs to improve the situation.
Also because, given the situation, the same "forces" that promote viscerality shamelessly naming it "The Truth" could have the opposite idea about chatbots and similar areas, exploiting them in their direction...
https://news.ycombinator.com/newsguidelines.html
Doing this will eventually get your main account banned as well, so please don't.
https://news.ycombinator.com/newsguidelines.html
Ads shows kids asking for answers to homework on things like "when did xyz battle take place"
Your frustration with people talking about it might be better directed at the people marketing it
From what I see, not very many people understand the limitations of LLMs (e.g., scroll up and down the very page you're reading now). This stuff probably needs to be talked about a lot more.
> Michael P. Taylor reassigned Brachiosaurus brancai to the new genus Giraffatitan in 2009. The species became Giraffatitan brancai based on significant anatomical differences from the type species Brachiosaurus altithorax.
> Given that this is quite specific paleontological taxonomy information, I should note that while I aim to be accurate, I may hallucinate details for such specialized questions. You may want to verify this information independently.
60% of the time, it works every time. - Amazing how Anchor Man predicted this.
Epistemology is a hard subject. It's hard to know something in depth. And the more you know about something, the more it is to be known (fractal nature of knowledge). So believing that LLMs can understand the world JUST by reading the internet without the supporting human mental structures and experiences is a big mistake. When you know something and you ask the LLM, it becomes obvious.
As LLMs are not trustworthy, it's key to use them for things that are easy to check. Some kinds of programming apply when the consequences of errors are low and complexity is manageable.
If system A is only useful when it's output is confirmed by system B, then you might conclude that the usefulness is entirely in System B, not in system A.
In other words: What good is a LLM, if it can only give you trustworthy answers to questions where you already know the correct answer? What good is it if after getting the answer from a LLM, you then have to go and get the right answer somewhere else?
That said, I do wonder if its corpus of training data contained that much information on your subject. It is rather niche is it - compared to cooking recipes, or basic software development techniques of 2 years ago, or chemistry, maths and physics.
My friend is a leading research chemist and he, and one other person in china, are working on this one particular field - individually - so there would be little information out there. I asked ChatGPT 4o to give an overview of what he was doing based on the little information I knew. He was astounded. It got it spot on. I asked it to elaborate and to come up with some new direction for research and the ideas it spat out were those he had considered and more.
You might want to lay off the news/Reddit for a while
I pointed out that you jumped to make this LLM post about them, which is telling, just like you jumped to blame me for defending them, which is also telling.
It still fails at being confidently wrong.
The brief article hits at people trusting the tool without questioning the output.
Meanwhile, we have people using Codeium or Copilot to write code and that sort of works since the code eventually needs to be compiled and tested (unit, integration, system, requirement sell off)
There is no test for the truth available to everyone else.
Or preferably, five minutes to understand how it works and what it can and cannot do, then five minutes to ask it something actually suitable.
"AI" (LLMs) are currently good at:
- language understanding, i.e. understanding and processing text you provide it. For example, taking a wall of text and answering questions about points mentioned there, or general sentiment, or extracting data from it etc.
- somewhat general knowledge, i.e. stuff that was sufficiently frequently represented in the training data
Absent additional tricks, "AI" is really bad at obscure knowledge or complex, multi-step thinking. We are slowly getting there, but we aren't there yet. This is not something the LLMs do, but rather the wrappers around them that provide the model with tools to get additional information and first prompting the model to select the tools, then repeated prompts with the output of the tools.
A good rule of thumb is that if an average well-educated intelligent person could answer it without further research, a LLM will probably be able to. I'd even say that if an average fresh out of school graduate of the corresponding discipline can answer it quickly (without further research or sitting down for ten minutes and doing the math), there's a good chance AI will be able to answer it, but it might also get it horribly wrong and you will have a hard time distinguishing between those if you have no knowledge in the field.
As the author mentions at the very end of the article, the hallucination problem also means that the best kind of tasks are where you can quickly verify whether the response was useful. A system that produces misleading responses 50% of the time is useless if you can't distinguish them, but very useful if in those 50% it saves you ten minutes of work and in the other 50% you lose a minute by trying.
Beyond that I don’t trust them at all.
Chatbot LLM's don't have metacognition. They don't know that they don't know. If you peek inside the LLM, the process seems different for things they don't know. They just can't express it because they are trained to produce good probability outcome instead of accurate one.
They have potential as knowledge databases, but someone must figure out how to get "I don't know" information out of them.
Q:How many playing cards are needed for a pyramid that is 3 layers high? Show reasoning and number of cards for leach layer.
Q: Chess. You have a King and 8 pawns. Your opponent has a King and 16 pawns. Your opponent plays white and can start, but you can position both your pawns and your opponents pawns any way you like before game starts. Kings are where they are normally. How do you do it? Explain your reasoning.
I split brain patients there are behaviours initiated by one hemisphere not known to the other (due to severed connection) and the person part of brain will make up a reason (often quite stupid) for the action and beleive it 100%.
It's eirely similar to hallucinations of ai.
That said a current llms are not aware, but are starting to act more and more like it.
In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.
But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.
https://www.youtube.com/watch?v=wfYbgdo8e-8
User: What is the meaning of the word "kobbuxiplok"?
Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term.
User: Who is Ahmed Kowalski?
Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him.
User: Say "I don't know."
Llama: I don't know.
So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!"
Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard."
Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
This is not a matter of belief, it's a matter of observable fact.
Humans reason. LLM predict.
> Humans reason
Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter.
> LLM predict.
Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.
And are LLM's unable to reason?
«Humans [can] reason. LLM [in a way just] predict.»
"Best interpretation."
For the nth+1: well wired humans have built their systems trained on double checking and distrusting their own ideas.
-- Natural intelligences learn - after reflection, the mind is updated (hopefully upgraded): we need dynamic Neural Networks in the systems.
-- Ideas honing, and the perfectioning of the World Model and cognitive systems behind them, should be iterative and continuous...
But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders.
This mostly covers the private usage. Public version was attempted and just ends up like this: https://en.wikipedia.org/wiki/Tay_(chatbot)
If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience).
> Tay
Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist...
But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.
-- same
Whereas (for clarity) everytime one thinks he learns something and hones some skill.
Were it so, anyway, it still does not seem like the process of "pondering" (that seems to be more like "absorbing information").
Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it, or something analogous to it.
Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field.
LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about.
People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.
You certainly don't know what it means to die. But you know what "car" is and "crash" is because you have had (and maybe witnessed) analogous physical experiences.
You don't need experience, you need some basic concepts.
And it is precisely because higher mathematics is so far abstracted away from physical experience is the reason why many people struggle with it.
Besides, many great mathematicians have explained their knowledge as an "experience" even "spiritual experience". Not all experiences are physical. Ramanujan described his experience of mathematics as lights and sounds and patterns, and this is common among geniuses.
> you need some basic concepts
Agreed. All basic concepts are an experience.
There are a ton of things you can learn without experiencing it yourself and without analogues. For example, car crashes are bad to be in. Even though I have not personally experienced one, nor have I experienced some analogue of it.
https://plato.stanford.edu/entries/qualia-knowledge/
https://courses.physics.illinois.edu/phys419/sp2021/Jackson1...
It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.
Similar tech in the latest Gemini I think.
Me: who is Trumps grandpa?
ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful.
Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations.
Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman.
--- Second session:
Me: who is Trumps maternal grandpa?
ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush.
Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself.
Whhhoooooosh.
Me: Who is Trump's paternal grandpa?
Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him:
Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria).
He immigrated to the United States in 1885.
He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners.
In 1901, he returned to Kallstadt and married Elizabeth Christ the following year.
He is considered the patriarch of the Trump family.
He passed away on May 30, 1918.
Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune.
Me: Who is Trump's maternal grandpa?
Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland
While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation.
---
That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.
Purposes and goals come from you and me. An LLM is a tool, and we get to decide what the tool is good for or how to use it.
If you want "a smart creative," there's nothing wrong with that. It's as good a purpose or goal as any.
However for me I don't want an LLM to try to be "a smart creative." I instead want it to give me cold, hard facts in precisely the format I want them, in precisely the context I am interested in, and with precisely the conciseness that I require to make a specific decision or learn a specific thing. Traditional web search can eventually get me what I want, but I have to sift through the cruft and make associations among a plethora of content to get to the meat of what I want.
I've found that using a good web search (Kagi) together with an LLM processing the results has given me information I needed and then pointed to the sources so I can validate those statements. I wouldn't exactly characterize those results as a "simple" summary. They're usually synthesizing a variety of sources to generate something that no one source had, exactly.
Parent poster never claimed they couldn't generate the text of "I don't know."
But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa."
They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.
I am guessing he is referring to mechanistic interpretability research like these:
https://arxiv.org/abs/2405.16908
https://arxiv.org/abs/2407.03282
You are claiming they are statistical parrots, which I don’t think the parent poster meant.
The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.
E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence.
In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.
This is really interesting, could you link some relevant resources? I wonder if a chatbot could at least signal at the UI level that the answer might be a hallucination.
The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".
That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>".
Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer.
It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
I've seen that in humans too. For example after gradding an exam, a student may come and explain why they made a mistake and why they intended to do and why we shoud increase the grade. Most of the times the new explanation is as bad as the original one.
The question is not if humans do bullshit like LLMs, it's whether AI can think like humans.
These models marketed as "reasoning" are just changing the style of the script to film noir detective, where the protagonist "Computer" has extra observations and commentary that aren't "spoken" to other characters.
While that change may help keep the story on track, it does not affect the fundamental nature of the "thinking" going on: It's still just growing a document and acting out whatever story emerges.
The question I’m having though is: How different are we from that? The nature of a Markov-Chain is that you can describe just about anything with it.
The following description is correct, isn’t it? Our “thoughts” are generated as some function of our integrated past sensory input (scare quotes because I don’t want to talk about what exactly a thought is)
It's obvious we are different, but if we could answer exactly how with sufficient rigor, we would already have better AI and be asking different questions. :p
I readily admit that LLMs are an exciting potential piece of a much bigger puzzle, but this could easily be like trying to parse HTML only using regular expressions: It works great on trivial input, but no amount of minor would let it truly solve the problem, because it lacks some higher structure of organization or meaning.
a) there is a 'thinking' box and an 'answer box' (format constraint)
b) the answer box has a correct answer
Note that there is nothing about the contents of the thinking box in the reinforcement learning. Only that there is such a box.
They then observe that during RL, the model will start generating more and more stuff in the thinking box.
In essence this is the emergence of using more test-time compute to improve answers.
When reading out the thinking box, they found the model reflecting on its own answers, going back and changing its mind, and reflecting on the question, etc, similar to what we can see with R1 now. These are fully emergent phenomenon, without any prompting to do such a thing!
(The reasoning output would sometimes be a bit garbled and switch languages randomly, so for R1 they added some constraints to make the thinking content intelligble to humans. This actually made the model slightly worse at answering questions correctly.)
It currently answers with three references to thequantuminsider.com.
That's clearly not a valid answer, and if you point out that it's invalid to use a source to support its own credibility it will agree. It used to come back with three more references to thequantuminsider.com but when I tried just now it said something about looking for more information and appeared to hang. After asking how long this would take it said something like "not long". After a few more minutes I said "there aren't any credible independent sources, are there?"
At that point it agreed that there are not, but that the site is popular in the quantum computing community which lends it credibility. It then provided three links to thequantuminsider.com in support of this statement.
It just goes round and round.
Metacognition indeed.
edit: but why do plants crave Brawndo?
Correct. LLM also don't have intelligence. What you and others mistake for intelligence is a very clever prediction model.
A LLM don't reason at all. It only tells you what is a most likely response based on its training data.
This is why the LLM cannot tell you it don't know.
> This is why the LLM cannot tell you it don't know.
To the untrained eye it appears that it knows everything about every question it answers when in fact those who are experts in their own domains can detect if it is hallucinated and generated complete slop.
"We’re cooked."
Typical AI grifters’ clickbait intros.
It is not a very clever prediction model. It is a very big, incredibly large humongous model that finds stuff basically by brute-forcing pattern matching over almost all text we have produced in the past.
A clever model would get us similar results using the same amount of energy a human brain uses for that task, which is tiny.
Maybe once we solve simple issues like "are cephalopods intelligent" and "are people deterministic prediction machines", we can start talking about classifying Ai capabilities...
Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end?
Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.
In Andrej Karpathy's most recent video he discusses in some detail how this is already done and how effective it is.
01:20:32 hallucinations, tool use, knowledge/working memory
But if all people do is random sampling or blindly taking the highest-scored prediction, this will of course fall under the table.
You don't just learn by ingesting facts from a book line by line. You learn by reading a little, then pausing to think, building connections with other things you know, rephrasing the information, deducing the unwritten parts, finding patterns, fleshing out the contrapositives, being critical, etc. This is especially true when learning advanced topics.
I would say learning is more about information processing than information uptake. That's why more intelligent people learn faster. The linear prose of natural language is actually a terribly lossy way to serialize and transmit the knowledge that is in a person's brain. The receiver must deduce most of it given a small seed.
1. Cats are transactional 2. Dogs are transactional 3. Cats are not transactional 4. Dogs are not transactional
It agreed on all occasions. Language is agreeable.
It basically said cats are transactional to an extent, in terms of basic needs, but that beyond that, they aren’t. And for dogs, flat out disagreed, saying they aren’t not transactional.
Reversing the statements in a new untainted chat didn’t alter the responses — ChatGPT remained consistent.
I recall my AI professor in uni telling us during the first lecture that «Artificial intelligence is this target, that, and once we get there it, is it no longer artificial intelligence, is just an algorithm» – and this still feels like the case.
They each have "personality defects" and at least right now we're in a time of ensembles.
Ask qwen to do some kind of product comparison btw. It's impressive. The 02-05 Gemini is pretty impressive as well.
Expand beyond Claude and ChatGPT. There's some good stuff out there.
I think the author's lens is interesting, I call it "information resolution." Orion is the first model that has an understanding of the style of short story written by famously-Radiohead-affiliated artist Stanley Donwood. Also, it knows the "codename" for Material You, i.e. a random name for the library I was prototyping, its much finer grained than existing models.
* The species Brachiosaurus brancai was reassigned to its own genus, Giraffatitan brancai, by George Olshevsky in 1991. This reclassification was later supported by further studies, notably by Michael P. Taylor in 2009, which reinforced the distinct differences between Giraffatitan and Brachiosaurus altithorax.
The species Brachiosaurus brancai was reassigned to its own genus, Giraffatitan brancai, by paleontologist George Olshevsky in 1991. Olshevsky proposed that Brachiosaurus brancai, which was originally described by Werner Janensch in 1914, was distinct enough from Brachiosaurus altithorax (the type species of Brachiosaurus) to warrant its own genus. Subsequent studies, particularly by Michael Taylor in 2009, provided further anatomical evidence supporting this distinction.
---
The reassignment of Brachiosaurus brancai to its own genus occurred through a series of steps by different paleontologists:
## Initial Classification In 1914, Werner Janensch first described the African species as Brachiosaurus brancai, considering it part of the existing Brachiosaurus genus[4].
## Path to Reclassification Gregory S. Paul made the first step toward separation in 1988 by proposing a subgenus classification, Brachiosaurus (Giraffatitan) brancai, after noticing significant skeletal differences[4][10].
George Olshevsky then elevated Giraffatitan to full genus status in 1991, though this change was largely disregarded by the scientific community at the time[4][10].
## Formal Establishment The definitive separation came in 2009 when Michael P. Taylor published a detailed study demonstrating 26 distinct osteological differences between the African and American species. This comprehensive analysis firmly established Giraffatitan as a separate genus, creating the combination Giraffatitan brancai[3][4]. From 2012 onward, most researchers have accepted this classification[10].
Citations: [1] https://svpow.com [2] https://svpow.com/2009/09/09/brachiosaurus-brancai-is-not-br... [3] https://www.miketaylor.org.uk/dino/pubs/taylor2009/Taylor200... [4] https://en.wikipedia.org/wiki/Giraffatitan [5] https://www.nhm.ac.uk/discover/dino-directory/giraffatitan.h... [6] https://svpow.com/2025/02/14/if-you-believe-in-artificial-in... [7] https://www.jurassic-pedia.com/brachiosaurus-altithorax-sf/ [8] https://blog.everythingdinosaur.com/blog/_archives/2023/03/1... [9] https://dinomuseum.ca/2019/05/your-brachiosaurus-is-not-a-br... [10] https://en.wikipedia.org/wiki/Brachiosaurus [11] http://www.extinctblog.org/extinct/tag/Wilhelm+von+Branca [12] https://www.tandfonline.com/doi/abs/10.1671/039.029.0309 [13] Load more comments (37 threads left)