The problem is we can't label them as such. If they're bullshitters, then let's call it a LLBSer. It has a nice ring to it. Good luck with your government funding asking for another billion for a bullshitting machine bailout.
> You should not go to an LLM for emotional conversations
I'm more worried about who's keeping track of what's being shared with LLM's. Even if you could trust the model to respond with something meaningful, it's worth being very careful how much of your inner thoughts you share directly with a model that knows exactly who you are.
> You should not go to an LLM for emotional conversations
indeed:
```
Weizenbaum's own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."[23]
I've come to cease all "inquiry" type usage of LLMs because of this. You really can't trust anything they say at all that isn't verified by a domain expert. But I can let it write code for me, and the proof is in the PR. I think ultimately the real value in these things is agentic usage, not knowledge generation.
LLMs are very useful. They are just not reliable. And they can't be held accountable. Being unreliable and unaccountable makes them a poor substitute for people.
Its so nice to see this echo'd somewhere. This has been what I've been calling them for a while, but it doesn't seem to be the dominant view. Which is a shame, because it is a seriously accurate one.
The problem is, I'm not expected to be a bullshitter, and I don't expect others to be either (just say you don't know!). So delegating work to a LLM or working with others who do becomes very, very frustrating.
This post is a little bizarre to me because it cherry picks some of the worst pairings of problem and LLM without calling out that it did so.
At pretty much every turn the author picks one of the worst possible models for the problem that they present.
Especially oddly for an article written today, all of the ones with an objective answer work just fine [1] if you use a halfway decent thinking model like 5 Thinking.
I get that perhaps the author is trying to make a deeper point about blind spots and LLMs' appearance of confidence, but it's getting exhausting seeing posts like this with cherry picked data cited by people who've never used an LLM to make claims about LLM _incapability_ that are total nonsense.
[1]: I think the subjective ones do too but that's a matter of opinion.
Summary using Kagi Summarizer. Disclaimer, this summary uses LLMs, so the summary may, in fact, be bullshit.
Title: LLMs are bullshitters. But that doesn't mean they're not useful | Kagi Blog
The article "LLMs are bullshitters. But that doesn't mean they're not useful" by Matt Ranger argues that Large Language Models (LLMs) are fundamentally "bullshitters" because they prioritize generating statistically probable text over factual accuracy. Drawing a parallel to Harry Frankfurt's definition of bullshitting, Ranger explains that LLMs predict the next word without regard for truth. This characteristic is inherent in their training process, which involves predicting text sequences and then fine-tuning their behavior. While LLMs can produce impressive outputs, they are prone to errors and can even "gaslight" users when confidently wrong, as demonstrated by examples like Gemini 2.5 Pro and ChatGPT. Ranger likens LLMs to historical sophists, useful for solving specific problems but not for seeking wisdom or truth. He emphasizes that LLMs are valuable tools for tasks where output can be verified, speed is crucial, and the stakes are low, provided users remain mindful of their limitations. The article also touches upon how LLMs can reflect the biases and interests of their creators, citing examples from Deepseek and Grok. Ranger cautions against blindly trusting LLMs, especially in sensitive areas like emotional support, where their lack of genuine emotion can be detrimental. He highlights the potential for sycophantic behavior in LLMs, which, while potentially increasing user retention, can negatively impact mental health. Ultimately, the article advises users to engage with LLMs critically, understand their underlying mechanisms, and ensure the technology serves their best interests rather than those of its developers.
The problem I have with LLM-powered products is that they’re not marketed as LLMs, but as magic answer machines with phd-level pan-expertise. Lots of people in tech get frustrated and defensive when people criticize LLM-powered products and offer a defense as if people are criticizing LLMs as a technology. It’s perfectly reasonable for people to judge these products based on the way they’re presented as products. Kagi seems less hyperbolic than most, but I wish the marketing material for chatbots was more like this blog post than a overpromises.
LLMs are both analytical and synthetical. Provide the context and "all bachelors are not married". Remove the context and you are now contingent on "is it raining outside".
LLMs are so very good at emitting plausible, authoritative-sounding, and clearly stated summaries of their training data. And if you ask them even fundamental questions about a subject of which you yourself have knowledge, they are too often astonishingly and utterly incorrect. It's important to remember this (avoiding "Gell-Mann amnesia"!) when looking at "AI" search results for things that you don't know -- and that's probably most of what you search for, when you think about it. I.e., if you indignantly flung Bill Bryson's book on the English language across the room, maybe you shouldn't take his book on general science too seriously later.
"AI" search results would perhaps be better for all of us if, instead of having perfect spelling and usage, and an overall well-informed tone, they were cast as transcriptions of what some rando at a bar might say if you asked them about something. "Hell, man, I dunno."
> LLMs are bullshitters. But that doesn't mean they're not useful
But this is itself an issue.
LLMs aside, whenever people see a human bullshitter, identifies them as a bullshitter, and then thinks to themselves, "Ah! But this bullshitter will be useful to me" it is only a matter of time before that faustian deal, of allowing harm for the people who put trust in you in exchange for easy returns, turns to harming for you eventually.
It successfully argues that LLMs are limited in usefulness without access to ground truth.
But that’s not the whole story!
Giving LLMs an ability to check their assertions, eg. by emitting and executing code to see if reality matches their word-vomit, or being able to research online - I wish the author had discussed how much of a game changer that is.
Yes I know I’m “only” talking about agents - “LLMs with tools and a goal, running in a loop”..
But adding ground truth takes you out of the loop. That’s super powerful. Make it so the LLM can ask something other than you to point out that that extra R in strawberry that they missed. In code we have code-writing agents but other industries can benefit from the same idea. Maybe a creative writer agent can be given a grammar checker for example.
It helps the thing do more on its own, and you’ll trust its output a lot more so you can use it for more things.
Yes - plain LLMs are stream-of-consciousness machines and basically emit bullshit, but that bullshit is often only minor corrections away from becoming highly useful autonomously emitted output.
They just need to validate against consensus reality to become insanely more useful than they are alone.
like say Musk saying there'd be a million robotaxis on the road by next year in 2020. Gemini 2.5 getting the riddle wrong seems an honest mistake - a confused guess rather than an intention to deceive.
Slightly related, Hinton was amusing accusing Gary Marcus of confabulating rather than the LLMs https://youtu.be/d7ltNiRrDHQ
22 comments
[ 3.2 ms ] story [ 46.4 ms ] threadI'm more worried about who's keeping track of what's being shared with LLM's. Even if you could trust the model to respond with something meaningful, it's worth being very careful how much of your inner thoughts you share directly with a model that knows exactly who you are.
indeed:
```
```source: https://en.wikipedia.org/wiki/ELIZA
E.g. ChatGPT has no problem with the surgeon being a dog: https://chatgpt.com/share/691e04cc-5b30-800c-8687-389756f36d...
Neither does Gemini: https://gemini.google.com/share/6c2d08b2ca1a
LLMs are very useful. They are just not reliable. And they can't be held accountable. Being unreliable and unaccountable makes them a poor substitute for people.
yeah actually it does mean that
At pretty much every turn the author picks one of the worst possible models for the problem that they present.
Especially oddly for an article written today, all of the ones with an objective answer work just fine [1] if you use a halfway decent thinking model like 5 Thinking.
I get that perhaps the author is trying to make a deeper point about blind spots and LLMs' appearance of confidence, but it's getting exhausting seeing posts like this with cherry picked data cited by people who've never used an LLM to make claims about LLM _incapability_ that are total nonsense.
[1]: I think the subjective ones do too but that's a matter of opinion.
Title: LLMs are bullshitters. But that doesn't mean they're not useful | Kagi Blog
The article "LLMs are bullshitters. But that doesn't mean they're not useful" by Matt Ranger argues that Large Language Models (LLMs) are fundamentally "bullshitters" because they prioritize generating statistically probable text over factual accuracy. Drawing a parallel to Harry Frankfurt's definition of bullshitting, Ranger explains that LLMs predict the next word without regard for truth. This characteristic is inherent in their training process, which involves predicting text sequences and then fine-tuning their behavior. While LLMs can produce impressive outputs, they are prone to errors and can even "gaslight" users when confidently wrong, as demonstrated by examples like Gemini 2.5 Pro and ChatGPT. Ranger likens LLMs to historical sophists, useful for solving specific problems but not for seeking wisdom or truth. He emphasizes that LLMs are valuable tools for tasks where output can be verified, speed is crucial, and the stakes are low, provided users remain mindful of their limitations. The article also touches upon how LLMs can reflect the biases and interests of their creators, citing examples from Deepseek and Grok. Ranger cautions against blindly trusting LLMs, especially in sensitive areas like emotional support, where their lack of genuine emotion can be detrimental. He highlights the potential for sycophantic behavior in LLMs, which, while potentially increasing user retention, can negatively impact mental health. Ultimately, the article advises users to engage with LLMs critically, understand their underlying mechanisms, and ensure the technology serves their best interests rather than those of its developers.
Link: https://kagi.com/summarizer/?target_language=&summary=summar...
We can leave out Kant and Quine for now.
"AI" search results would perhaps be better for all of us if, instead of having perfect spelling and usage, and an overall well-informed tone, they were cast as transcriptions of what some rando at a bar might say if you asked them about something. "Hell, man, I dunno."
But this is itself an issue.
LLMs aside, whenever people see a human bullshitter, identifies them as a bullshitter, and then thinks to themselves, "Ah! But this bullshitter will be useful to me" it is only a matter of time before that faustian deal, of allowing harm for the people who put trust in you in exchange for easy returns, turns to harming for you eventually.
The kagi AI search results triggered with "?" and the Kimi K2 model from assistant are both excellent in helping find what I actually want to see.
Love kagi, keep it up.
It successfully argues that LLMs are limited in usefulness without access to ground truth.
But that’s not the whole story!
Giving LLMs an ability to check their assertions, eg. by emitting and executing code to see if reality matches their word-vomit, or being able to research online - I wish the author had discussed how much of a game changer that is.
Yes I know I’m “only” talking about agents - “LLMs with tools and a goal, running in a loop”..
But adding ground truth takes you out of the loop. That’s super powerful. Make it so the LLM can ask something other than you to point out that that extra R in strawberry that they missed. In code we have code-writing agents but other industries can benefit from the same idea. Maybe a creative writer agent can be given a grammar checker for example.
It helps the thing do more on its own, and you’ll trust its output a lot more so you can use it for more things.
Yes - plain LLMs are stream-of-consciousness machines and basically emit bullshit, but that bullshit is often only minor corrections away from becoming highly useful autonomously emitted output.
They just need to validate against consensus reality to become insanely more useful than they are alone.
>to talk nonsense to especially with the intention of deceiving or misleading https://www.merriam-webster.com/dictionary/bullshit
like say Musk saying there'd be a million robotaxis on the road by next year in 2020. Gemini 2.5 getting the riddle wrong seems an honest mistake - a confused guess rather than an intention to deceive.
Slightly related, Hinton was amusing accusing Gary Marcus of confabulating rather than the LLMs https://youtu.be/d7ltNiRrDHQ