Hallucination Is Inevitable: An Innate Limitation of Large Language Models (2025) (arxiv.org)

IANAL, nor expert in this space.

But might any such care to comment on the consequences, if this "it is impossible, even in theory, to eliminate LLM hallucinations" result holds up?

From that abstract it doesn't sound like they allowed for the possibility that the LLM could be trained to say "I don't know" for some things.

Describing it as a limitation is the problem. Hallucination is the core feature. It's the only thing they do!

>Submitted on 22 Jan 2024 (v1), last revised 13 Feb 2025 (this version, v2)

LLM or transformers just merely extracting signals from human text and build a "contextualized" predictor over a long sequence of words sorted by the information (technically it is attention) of each token, then generate sentences that way, one by one into other sequences at a time.

But the biggest problem is, even human itself is subjectable to hallucination. That is called being delusional, or being drugged. So it is inevitable from the first principle.

They prove that no finite amount of training data is enough to extrapolate an adversarially constructed non-continuous function. It's something akin to the no free lunch theorem (NFL).

No one uses the NFL to "prove" that LLMs can't learn to be the best optimizers, because it also proves that people can't be the best optimizers, but we manage somehow, so the theorem is irrelevant.

This is a fallacy of proving too much.