Fun, though as hinted at the end, the point of LLM "truth" probes is to measure the model's internal judgment of truthfulness. There's no reason this judgment, even if measured with 100% accuracy, couldn't be mistaken or logically inconsistent.
I think this article pushes the premise farther than is reasonable.
The best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.
It looks like Zach Weinersmith predicted this exact line of research 11 years ago [1], when he suggested testing the liar sentence using fMRI.
The same analysis applies: the probe tells us what the LLM thinks about the truth value if the sentence, not the truth value of the sentence. I don't think anyone claimed that these probes were truth oracles.
Aren't there definitions of Truth that are not the negation of Falsehood? Can't there be a function True(x) that is not equal to !False(x)? Can't there be a third function Paradox(x) such that these counterexamples can be considered paradoxes and therefore outside of the truth?
I'm admittedly not a logician and don't formally study paradoxes, but I never quite understood the whole category of "this sentence is false" paradoxes, I feel that the sentence is paradox and therefore it's neither false nor true.
I do agree that it's a truth vector sounds like a silly panacea fantasy, though. But more logical formality is not the counter argument that would convince me of it, rather I believe that there's less formal and rigorous ways to get closer to truth.
> It might seem absurd to you to even suggest superhuman AIs could function as a truth-oracle (it certainly does to me), but there are two reasons to take it seriously. First, it is how these things will be used practically by the vast majority of people. They are already replacing standard Google search results, and I’ve had many discussions end with people delegating final authority on the truth to an AI.
There are already organizations taking advantage of this, actively producing "AI propaganda", meaning propaganda aimed at the LLMs themselves in order to influence their understanding of what is truthful and bend it towards powerful actors' agendas. They're not even hiding the fact that they're doing this.
Title is a bit clickbaitish, but the content is well worth reading - came in with my pitchfork ready and left agreeing with basically all of it, with questions like ‘what if the probe could return 3 dimensions: truthfulness, knowledge confidence and decidability?’
Also the observation that people treat LLMs like oracles when they’re everything but is spot on, something I’ve also been thinking about and it’s quite a bit scary.
As a programmer I was never impressed in such paradoxes. For me it was kinda obvious that in any sufficiently complex language you can create eqivalent of buggy infinite loop/recursion.
I think that one of the main problems with LLMs is that we're shoveling everything in to them, without any guidance as to what is "real"/"true"/"factual" - crazy anti-vaxxer-cooker stuff is is sitting in their along with science without any concept of scientific reality and no guidance to what is real and what is insane minds spinning on each other
H: All right, all right... and the limitation of human intelligence?
M: Mr Gödel is telling you that the theorem limits formal systems. So you see your
intelligence as a formal system, as a machine?
H (pompously): Indeed. I have the impression that everything I do ought to be done by a
machine, which could moreover speak just as well in my place.
G: From where I am, it is difficult to tell whether you exist or whether you are the virtual
creation of a GAT - a Generator of Automatic Truisms. Intelligence does not exist without
error, perhaps even without obstinacy in error; but who would take the risk of giving a
computer that kind of psychology? As for the incompleteness theorem, it certainly did not
foresee bad-tempered theories...
---
Gödel's Theorem, or an Evening with Mr Homais
Jean-Yves Girard
> Second, some people do really believe in a kind of platonic representation space that all models converge on, and that represents the “true” state of the world. If truth is indeed an objective part of the world, then you might expect such a universal truth direction to emerge as models get better. This
This really sounds like all those scifi stories with people trying to find god in the computer.
How could it be there, in the jumbled mirror of internet scraped texts.
> This resolved the most basic liar paradox, but not every diagonal attack, since not all functions on [0, 1] have fixed points. To make this work in general, we can for example allow only continuous functions on [0, 1] (which always have a fixed point by Brouwer’s fixed-point theorem). But that restriction comes at the cost of expressivity: "This sentence has truth score less than 0.5" is not a continuous function of the truth score of the sentence.
Ok, I get it. Either expressiveness or completeness, but the question arises: did mathematicians explore systems with limits on expressiveness? In a field of computer programming there is Rust with limited expressiveness that doesn't solve all the problems, but still makes things much simpler. How about a mathematics with limited expressiveness and some unsafe blocks here and there?
14 comments
[ 0.22 ms ] story [ 7.1 ms ] threadThe best anyone expects from an LLM "truth vector" is that it would encode the model's belief about whether the statement is true. Of course a perfect truth oracle is impossible.
The same analysis applies: the probe tells us what the LLM thinks about the truth value if the sentence, not the truth value of the sentence. I don't think anyone claimed that these probes were truth oracles.
[1] https://smbc-comics.com/index.php?id=3657
I'm admittedly not a logician and don't formally study paradoxes, but I never quite understood the whole category of "this sentence is false" paradoxes, I feel that the sentence is paradox and therefore it's neither false nor true.
I do agree that it's a truth vector sounds like a silly panacea fantasy, though. But more logical formality is not the counter argument that would convince me of it, rather I believe that there's less formal and rigorous ways to get closer to truth.
There are already organizations taking advantage of this, actively producing "AI propaganda", meaning propaganda aimed at the LLMs themselves in order to influence their understanding of what is truthful and bend it towards powerful actors' agendas. They're not even hiding the fact that they're doing this.
Also the observation that people treat LLMs like oracles when they’re everything but is spot on, something I’ve also been thinking about and it’s quite a bit scary.
As a programmer I was never impressed in such paradoxes. For me it was kinda obvious that in any sufficiently complex language you can create eqivalent of buggy infinite loop/recursion.
M: Mr Gödel is telling you that the theorem limits formal systems. So you see your intelligence as a formal system, as a machine?
H (pompously): Indeed. I have the impression that everything I do ought to be done by a machine, which could moreover speak just as well in my place.
G: From where I am, it is difficult to tell whether you exist or whether you are the virtual creation of a GAT - a Generator of Automatic Truisms. Intelligence does not exist without error, perhaps even without obstinacy in error; but who would take the risk of giving a computer that kind of psychology? As for the incompleteness theorem, it certainly did not foresee bad-tempered theories...
---
Gödel's Theorem, or an Evening with Mr Homais Jean-Yves Girard
Translation: https://files.catbox.moe/kac0wu.pdf
Original: https://perso.ens-lyon.fr/pierre.lescanne/ENSEIGNEMENT/LOGIQ...
This really sounds like all those scifi stories with people trying to find god in the computer. How could it be there, in the jumbled mirror of internet scraped texts.
Ok, I get it. Either expressiveness or completeness, but the question arises: did mathematicians explore systems with limits on expressiveness? In a field of computer programming there is Rust with limited expressiveness that doesn't solve all the problems, but still makes things much simpler. How about a mathematics with limited expressiveness and some unsafe blocks here and there?