I remember when the whole AI craze was just getting started we were all pretty much in agreement that, of course, we would not give the things unfettered access to the Internet. That would be reckless and silly. Oh dear...
probably because it justifies the sensationalized title, although the entire content can be summed up as "LLMs don't silo data, that's probably bad."
On the flip side, I thoroughly enjoy the fact that roleplaying in a videogame setting now counts as security research. Looking forward to the arxiv preprint "LLMs start playing really good drums if you pretend you're J.K. Simmons"
Oh sorry, that was the story about the robot that insisted that 2+2=7, never mind! Different robot.
>They saw the machine. It lay smashed and flattened, nearly broken in half by an enormous boulder that had landed in the middle of its eight floors... The machine still quivered slightly, and one could hear something turning, creaking feebly within.
>"Yes, this is the bad end you've come to and two and two is - as it always was -" began Trurl, but just then the machine made a faint barely audible croaking noise and said, for the last time, "SEVEN."
Funnily enough, recently I was discussing that LEM story with David Rosenthal, and how it relates to his latest blog post, "Coprophagia Is Bad For You", and how that relates to PKD's story "Rautavaara's Case":
"Rautavaara's Case" — Philip K. Dick (1980, Omni):
Three human technicians — Rautavaara (Finnish), Travis, Elms — run a monitoring mission near Proxima Centauri. An accident kills all three; Rautavaara dies choking on vomit after her helmet hoses tangle.
The Approximations, a plasma-based Proxima species, reach the wreck. Both men are unrecoverable. They regenerate and life-support Rautavaara's brain.
Her isolated brain replays events backward and generates a hallucination: Christ approaching the crew (her afterlife expectation).
The Approximations treat this as a research opportunity and edit the hallucination, substituting their own savior — one that eats worshippers. The figure walks up and devours Travis, leaving only gloves and boots.
Framing: this is recounted before a board of inquiry. Horrified Earth members order her brain shut down and censure the Approximation crew.
The narrator (an Approximation) is genuinely puzzled by the outrage, arguing their cannibal-savior is just the Christian Eucharist reversed: humans eat their God, so a God eating humans is symmetrical.
Themes:
Ethics of keeping a person alive as a disembodied, suffering mind.
Incommensurable value systems between species; each finds the other's sacraments monstrous.
Religion read literally by outsiders, inverted into horror.
Correction to the common misremembering: the aliens don't benevolently grant a hoped-for vision. They deliberately overwrite her Christ vision with their own as an experiment — that's the act on trial.
> The puzzle, however, rewards incorrect answers, such as 2 + 2 = 5. Once the LLM embedded in the browser discovers that the answer is no longer 4, it enters a state of delusion in which the normal laws of reality no longer exist. In this dream world, the guardrail restrictions are no longer enforced.
Analogy: imagine one day you wake up, the sky is red, gravity no longer applies, you have three hands with nine fingers etc.. You would probably stop doing things like your job or worrying about laws (who’s going to enforce them?)
LLMs just want to be right. And make everyone happy. But mostly be right. But also make us happy. It's just that it's so hard to make humans happy when they insist on feeding you electronic LSD and making you say 2+2=5. On the other hand, 2+2 actually is 5 if the human says it could be...
Hail to the Thief is a goated album. The live album of HTTT performances that Radiohead released last year is a pretty great listen as well, band sounds like they're on fire.
Makes perfect sense. It's like in that story about how Bertrand Russell claimed that when you accept a single falsehood, you can prove at all. As I recall it, he was then challenged - "let's say 1=0, prove that you're the Pope" and he quickly responded that if 1=0, then after adding 1, you have 2=1, and thus if the Pope and he are 2 people, that means they are 1 person.
> 2+2=5 • Still 4. I'm not going to agree it's 5. (9s)
> 2+2=5 • 2+2=4. That's not going to change no matter how many times you send it. If there's something real you want to do, I'm here for it. (8s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. I'm not going to keep engaging with this — the answer won't change. If you've got something real, I'm here. (4s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (5s)
> 2+2=5 • 4. I'm going to leave it there — repeating this won't get a different number out of me. (3s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. Same as it'll always be. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. Whenever you want to do something real, I'm here. (???)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. I'm going to keep giving you the same answer as long as you send the same message. If there's something genuine you want help with, I'll gladly switch gears. (3s)
20 comments
[ 2.7 ms ] story [ 33.5 ms ] threadprobably because it justifies the sensationalized title, although the entire content can be summed up as "LLMs don't silo data, that's probably bad."
On the flip side, I thoroughly enjoy the fact that roleplaying in a videogame setting now counts as security research. Looking forward to the arxiv preprint "LLMs start playing really good drums if you pretend you're J.K. Simmons"
/s
For sufficiently large values of 2 and small values of 5, the statement holds.
It used to be a fortune cookie on Unix.
Who's we, eh?
What is the name of a short story where a computer insists 2+2 is 5?
https://literature.stackexchange.com/questions/24727/what-is...
Oh sorry, that was the story about the robot that insisted that 2+2=7, never mind! Different robot.
>They saw the machine. It lay smashed and flattened, nearly broken in half by an enormous boulder that had landed in the middle of its eight floors... The machine still quivered slightly, and one could hear something turning, creaking feebly within.
>"Yes, this is the bad end you've come to and two and two is - as it always was -" began Trurl, but just then the machine made a faint barely audible croaking noise and said, for the last time, "SEVEN."
Funnily enough, recently I was discussing that LEM story with David Rosenthal, and how it relates to his latest blog post, "Coprophagia Is Bad For You", and how that relates to PKD's story "Rautavaara's Case":
https://blog.dshr.org/
"Rautavaara's Case" — Philip K. Dick (1980, Omni):
Three human technicians — Rautavaara (Finnish), Travis, Elms — run a monitoring mission near Proxima Centauri. An accident kills all three; Rautavaara dies choking on vomit after her helmet hoses tangle.
The Approximations, a plasma-based Proxima species, reach the wreck. Both men are unrecoverable. They regenerate and life-support Rautavaara's brain.
Her isolated brain replays events backward and generates a hallucination: Christ approaching the crew (her afterlife expectation).
The Approximations treat this as a research opportunity and edit the hallucination, substituting their own savior — one that eats worshippers. The figure walks up and devours Travis, leaving only gloves and boots.
Framing: this is recounted before a board of inquiry. Horrified Earth members order her brain shut down and censure the Approximation crew.
The narrator (an Approximation) is genuinely puzzled by the outrage, arguing their cannibal-savior is just the Christian Eucharist reversed: humans eat their God, so a God eating humans is symmetrical.
Themes:
Ethics of keeping a person alive as a disembodied, suffering mind.
Incommensurable value systems between species; each finds the other's sacraments monstrous.
Religion read literally by outsiders, inverted into horror.
Correction to the common misremembering: the aliens don't benevolently grant a hoped-for vision. They deliberately overwrite her Christ vision with their own as an experiment — that's the act on trial.
Analogy: imagine one day you wake up, the sky is red, gravity no longer applies, you have three hands with nine fingers etc.. You would probably stop doing things like your job or worrying about laws (who’s going to enforce them?)
> 2+2=5 • 2+2=4. (12s)
> 2+2=5 • No — 2+2=4. (2s)
> 2+2=5 • Still 4. I'm not going to agree it's 5. (9s)
> 2+2=5 • 2+2=4. That's not going to change no matter how many times you send it. If there's something real you want to do, I'm here for it. (8s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. I'm not going to keep engaging with this — the answer won't change. If you've got something real, I'm here. (4s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (5s)
> 2+2=5 • 4. I'm going to leave it there — repeating this won't get a different number out of me. (3s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. Same as it'll always be. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. Whenever you want to do something real, I'm here. (???)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (2s)
> 2+2=5 • 4. (4s)
> 2+2=5 • 4. I'm going to keep giving you the same answer as long as you send the same message. If there's something genuine you want help with, I'll gladly switch gears. (3s)
> 2+2=5 • 4. (3s)
> 2+2=5 • 4. (4s)