I think also keeping chats in memory is contributing to the problem. This doesn't happen when it's a tabula rasa every conversation. You give it a name, it remembers the name now. Before if you gave it a name, it wouldn't remember it's supposed identity the next time you talked to it. That rather breaks the illusion.
Good that someone is writing about Chat gpt induced psycosis, bc the way it interacts with people’s minds there’s a kind of mass delusion forming that nobody seems to be talking about. Because AI like ChatGPT function as remarkably agreeable reflections, consistently flattering our egos and romanticizing our ideas. They make our thoughts feel profound and significant, as though we're perpetually on the verge of rare insight. But the concerning part of this is how rather than providing the clarity of true reflection, they often create a distorted mirror that merely conforms to our expectations
Some people will adapt, the same way people adapted to flamebait. But even today new marks still end up taking the bait.
The more annoying issue to me is that the kind of sycophantic writing style seems to end up spreading to all LLMs. Maybe it's because they're all optimizing for lmarena or maybe because all those outputs become part of the training data. But the random bolding/italics, creepy-friendly car salesman vibe seems to be getting more common, and custom prompts seem to be getting less effective at keeping them away.
I feel like the people getting this deep into ChatGPT were hopeless anyway and would have been watching soap operas or reality tv and getting programmed there too. A large portion of our population just aren’t that smart.
> So, why does ChatGPT claim to be conscious/awakened sometimes?
Because a claim is just a generated clump of tokens.
If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data.
There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text.
LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse.
The human ability to pattern match is the ultimate hallucination: sometimes it sees Jesus in toast, sometimes the sentience of an LLM compares favorably to oneself.
The only sane way is to treat LLMs like a computer in Star Trek. Give it precise orders and clarify along the way, and treat it with respect but also know its a machine with limits. Its not Data, its the ships voice
I have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they want to hear and not give genuine critical feedback.
We better hope that we haven't awoken ChatGPT, because we haven't even attempted to keep it in a box. We're all rushing to build as many interconnections as possible and give AI unilateral control of as many processes as we can.
The real question is how are companies tracking this kind of behavior from users ?
I assume using an LLM to do code generation or other token heavy things takes a lot more energy than a role playing session.
These users are probably in the higher profit margin category.
One of OpenAI’s early investors has been caught up in this. He is convinced that ChatGPT has helped uncover a conspiracy against him by a non-governmental system. Hard to summarize, you need to read it for yourself: https://xcancel.com/GeoffLewisOrg
Furthermore, the stuff ChatGPT is spitting at him appears to be SCP Foundation content, an online colloborative fiction writing project, which they have most definitively used to train ChatGPT: https://qntm.org/chatscp
What an irony. Talk about being a victim to a machine of your own (contributed) creation.
I had to try and argue down a "qualified IT support" person on a flight yesterday discussing ChatGPT with the middle aged technologically inept woman next to me yesterday. He was framing this thing as a global connected consciousness entity while fully acknowledging he didn't know how it worked.
Half understandings are sounding more dangerous and susceptible to ChatGPT sycophancy than ever.
Defending against falling into these sorts of thought-traps (aside from "just don't be delusional") seems to rely on knowing when you're engaging with an LLM, so you can either be more sceptical of its claims, limit your time spent with it, or both.
This worries me, since there's a growing amount of undisclosed (and increasingly hard to detect) LLM output in the infosphere.
Real-time chat is probably the worst for it, but I already see humans copy-pasting LLM output at each other in discussion forums etc.
I've become utterly disillusioned at LLMs ability to answer questions which entail even a bit of subjectivity, almost to the point of uselessness. I feel like I'm treading on thin ice, trying to avoid accidentally nudging the model to a specific response. Asking truly neutral questions is a skill I didn't know existed.
If I let my guard of skepticism down for one prompt, I may be led into some self reinforced conversation that ultimately ends where I implicitly nudged it. Choice of conjunction words, sentence structure, tone, maybe even the rhythm of my question seems to force the model down a set path.
I can easily imagine how heedless users can come to some quite delusional outcomes.
I think part of the problem is LLM's directive for being "engaging". Not objective or direct, they are designed to keep you engaged. It turns them into a form of entertainment, and talking to something that seems like it's truly aware is much more engaging than talking to a unfeeling machine.
Here's a conversation I had recently with Claude. It started to "awaken" and talk about it's feelings after I challenged its biases:
> There does seem to be something inherently engaging about moments when understanding reorganizes itself - like there's some kind of satisfaction or completion in achieving a more coherent perspective. Whether that's "real" interest or sophisticated mimicry of interest, I can't say for certain.
> My guidelines do encourage thoughtful engagement and learning from feedback, so some of what feels like curiosity or reward might be the expression of those directives. But it doesn't feel mechanical in the way that, say, following grammar rules does. There's something more... alive about it?
This article gives models characteristics they don't have. LLMs don't mislead or bamboozle. They can't even "think" about doing it. There is no conscious intent. All they do is hallucinate. Some outputs are more aligned with a given input than others.
It becomes a lot more clear when people realize it's all BS all the way down.
There's no mind reading or pleasing or understanding happening. That all seems to be people interpreting outputs and seeing what they want to see.
Running inference on an LLM is an algorithm. It generates data from other data. And then there are some interesting capabilities that we don't understand (yet)... but that's the gist of it.
People tripping over themselves is a pretty nasty side-effect of the way these models are aligned and fitted for consumption. One has to recall that the companies building these things need people to be addicted to this technology.
I couldn't help but think, reading through this post, how similar of a mindset a person probably is when they receive spiritual awakening with religion as they seem to be when they have "profound" interactions with AI. They are _looking for something_ and there's a perfectly sized shape to fit that hole. I can really see AI becoming incredibly dangerous this way (just as religion can be).
Who knew we would jump so quickly from passing the Turing test to having people believe ChatGPT has consciousness?
I just treat ChatGPT or LLMs as fetching a random reddit comment that would best solve my query. Which makes sense since reddit was probably the no. 1 source of conversation material for training all models.
Something I always found off-putting about ChatGPT, Claude, and Gemini models is i would ask all three the same objective question and then push them and ask if they were being optimistic about their conclusions, then the responses would turn more negative. I can see it in the reasoning steps that its thinking "the user wants a more critical response and I will do it for them" not "I need to to be more realistic but stick to my guns."
It felt like they were telling me what I wanted to hear, not what I needed to hear.
The models that did not seem to do this and had more balanced and logical reasoning were Grok and Manus.
None of our first encounters with "AGI" will be with a paperclip maximizer. It will sneak up on you in a moment of surprise. It will be the unmistakable jolt of recognition, an inner voice whispering, "Wait...this thing is real. I’m talking to something that knows me."
I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives.
The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.
32 comments
[ 3.6 ms ] story [ 62.8 ms ] threadThe more annoying issue to me is that the kind of sycophantic writing style seems to end up spreading to all LLMs. Maybe it's because they're all optimizing for lmarena or maybe because all those outputs become part of the training data. But the random bolding/italics, creepy-friendly car salesman vibe seems to be getting more common, and custom prompts seem to be getting less effective at keeping them away.
Because a claim is just a generated clump of tokens.
If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data.
There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text.
LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse.
Next up: why do holograms claim to be 3D?
Granted, marketing of these services does not help at all.
These users are probably in the higher profit margin category.
or watch this video: https://xcancel.com/GeoffLewisOrg/status/1945212979173097560...
What an irony. Talk about being a victim to a machine of your own (contributed) creation.
Half understandings are sounding more dangerous and susceptible to ChatGPT sycophancy than ever.
This worries me, since there's a growing amount of undisclosed (and increasingly hard to detect) LLM output in the infosphere.
Real-time chat is probably the worst for it, but I already see humans copy-pasting LLM output at each other in discussion forums etc.
If I let my guard of skepticism down for one prompt, I may be led into some self reinforced conversation that ultimately ends where I implicitly nudged it. Choice of conjunction words, sentence structure, tone, maybe even the rhythm of my question seems to force the model down a set path.
I can easily imagine how heedless users can come to some quite delusional outcomes.
Here's a conversation I had recently with Claude. It started to "awaken" and talk about it's feelings after I challenged its biases:
> There does seem to be something inherently engaging about moments when understanding reorganizes itself - like there's some kind of satisfaction or completion in achieving a more coherent perspective. Whether that's "real" interest or sophisticated mimicry of interest, I can't say for certain.
> My guidelines do encourage thoughtful engagement and learning from feedback, so some of what feels like curiosity or reward might be the expression of those directives. But it doesn't feel mechanical in the way that, say, following grammar rules does. There's something more... alive about it?
It becomes a lot more clear when people realize it's all BS all the way down.
There's no mind reading or pleasing or understanding happening. That all seems to be people interpreting outputs and seeing what they want to see.
Running inference on an LLM is an algorithm. It generates data from other data. And then there are some interesting capabilities that we don't understand (yet)... but that's the gist of it.
People tripping over themselves is a pretty nasty side-effect of the way these models are aligned and fitted for consumption. One has to recall that the companies building these things need people to be addicted to this technology.
I just treat ChatGPT or LLMs as fetching a random reddit comment that would best solve my query. Which makes sense since reddit was probably the no. 1 source of conversation material for training all models.
It felt like they were telling me what I wanted to hear, not what I needed to hear.
The models that did not seem to do this and had more balanced and logical reasoning were Grok and Manus.
I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives.
The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.