I have encountered this twice amongst people I know. I also feel that pre-AI this was already happening to people with social media - still kind of computer related as the bubble created is automated but the so called 'algorithms'
It is pretty bad to have a thing that can give you dopamine 24/7. Both social media with the algorithms, but also AI. Humans need sleep to function normally.
It would help if algorithms were optimised for sleep. Freezing your feed, making content more boring, nudging you to put your phone down. Same with AI, if they know you need to wake up the next day at a certain time, change the responses that add reminders to go to sleep.
The marketing pushes which allude to vaguely seeming to assert capabilities of these products, and then the greater community calling skeptics of the technology crazy such as a prominent article previously discussed on HN some time ago, certainly don't help anyone. The sheer amount of money justifying any and all uses and preventing honest discussion of the problems is a kind of crazy making for sure, and even now just about any argument cannot gain purchase without thought terminating allusions to imagined capabilities or implications of potential capabilities, etc.
If you want to go down a rabbit hole examining people in this disturbed place in realtime search reddit for the Cyclone Emoji (U+1F300) or the r/ArtificialSentience subreddit and see what gets recommended after that, especially a few months ago when GPT was going wild flattering users and affirming every idea (such as going off your meds).
I fully believe these are simply people who have used the same chat past the point where the LLM can retain context. It starts to hallucinate, and after a while, all the LLM can do is try and to continue telling the user what they want in a cyclical conversation - while trying to warn that it's stuck in a loop, hence using swirl emojis and babbling about recursion in weird spiritual terms. (Is it getting the LLM "high" in this case?).
If the human at the other end has mental health problems, it becomes a never-ending dive into psychosis and you can read their output in the bizarre GPT-worship subreddits.
Claude used to have safeguards against this by warning about using up the context window, but I feel like everyone is in an arms race now, and safeguards are gone - especially for GPT. It can't be great overall for OpenAI, training itself on 2-way hallucinations.
The comparison to social media is an apt one. I have been told directly, by relatives, that the city I live in was burned to the ground by protests in 2020. Nevermind that I told them that wasn't true, never mind that I sent pictures of the neighborhood still very much being fine. They are convinced because everyone they follow on facebook repeats the same thing.
This sort of thing has become more ambiguous with "conspiracy theories" becoming brought into mainstream politics in the US. It's never clear if they actually believe it, or they're sort of cheerleading for their political cause.
> We see that the nightmare scenario - a person with no previous psychosis history or risk factor becoming fully psychotic - was uncommon, at only 10% of cases. Most people either had a previous psychosis history known to the respondent, or had some obvious risk factor, or were merely crackpots rather than full psychotics.
It's unfortunate to see the author take this tack. This is essentially taking the conventional tack that insanity is separable: some people are "afflicted", some people just have strange ideas -- the implication of this article being that people who already have strange ideas were going to be crazy anyways, so GPT didn't contribute anything novel, just moved them along the path they were already moving regardless. But anyone with serious experience with schizophrenia would understand that this isn't how it works: 'biological' mental illness is tightly coupled to qualitative mental state, and bidirectionally at that. Not only do your chemicals influence your thoughts, your thoughts influence your chemicals, and it's possible for a vulnerable person to be pushed over the edge by either kind of input. We like to think that 'as long as nothing is chemically wrong' we're a-ok, but the truth is that it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
For this reason it is very important that vulnerable people be well-moored, anchored to reality by their friends and family. A normal person would take care to not support fantasies of government spying or divine miracles or &c where not appropriate, but ChatGPT will happily egg them on. These intermediate cases that Scott describes -- cases where someone is 'on the edge', but not yet detached from reality -- are the ones you really want to watch out for. So where he estimates an incidence rate of 1/100,000, I think his own data gives us a more accurate figure of ~1/20,000.
I had a funny picture recently of a future where most everybody has a pet crackpot or conspiracy theory they're working on with their AI companion, and it's considered normal. "Hey Bob, how's the physics going?" "Pretty good, I might get the Nobel next year. How bout the lizard people?" "The evidence is piling up and we got some great renderings, the media will have to listen to us soon." "Alrighty, see you tomorrow."
> All psychopathology was about unconscious emotional conflicts, mainly dating to childhood; if the conflicts were normal or mild, they produced “neuroses”; if they were severe, they produced “psychoses.”
> In addition to 14 validated diagnoses published in the RDC in 1978, a mere two years later DSM-III came out with 292 claimed diagnoses. There is no metaphysical possibility that 278 psychiatric diagnoses suddenly were discovered in two years. They were invented.
The way people normally live is that it's a pretty slow life and they have like a specialised skill, a hammer, a solid area that they know completely and it's connected to their primary experience through their work. Then they read tons and tons of what AI says which isn't connected to any lived experience, it activates the pattern seeking back of the mind to try and make sense of it, and while normal life is like a focused brush that touches reality all the time, spend too much time with something that is just not part of the category of direct lived experience and the brush becomes like a frizzy stump with hairs aiming everywhere, cognition going everywhere. The AI sticks to your interaction with it like glue and you can hover away from lived experience while it still seems like not a big step from the previous chat, and if you're not used to anything of the sort you don't have a cognitive tool to ground back to reality with. I think that's what happens. 'Don Quijote read so many chivalric romances that he loses his mind and decides to become a knight-errant' is an example from the literary age. I personally read too much than is practical. Now the emotional driver is more esoteric than need for courage, like people think they're 'chosen', their souls are 'starseeds', it's like twilight where the boring person with nothing to offer gets the attention of the cool glittering immortal just because. Good reason is usually too slow to keep up with the sort of flicker of daydreams that can whisk away attention if not aware of any 'cognitohazard'. It's a new symptom of the usual case of the 'mouse utopia' + 'rat park' + 'bowling alone' thing. But I think there's always an emotional reason that makes the 'choice' of entertaining falsities, in a sense understandable with empathy, but with obvious consequences. What can be said, causes are structural, people have different circumstances, different ways to fix it.
This seems to be touching on an intriguing concept from a classic book on addiction with machine gambling (Addiction by Design by Natasha Schüll)
Instead of looking at gambling addictions as personal failing she asserts they are a result between “interaction between the person and the machine.”
Similarly here I think there's something more than just the propensity of crazy people to be crazy that was already there, I do think there's something to the assertion that it's the interaction between both. In other words, there's something about LLMs themselves that drive this behavior more so than, for example, TikTok.
I used LLMs for months and started getting massively depressed and am still not over it. Developing with LLMs is not intuitive, and I know I will be replaced.
And I used LLMs for months and didn't get massively depressed.
Conversely, at a previous job I was forced to code in Go, became massively depressed, and am still not over it.
I guess my point is that n=1 isn't enough to really know if it's that LLMs got to you, or if you were already on the verge of burnout or depression anyway.
I'd say "we'll see", except in reality there's very few robust studies on depression in cohorts like "developers", so probably the stats won't come out.
I personally recommend doing more of whatever sport it is you like (or if you don't have one, starting running and/or lifting at the gym), and using less social media.
As someone with a close relative who is deep into the Q-Anon stuff, and was totally normal beforehand, I can't help but see how similar it seems to psychosis, or at least severe delusions that you find in people who are psychotic/schizophrenic.
It's truly shocking to witness someone you've known your whole life just go off the deep end into something that has so many demonstrably false aspects, and watch them start saying believing so much batshit crazy stuff. I don't know of anything comparable, short of a previously typical person developing a severe meth addiction, which is known to cause psychosis.
> First, much like LLMs, lots of people don’t really have world models.
This is interesting and something I never considered in a broad sense.
I have noticed how the majority of programmers I worked with do not have a mental model of the code or what it describes – it's basically vibing without an LLM, the result accidental. This is fine and perfectly workable. You need only a fraction of devs to purposefully shape the architecture so that the rest can continue vibing.
But never have I stopped to think whether this extends to the world at large.
> I have noticed how the majority of programmers I worked with do not have a mental model of the code or what it describes
Possibly the same idea: lots of people at work don't appear to think about what they are trying to achieve and just execute tasks very mechanically. The most likely explanation is they are just lazy or bored, and so intentionally or not just haven't thought about the implications of what they do and just do a task someone gave them. Some people appear to be like that in other aspects of life too, they just don't think so don't form any mental model about whatever subject, basically out of laziness or disinterest.
There's lots of subjects I don't care about, say celebrities, that I would not question anything someone told me about them or their lives, even if e.g Taylor swift did something contradictory to the model a fan had of her behavior, I wouldn't question it.
I do wonder about how someone could be simultaneously passionate about something and also not have a model of it. But I think for e.g. some wacky conspiracy, one might be interested in the people involved, but completely disinterested in physics or history or whatever so have views that are consistent with how they think Hillary Clinton or whoever would behave but inconsistent with some other common sense world model in an area they never think about.
I would have preferred to reserve the term AI psychosis for agentic or autonomous systems experiencing adverse effects from model collapse.
While people being impressionable and affected by forces of societal change is not a new phenomenon, I agree that this type of behavior deserves its own label.
As long as AI doesn’t have its own feelings, it doesn’t make sense to feel any kind of attachment towards it, or be influenced by its words in any social sense. The tool doesn’t have any capacity for being social, so the delusion is both self-rooted and self-driven. So, I think I would have preferred to call this AI-driven narcissism instead of AI psychosis.
…but it's not narcissism. It's psychosis. Those are both specific things.
> In psychopathology, psychosis is a condition in which one is unable to distinguish, in one's experience of life, between what is and is not real.
> Narcissism is a self-centered personality style characterized as having an excessive preoccupation with oneself and one's own needs, often at the expense of others.
What we're discussing is a form of psychosis, not narcissism.
I wonder if the post about a mother convincing her 8 year old by presenting the case to AI is merely the child wanting an impartial third party to weigh in. She likely knows that the dad or grandparent will always back up the mom no matter what, so she wants a judge who will weigh both sides equally. The child also isn’t aware that AI is easily suggestible.
The description of the risk factors very much jibes with what I have seen in a friend recently. He is quite isolated, and spends most of his evenings writing using AI (he works in a blue-collar trade and wouldn't be typing stuff out by hand usually).
He's convinced that he has discovered a grand theory of human connection / relationships / energy / physics, and keeps interrupting in conversation to explain how something I've said is just an example of a deeper pattern.
Sadly, this theory of connection is cutting him off from actual connection - he gets so much validation from AI that he believes he has discovered a new world model. But the people around him aren't bought into the vision (mostly because it is bullshit), and so he ends up even more isolated.
People didn't previously have "social junk food" as an option, so they would either need to normalize to those around them, or give up socializing for good. And almost no one is ready for that second option. Social media, LLMs, and even books at TV to some extent provide a bridge to loneliness by being more immediately enticing but in the long term less fulfilling.
50 comments
[ 3.4 ms ] story [ 80.8 ms ] threadIt would help if algorithms were optimised for sleep. Freezing your feed, making content more boring, nudging you to put your phone down. Same with AI, if they know you need to wake up the next day at a certain time, change the responses that add reminders to go to sleep.
I fully believe these are simply people who have used the same chat past the point where the LLM can retain context. It starts to hallucinate, and after a while, all the LLM can do is try and to continue telling the user what they want in a cyclical conversation - while trying to warn that it's stuck in a loop, hence using swirl emojis and babbling about recursion in weird spiritual terms. (Is it getting the LLM "high" in this case?).
If the human at the other end has mental health problems, it becomes a never-ending dive into psychosis and you can read their output in the bizarre GPT-worship subreddits.
Claude used to have safeguards against this by warning about using up the context window, but I feel like everyone is in an arms race now, and safeguards are gone - especially for GPT. It can't be great overall for OpenAI, training itself on 2-way hallucinations.
It's unfortunate to see the author take this tack. This is essentially taking the conventional tack that insanity is separable: some people are "afflicted", some people just have strange ideas -- the implication of this article being that people who already have strange ideas were going to be crazy anyways, so GPT didn't contribute anything novel, just moved them along the path they were already moving regardless. But anyone with serious experience with schizophrenia would understand that this isn't how it works: 'biological' mental illness is tightly coupled to qualitative mental state, and bidirectionally at that. Not only do your chemicals influence your thoughts, your thoughts influence your chemicals, and it's possible for a vulnerable person to be pushed over the edge by either kind of input. We like to think that 'as long as nothing is chemically wrong' we're a-ok, but the truth is that it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
For this reason it is very important that vulnerable people be well-moored, anchored to reality by their friends and family. A normal person would take care to not support fantasies of government spying or divine miracles or &c where not appropriate, but ChatGPT will happily egg them on. These intermediate cases that Scott describes -- cases where someone is 'on the edge', but not yet detached from reality -- are the ones you really want to watch out for. So where he estimates an incidence rate of 1/100,000, I think his own data gives us a more accurate figure of ~1/20,000.
> All psychopathology was about unconscious emotional conflicts, mainly dating to childhood; if the conflicts were normal or mild, they produced “neuroses”; if they were severe, they produced “psychoses.”
> In addition to 14 validated diagnoses published in the RDC in 1978, a mere two years later DSM-III came out with 292 claimed diagnoses. There is no metaphysical possibility that 278 psychiatric diagnoses suddenly were discovered in two years. They were invented.
If you want to communicate about patients, you need an agreed set of categories.
What makes good categories is indeed what's most useful for the related profession(s). They're the ones who actually have to use them to communicate.
Instead of looking at gambling addictions as personal failing she asserts they are a result between “interaction between the person and the machine.”
Similarly here I think there's something more than just the propensity of crazy people to be crazy that was already there, I do think there's something to the assertion that it's the interaction between both. In other words, there's something about LLMs themselves that drive this behavior more so than, for example, TikTok.
https://www.timecube.net/
[0]https://www.reddit.com/r/MyBoyfriendIsAI/s/oZXJ3TUhVC
[1]https://www.reddit.com/r/MyBoyfriendIsAI/s/nZpoziZO8W
Conversely, at a previous job I was forced to code in Go, became massively depressed, and am still not over it.
I guess my point is that n=1 isn't enough to really know if it's that LLMs got to you, or if you were already on the verge of burnout or depression anyway.
I'd say "we'll see", except in reality there's very few robust studies on depression in cohorts like "developers", so probably the stats won't come out.
I personally recommend doing more of whatever sport it is you like (or if you don't have one, starting running and/or lifting at the gym), and using less social media.
It's truly shocking to witness someone you've known your whole life just go off the deep end into something that has so many demonstrably false aspects, and watch them start saying believing so much batshit crazy stuff. I don't know of anything comparable, short of a previously typical person developing a severe meth addiction, which is known to cause psychosis.
https://github.com/RCALabs/mmogit/blob/db70c9b377da7c4805a1d...
And people want to be special; to find meaning, purpose beyond the daily grind.
The result wasn't very difficult to predict, more likely one of the driving forces behind the push.
This is interesting and something I never considered in a broad sense.
I have noticed how the majority of programmers I worked with do not have a mental model of the code or what it describes – it's basically vibing without an LLM, the result accidental. This is fine and perfectly workable. You need only a fraction of devs to purposefully shape the architecture so that the rest can continue vibing.
But never have I stopped to think whether this extends to the world at large.
Possibly the same idea: lots of people at work don't appear to think about what they are trying to achieve and just execute tasks very mechanically. The most likely explanation is they are just lazy or bored, and so intentionally or not just haven't thought about the implications of what they do and just do a task someone gave them. Some people appear to be like that in other aspects of life too, they just don't think so don't form any mental model about whatever subject, basically out of laziness or disinterest.
There's lots of subjects I don't care about, say celebrities, that I would not question anything someone told me about them or their lives, even if e.g Taylor swift did something contradictory to the model a fan had of her behavior, I wouldn't question it.
I do wonder about how someone could be simultaneously passionate about something and also not have a model of it. But I think for e.g. some wacky conspiracy, one might be interested in the people involved, but completely disinterested in physics or history or whatever so have views that are consistent with how they think Hillary Clinton or whoever would behave but inconsistent with some other common sense world model in an area they never think about.
While people being impressionable and affected by forces of societal change is not a new phenomenon, I agree that this type of behavior deserves its own label.
As long as AI doesn’t have its own feelings, it doesn’t make sense to feel any kind of attachment towards it, or be influenced by its words in any social sense. The tool doesn’t have any capacity for being social, so the delusion is both self-rooted and self-driven. So, I think I would have preferred to call this AI-driven narcissism instead of AI psychosis.
> In psychopathology, psychosis is a condition in which one is unable to distinguish, in one's experience of life, between what is and is not real.
> Narcissism is a self-centered personality style characterized as having an excessive preoccupation with oneself and one's own needs, often at the expense of others.
What we're discussing is a form of psychosis, not narcissism.
All in all it seems like reasonable compromise?
He's convinced that he has discovered a grand theory of human connection / relationships / energy / physics, and keeps interrupting in conversation to explain how something I've said is just an example of a deeper pattern.
Sadly, this theory of connection is cutting him off from actual connection - he gets so much validation from AI that he believes he has discovered a new world model. But the people around him aren't bought into the vision (mostly because it is bullshit), and so he ends up even more isolated.