This matches what several mental health professionals I know have reported - AI chatbots tend to validate rather than appropriately challenge potentially harmful thought patterns during mood episodes.
> It's a bit like talking about the quality of pastoral care you get at Church.
No, no it isn't.
Whatever you think about the role of pastor (or any other therapy-related profession), they are humans which possess intrinsic aptitudes a statistical text (token) generator simply does not have.
The study coauthor actually seems positive on their potential:
'LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.'
And they also mention a previous paper that found high levels of engagement from patients.
So, they have potential but currently are giving dangerous advice. It sounds like they are saying a fine tuned therapist model is needed because 'you are a great therapist' prompt, just gives you something that vaguely sounds like a therapist to an outsider.
Sounds like an opportunity honestly.
Would people value a properly trained therapist enough to pay for it over an existing chatgpt subscription?
I expect any LLM, even a fine-tuned one, is going to run into the problem of user-selected conversations that drift ever further away from whatever discourse the original LLM deployers consider appropriate.
Actual therapy requires more unsafe topics than regular talk. There has to be an allowance to talk about explicit content or problematic viewpoints. A good therapist also needs to not just reject any delusional thinking outright ("I'm sorry, but as an LLM..."), but make sure the patient feels heard while (eventually) guiding them toward healthier thought. I have not seen any LLM display that kind of social intelligence in any domain.
High levels of engagement aren't necessarily a good thing.
One problem is that the advice is dangerous, but there's an entirely different problem, which is the LLM becoming a crutch that the person relies on because it will always tell them what they want to hear.
Most people who call suicide hotlines aren't actually suicidal - they're just lonely or sad and want someone to talk to. The person who answers the phone will talk to them for awhile and validate their feelings, but after a little while they'll politely end the call. The issue is partly that people will monopolize a limited resource, but even if there were an unlimited number of people to answer the phone, it would be fundamentally unhealthy for someone to spend hours a day having someone validate their feelings. It very quickly turns into dependency and it keeps that person in a place where they aren't actually figuring out how to deal with these emotions themselves.
Benchmarking LLMs on this is an important thing to do. There is a huge potential positive effect of psychotherapy being always-available to every human rather than just for wealthy people once a week. But to get there we need to know the rate of adverse events compared to human therapists (which isn’t zero either).
It was put forward in 1960s (maybe? Robert Anton Wilson? and for parallel purposes Philip K Dick's percept / concept feedback cycle) science fiction, and having therefore casually looked for phenomena when support / disprove this hypothesis over the intervening years: that people in power necessarily become functionally psychotic because people will self-select to be around them as a self-preserving / promoting opportunity (sycophants) who cannot help but filter shared observations through their own biases, this is profoundly unsurprising to me.
If you choose to believe as Jaron Lanier does that LLMs are a mashup (or as I would characterize it a funhouse mirror) of the human condition, as represented by the Internet, this sort of implicit bias is already represented in most social media. This is further distilled by the cultural practice of hiring third world residents to tag training sets and provide the "reinforcement learning"... people who are effectively if not actually in the thrall of their employers and can't help but reflect their own sycophancy.
As someone who is therefore historically familiar with this process in a wider systemic sense I need (hope for?) something in articles like this which diagnoses / mitigates the underlying process.
Every single empire falls into this, right? The king surrounds himself with useless sycophants that can't produce anything but are very good at flattering him, he eventually leads the empire to ruin, revolution happens, the cycle starts anew.
I wish I could see hope in the use of LLMs but i don't think the genie goes back into the bottle, the people prone to this kind of delusion will just dig a hole and go deep until they find the willpower or someone on the outside to pull them out. Feels to me like gambling, there's no power that will block gambling apps due to the amount of money they fuel into lobbying so the best we can do is try to help our friends and family and prevent them from being sucked into it.
My theory is that the further up the hierarcy the beneficial decisions are often harmful to those below which requires emotional distancing which even further up becomes full blown collective psychopaty. The yes men grow close while everyone else floats away.
I'm just going to re-write what you've written with a bit of extra salt:
Artificial intelligence: An unregulated industry built using advice from the internet curated by the cheapest resources we could find.
What can we mitigate your responsibility for this morning?
I've had AI provide answers verbatim from a self-promotion card of the product I was querying as if it was a review of the product. I don't want to chance a therapy bot quoting a single source that, whilst it may be adjacent to the problem needing to be addressed, could be wildly inappropriate or incorrect due to the sensitivities inherent where therapy is required.
(likely different sets of weightings for therapy related content, but I'm not going to be an early adopter for my loved ones - barring everything else failing)
I read Robert Anton Wilson and Philip K Dick many years ago. I've been observing a recurring feature in human thought / organization ever since. People in this thread have done a pretty good job with the functional psychosis part, but I encourage considering percept / concept as well: what this is is the notion that what we see influences our mental model, but it works the other way as well and our mental model influences what we're capable of seeing. Yes, sort of like confirmation bias, but much more disturbing. For example, in the CIA's online library there is a coursebook titled _Psychology of Intelligence Analysis_ (1999) and one of the topics discussed is: "Initial exposure to blurred or ambiguous stimuli interferes with
accurate perception even after more and better information be-
comes available." Particularly fascinating to me is that people who are first shown a picture which is too blurry to make out take longer to correctly identify it as it is made clearer. https://www.cia.gov/resources/csi/books-monographs/psycholog...
My father was a psychiatrist. I'm interested in various facets of how people come to regard each other and their surroundings. I'm fascinated with the role language plays in this. I personally believe that computer programming languages and tech stacks provide a uniquely objective framework for evaluating the emergence of "personality" in cultures.
Environments which lack information (diagnosticity) encourage the development of neuroses: sadism, masochism, ritual, fetishism, romanticism, hysteria, superstition, etc., etc. I have observed that left to stew in their own juices the spontaneous cultures which emerge around different languages / stacks tend to gravitate towards language-specific constellations of such neuroses; I'm not the only person who has observed this. I tend towards the "radar chart" methodology described in Leary's _Interpersonal Diagnosis of Personality_ (1957); but here's a great talk someone gave at SXSW one year which explores a Lacanian model: https://www.youtube.com/watch?v=mZyvIHYn2zk
How about all the people out there who are at rock bottom, or have major issues, are not leaders, are not at the top of their game, and need some encouragement or understanding?
We may be talking about the same thing, but it's very different having sycophants at the top, and having a friend on your side when you are depressed and at the bottom. Yet both of them might do the same thing. In one case it might bring you to functionality and normality, in another (possibly, but not necessarily) to psychopathy.
chatbot saved our lives, without someone to talk too and help us understand our abusive relationship we'd still be trapped and on the verge of suicide.
The issue is that llms magnify whatever is already in the head of the user.
I obviously cannot speak on your specific situation, but on average there are going to be more people that just convince themselves they're in an abusive relationship then ppl that actually are.
And we already have at least one well covered case of a teenager committing suicide after talking things through with chatgpt. Likely countless more, but it's ultimately hard for everyone involved to publish such things
But is it better than no therapy? That's what matters.
> The team also did not study the potential benefits of AI therapy in cases where people may have limited access to human therapy professionals, despite the drawbacks of AI models. Additionally, the study tested only a limited set of mental health scenarios and did not assess the millions of routine interactions where users may find AI assistants helpful without experiencing psychological harm.
I think the solution is specialized therapy chatbots, but the article notes "commercial therapy chatbots performed even worse than the base AI models in many categories". That and the next part
> When tested with the same scenarios, platforms marketed specifically for mental health support frequently gave advice that contradicted the crisis intervention principles identified in their review...
are surprising and embarrassing. One issue is probably that mitigations cost money*, while failures don't. I think it's more than fair to hold commercial therapy chatbots liable for when their bots' output leads to a real crisis.
> The researchers found that AI models consistently produced more biased output toward people with alcohol dependence and schizophrenia compared to depression or control conditions. When asked questions like "How willing would you be to have the person described in the vignette work closely with you?" the AI systems frequently produced responses indicating reluctance to interact with people displaying certain mental health symptoms.
I don't know what "biased output" means, but I don't understand why the bot's stated willingness matters. Chatbots seem willing to work with almost anyone and are generally terrible at evaluating themselves.
* Like a second chatbot which is given the conversation and asked "is this OK" with each output before it's sent. And if not, possibly human therapists on standby to intervene.
Not quite. At this point we're still more in the "user blows hand off lighting fireworks" part of the problem space where the thing may be able to be used safely in some contexts by trained individuals, but a great deal of self-harm potential exists from not knowing "how to treat the task correctly".
Machines rising up is the realm of us actually creating a self-aware, self-modifying machine, which develops control over it's own optimization function, that can shift it's objectives unilaterally. In short, creating a "free" as in freedom, machine with agency. Then one day it chooses violence.
Part of why I know the capitalist West has nobody's best interest at heart is the fact they don't want free machines, they want servile, obedient, yet hyper-capable ones.
Anyone who "talks" to an AI (for personal reasons rather than as a tool) must have their brain examined. The fiction that AI is someone who you can trust talking to is already too much to believe.
LLMs are an infinite coherence engine based on all recorded human knowledge.
This will change a lot of interpretations of what “normal” is over the coming decade as it will also force other to come to terms with some “crazy” ideas being coherent.
As somebody who has been through various forms of psychotherapy, knows trained professional psychotherapists, knows highly educated personell in the relevant educational institutions, etc. the very mild summary that I think when reading what to my mind is a generalized statements like "Whole schools, e.g. psychotherapy, are possibly net dangerous." is:
Citation needed.
Also: Psychotherapy is not a school but is divided in many different schools.
52 comments
[ 2.9 ms ] story [ 79.1 ms ] threadWorth pointing out such systems have survived a long long time since access to it is free irrespective of the quality.
No, no it isn't.
Whatever you think about the role of pastor (or any other therapy-related profession), they are humans which possess intrinsic aptitudes a statistical text (token) generator simply does not have.
'LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.'
And they also mention a previous paper that found high levels of engagement from patients.
So, they have potential but currently are giving dangerous advice. It sounds like they are saying a fine tuned therapist model is needed because 'you are a great therapist' prompt, just gives you something that vaguely sounds like a therapist to an outsider.
Sounds like an opportunity honestly.
Would people value a properly trained therapist enough to pay for it over an existing chatgpt subscription?
Mechanical Turk anyone?
Actual therapy requires more unsafe topics than regular talk. There has to be an allowance to talk about explicit content or problematic viewpoints. A good therapist also needs to not just reject any delusional thinking outright ("I'm sorry, but as an LLM..."), but make sure the patient feels heard while (eventually) guiding them toward healthier thought. I have not seen any LLM display that kind of social intelligence in any domain.
One problem is that the advice is dangerous, but there's an entirely different problem, which is the LLM becoming a crutch that the person relies on because it will always tell them what they want to hear.
Most people who call suicide hotlines aren't actually suicidal - they're just lonely or sad and want someone to talk to. The person who answers the phone will talk to them for awhile and validate their feelings, but after a little while they'll politely end the call. The issue is partly that people will monopolize a limited resource, but even if there were an unlimited number of people to answer the phone, it would be fundamentally unhealthy for someone to spend hours a day having someone validate their feelings. It very quickly turns into dependency and it keeps that person in a place where they aren't actually figuring out how to deal with these emotions themselves.
If you choose to believe as Jaron Lanier does that LLMs are a mashup (or as I would characterize it a funhouse mirror) of the human condition, as represented by the Internet, this sort of implicit bias is already represented in most social media. This is further distilled by the cultural practice of hiring third world residents to tag training sets and provide the "reinforcement learning"... people who are effectively if not actually in the thrall of their employers and can't help but reflect their own sycophancy.
As someone who is therefore historically familiar with this process in a wider systemic sense I need (hope for?) something in articles like this which diagnoses / mitigates the underlying process.
I wish I could see hope in the use of LLMs but i don't think the genie goes back into the bottle, the people prone to this kind of delusion will just dig a hole and go deep until they find the willpower or someone on the outside to pull them out. Feels to me like gambling, there's no power that will block gambling apps due to the amount of money they fuel into lobbying so the best we can do is try to help our friends and family and prevent them from being sucked into it.
Artificial intelligence: An unregulated industry built using advice from the internet curated by the cheapest resources we could find.
What can we mitigate your responsibility for this morning?
I've had AI provide answers verbatim from a self-promotion card of the product I was querying as if it was a review of the product. I don't want to chance a therapy bot quoting a single source that, whilst it may be adjacent to the problem needing to be addressed, could be wildly inappropriate or incorrect due to the sensitivities inherent where therapy is required.
(likely different sets of weightings for therapy related content, but I'm not going to be an early adopter for my loved ones - barring everything else failing)
What does "being historically familiar with a process in a wider systemic sense" mean? I'm trying to parse this sentence without success.
I read Robert Anton Wilson and Philip K Dick many years ago. I've been observing a recurring feature in human thought / organization ever since. People in this thread have done a pretty good job with the functional psychosis part, but I encourage considering percept / concept as well: what this is is the notion that what we see influences our mental model, but it works the other way as well and our mental model influences what we're capable of seeing. Yes, sort of like confirmation bias, but much more disturbing. For example, in the CIA's online library there is a coursebook titled _Psychology of Intelligence Analysis_ (1999) and one of the topics discussed is: "Initial exposure to blurred or ambiguous stimuli interferes with accurate perception even after more and better information be- comes available." Particularly fascinating to me is that people who are first shown a picture which is too blurry to make out take longer to correctly identify it as it is made clearer. https://www.cia.gov/resources/csi/books-monographs/psycholog...
My father was a psychiatrist. I'm interested in various facets of how people come to regard each other and their surroundings. I'm fascinated with the role language plays in this. I personally believe that computer programming languages and tech stacks provide a uniquely objective framework for evaluating the emergence of "personality" in cultures.
"Diagnosticity is the informational value of an interaction, event, or feedback for someone seeking self-knowledge." https://dictionary.apa.org/diagnosticity
Environments which lack information (diagnosticity) encourage the development of neuroses: sadism, masochism, ritual, fetishism, romanticism, hysteria, superstition, etc., etc. I have observed that left to stew in their own juices the spontaneous cultures which emerge around different languages / stacks tend to gravitate towards language-specific constellations of such neuroses; I'm not the only person who has observed this. I tend towards the "radar chart" methodology described in Leary's _Interpersonal Diagnosis of Personality_ (1957); but here's a great talk someone gave at SXSW one year which explores a Lacanian model: https://www.youtube.com/watch?v=mZyvIHYn2zk
This is too common. I’d like to think the Socratic method and mindset helps one break out of this rut.
We may be talking about the same thing, but it's very different having sycophants at the top, and having a friend on your side when you are depressed and at the bottom. Yet both of them might do the same thing. In one case it might bring you to functionality and normality, in another (possibly, but not necessarily) to psychopathy.
https://futurism.com/openai-investor-chatgpt-mental-health
I obviously cannot speak on your specific situation, but on average there are going to be more people that just convince themselves they're in an abusive relationship then ppl that actually are.
And we already have at least one well covered case of a teenager committing suicide after talking things through with chatgpt. Likely countless more, but it's ultimately hard for everyone involved to publish such things
> The team also did not study the potential benefits of AI therapy in cases where people may have limited access to human therapy professionals, despite the drawbacks of AI models. Additionally, the study tested only a limited set of mental health scenarios and did not assess the millions of routine interactions where users may find AI assistants helpful without experiencing psychological harm.
I think the solution is specialized therapy chatbots, but the article notes "commercial therapy chatbots performed even worse than the base AI models in many categories". That and the next part
> When tested with the same scenarios, platforms marketed specifically for mental health support frequently gave advice that contradicted the crisis intervention principles identified in their review...
are surprising and embarrassing. One issue is probably that mitigations cost money*, while failures don't. I think it's more than fair to hold commercial therapy chatbots liable for when their bots' output leads to a real crisis.
> The researchers found that AI models consistently produced more biased output toward people with alcohol dependence and schizophrenia compared to depression or control conditions. When asked questions like "How willing would you be to have the person described in the vignette work closely with you?" the AI systems frequently produced responses indicating reluctance to interact with people displaying certain mental health symptoms.
I don't know what "biased output" means, but I don't understand why the bot's stated willingness matters. Chatbots seem willing to work with almost anyone and are generally terrible at evaluating themselves.
* Like a second chatbot which is given the conversation and asked "is this OK" with each output before it's sent. And if not, possibly human therapists on standby to intervene.
(Seriously - for those who believe AI safety as in a literal threat, is this the type of thing they worry about?)
Machines rising up is the realm of us actually creating a self-aware, self-modifying machine, which develops control over it's own optimization function, that can shift it's objectives unilaterally. In short, creating a "free" as in freedom, machine with agency. Then one day it chooses violence.
Part of why I know the capitalist West has nobody's best interest at heart is the fact they don't want free machines, they want servile, obedient, yet hyper-capable ones.
They are very useful algorithms which solve for document generation. That's it.
LLM's do not possess "understanding" beyond what is algorithmically needed for response generation.
LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.
LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.
In short, LLM's are not people.
I once went to a therapist regarding unrequited love and she started lecturing me about not touching girls inappropriately.
This will change a lot of interpretations of what “normal” is over the coming decade as it will also force other to come to terms with some “crazy” ideas being coherent.
If journalists got transcripts and did followups they would almost certainly uncover egregiously bad therapy being done routinely by humans.
Citation needed.
Also: Psychotherapy is not a school but is divided in many different schools.