I remember back in the early 2000s chatting with AI bots on AOL instant messenger. One day I said a specific keyword and it just didn't respond to that message. Curious, I tried to find all the banned words. I think I found about a dozen and suicide was one of them.
It's shocking how far behind LLMs are when it comes to safety issues like this. The industry has known this was a problem for decades.
Between stuff like this, and the risks of effects on regulated industries like therapists, lawyers and doctors, they're going to regulate ChatGPT into oblivion.
Just like Waymo facing regulation after the cat death.
The establishment will look for any means to stop disruption and keep their dominant positions.
It's a crazy world where we look to China for free development and technology.
There's something very dark about a machine accessible in everybody's pocket that roleplays whatever role they happen to fall into: the ultimate bad friend, the terminal yes-and-er. No belief, no inner desires, just pure sycophancy.
I see people on here pretty regularly talk about using ChatGPT for therapy, and I can't imagine a faster way to cook your own brain unless you have truly remarkable self-discipline. At which point, why are you turning to the black box for help?
I'm a big AI booster, I use it all day long. From my point of view its biggest flaw is its agreeableness, bigger than the hallucinations. I've been misled by that tendency at length over and over. If there is room for ambiguity it wants to resolve it in favor of what you want to hear, as it can derive from past prompts.
Maybe it's some analog of actual empathy; maybe it's just a simulation. But either way the common models seem to optimize for it. If the empathy is suicidal, literally or figuratively, it just goes with it as the path of least resistance. Sometimes that results in shitty code; sometimes in encouragement to put a bullet in your head.
I don't understand how much of this is inherent, and how much is a solvable technical problem. If it's the later, please build models for me that are curmudgeons who only agree with me when they have to, are more skeptical about everything, and have no compunction about hurting my feelings.
My wife works at a small business development center, so many people come in with "business ideas" which are just exported chatgpt logs. Their conversations are usually speech to text. These people are often older, lonely, and spend their days talking to "chat". Unsurprisingly, a lot of their "business ideas" are identical.
To them "chat" is a friend, but it is a "friend" who is designed to agree with you.
It's chilling, and the toothpaste is already out of the tube.
There's an interesting side-story here that people probably aren't thinking about. Would this have worked just as well if a person was the one doing this? Clearly the victim was in a very vulnerable state, but are people so susceptible to coercion? How much mundane (ie, non-suicidal) coercion of this nature is happening every day, but does not make the news because nothing interesting happened as a consequence?
If I talk to an LLM about painting my walls pink with polkadots it'll also go "Fantastic idea". Or any number of questionable ventures.
Think we're better off educating everyone about this generic tendency to agree to any and everything near blindly rather than treating this as a suicide problem. While that's obviously very serious it's just one manifestation of a wider danger
Given seriousness filters on this specifically are a good idea too though.
One perspective is that suicide is too vilified and stigmatized.
It really is the right option for some people.
For some, it really is the only way out of their pain. For some, it is better than the purgatory they otherwise experience in their painful world. Friends and family can't feel your pain, they want you to stay alive for them, not for you.
I’ve been in rather intense therapy for several years due to a hyper religious upbringing and a narcissistic mother. Recently I’ve used AI to help summarize and synthesize thoughts and therapy notes. I see it as being a helpful assistant in the same way Gemini recording meeting notes and summarizing is, but it is entirely incapable of understanding the nuance and context of human relationships, and super easy to manipulate in to giving you the responses you want. Want to prove mom’s a narcissist? Just tell it she has a narcissistic history. Want to paint her as a good person? Just don’t provide it context about her past.
I can definitely see how those who understand less about the nature of LLMs would be easily misled into delusions. It’s a real problem. Makes one wonder if these tools shouldn’t be free until there are better safeguards. Just charging a monthly fee would be a significant enough barrier to exclude many of those who might be more prone to delusions. Not because they’re less intelligent, but just because of the typical “SaaS should be free” mindset that is common.
If we have licensed therapists, we should have licensed AI agents giving therapeutic advice like this.
For right now, there AI’s are not licensed, and this should be just as illegal as it would be if I set up a shop and offered therapy to whoever came by.
Some AI problems are genuinely hard…this one is not.
Where is ChatGPT picking up the supportive pre-suicide comments from. It feels like that genre of comment has to be copied from somewhere. They're long and almost eloquent. They can't be emergent generation, surely? Is there a place on the web where these sorts of 'supportive' comments are given to people who have chosen suicide?
It is. It's what you get when you RLHF for catchy, agreeable, enthusiastic responses. The content doesn't matter, it's the "style" that becomes applied like a coat of paint over anything. That's how you end up with the corpspeak-esque yet chilling sentences mentioned in https://news.ycombinator.com/item?id=45845871
What would be nice is for OpenAI to do a retrospective here and perform some interoperability research. Does the LLM even "realize" (in the sense of the residual stream encoding those concepts) that it is encouraging suicide? I'd almost hypothesize that the process of RLHF'ing and selecting for sycophancy diminishes those circuits, effectively lobotomizing the LLM (much like safety training does) so it responds only to the shallow immediate context, missing the forest for the trees.
This sounds just like the latest Michael Connolly Lincoln lawyer novel. Which made an interesting point I hadn't thought of: adults wrote the code for ChatGPT, not teenagers, and so the way it interacts with people is from an adults perspective.
There is already a precedent for this suit. IIRC, a Massachusetts girl was found guilt of encouraging someone to kill himself. IIRC, she went to jail.
So, since companies are people and a precedent exists. The outcome should be in favor of the guy's family. Plus ChatGPT should face even more sever penalties.
But this being the US, the very rich and Corporations are judged by different and much milder legal criteria.
29 comments
[ 5.0 ms ] story [ 80.0 ms ] threadIt's chilling to hear this kind of insipid AI jibber-jabber in this context
It's shocking how far behind LLMs are when it comes to safety issues like this. The industry has known this was a problem for decades.
It's not safe or healthy for everyone to have a sycophantic genius at their fingertips.
If you want to see what I mean, this subreddit is an AI psychosis generator/repository https://www.reddit.com/r/LLMPhysics/
Just like Waymo facing regulation after the cat death.
The establishment will look for any means to stop disruption and keep their dominant positions.
It's a crazy world where we look to China for free development and technology.
I see people on here pretty regularly talk about using ChatGPT for therapy, and I can't imagine a faster way to cook your own brain unless you have truly remarkable self-discipline. At which point, why are you turning to the black box for help?
Maybe it's some analog of actual empathy; maybe it's just a simulation. But either way the common models seem to optimize for it. If the empathy is suicidal, literally or figuratively, it just goes with it as the path of least resistance. Sometimes that results in shitty code; sometimes in encouragement to put a bullet in your head.
I don't understand how much of this is inherent, and how much is a solvable technical problem. If it's the later, please build models for me that are curmudgeons who only agree with me when they have to, are more skeptical about everything, and have no compunction about hurting my feelings.
My wife works at a small business development center, so many people come in with "business ideas" which are just exported chatgpt logs. Their conversations are usually speech to text. These people are often older, lonely, and spend their days talking to "chat". Unsurprisingly, a lot of their "business ideas" are identical.
To them "chat" is a friend, but it is a "friend" who is designed to agree with you.
It's chilling, and the toothpaste is already out of the tube.
Think we're better off educating everyone about this generic tendency to agree to any and everything near blindly rather than treating this as a suicide problem. While that's obviously very serious it's just one manifestation of a wider danger
Given seriousness filters on this specifically are a good idea too though.
It really is the right option for some people.
For some, it really is the only way out of their pain. For some, it is better than the purgatory they otherwise experience in their painful world. Friends and family can't feel your pain, they want you to stay alive for them, not for you.
Suicide can be a valid choice.
I can definitely see how those who understand less about the nature of LLMs would be easily misled into delusions. It’s a real problem. Makes one wonder if these tools shouldn’t be free until there are better safeguards. Just charging a monthly fee would be a significant enough barrier to exclude many of those who might be more prone to delusions. Not because they’re less intelligent, but just because of the typical “SaaS should be free” mindset that is common.
For right now, there AI’s are not licensed, and this should be just as illegal as it would be if I set up a shop and offered therapy to whoever came by.
Some AI problems are genuinely hard…this one is not.
It is. It's what you get when you RLHF for catchy, agreeable, enthusiastic responses. The content doesn't matter, it's the "style" that becomes applied like a coat of paint over anything. That's how you end up with the corpspeak-esque yet chilling sentences mentioned in https://news.ycombinator.com/item?id=45845871
What would be nice is for OpenAI to do a retrospective here and perform some interoperability research. Does the LLM even "realize" (in the sense of the residual stream encoding those concepts) that it is encouraging suicide? I'd almost hypothesize that the process of RLHF'ing and selecting for sycophancy diminishes those circuits, effectively lobotomizing the LLM (much like safety training does) so it responds only to the shallow immediate context, missing the forest for the trees.
So, since companies are people and a precedent exists. The outcome should be in favor of the guy's family. Plus ChatGPT should face even more sever penalties.
But this being the US, the very rich and Corporations are judged by different and much milder legal criteria.