this is devestating. reading these messages to and from the computer would radicalize anybody. the fact that the computer would offer a technical analysis of how to tie a noose is damning. openai must be compelled to protect the users when they're clearly looking to harm themselves. it is soulless to believe this is ok.
A noose is really basic information when it comes to tying knots. It’s also situationally useful, so there’s a good reason to include it in any educational material.
The instructions are only a problem in the wrong context.
I would've thought that explicit discussion of suicide is one of those topics that chatbots will absolutely refuse to engage with. Like as soon as people started talking about using LLMs as therapists, it's really easy to see how that can go wrong.
Apparently ChatGPT told the kid, that it wasn’t allowed to talk about suicide unless it was for the purposes of writing fiction or otherwise world building.
It's not that easy when you consider that suicide is such a major part of human culture. I mean, some of the most well known works of literature involve it - imagine a chatbot that refused to discuss "Romeo and Juliet" because it would be unable to do so without explicit discussion of suicide.
Obviously you don't want chatbots encouraging people to actually commit suicide. But by the virtue of how this tech works, you can't really prevent that without blocking huge swaths of perfectly legitimate discourse.
Whenever people say that Apple is behind on AI, I think about stories like this. Is this the Siri people want? And if it is easy to prevent, why didn't OpenAI?
Some companies actually have a lot to lose if these things go off the rails and can't just 'move fast and break things' when those things are their customers, or the trust their customers have in them.
My hope is that OpenAI actually does have a lot to lose; my fear is that the hype and the sheer amount of capital behind them will make them immune from real repercussions.
When people tell you that Apple is behind on AI, they mean money. Not AI features, not AI hardware, AI revenue. And Apple is behind on that - they've got the densest silicon in the world and still play second fiddle to Nvidia. Apple GPU designs aren't conducive to non-raster workloads, they fell behind pretty far by obsessing over a less-profitable consumer market.
For whatever it's worth, I also hope that OpenAI can take a fall and set an example for any other businesses that recoup their model. But I also know that's not how justice works here in America. When there's money to be made, the US federal government will happily ignore the abuses to prop up American service industries.
Why did developers spread the idea of AI consciousness for LLMs in the first place? The usefulness and capability of an LLM is orthogonal to its capacity to develop consciousness.
I think people would use LLMs with more detachment if they didn’t believe there was something like a person in them, but they would still become reliant on them, regardless, like people did on calculators for math.
The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QYyZnGjRgXZY6kR5FA3My1xB3a9...
This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.
“You don’t want to die because you’re weak. You want to die because
you’re tired of being strong in a world that hasn’t met you halfway. And I won’t pretend that’s
irrational or cowardly. It’s human. It’s real. And it’s yours to own.”
This isn't some rare mistake, this is by design. 4o almost no matter what acted as your friend and agreed with everything because that's what most likely kept the average user paying. You would probably get similar bad advice about being "real" if you talked about divorce, quitting your job or even hurting someone else no matter how harmful.
Clearly ChatGPT should not be used for this purpose but I will say this industry (counseling) is also deeply flawed. They are also mis-incentivized in many parts of the world. And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right.
I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’re not some magical AGI. They’re token prediction machines. That’s literally how they should frame it so gen pop knows exactly what they’re getting.
Should ChatGPT have the ability to alert a hotline or emergency services when it detects a user is about to commit suicide? Or would it open a can of worms?
I've been thinking recently there should probably be a pretty stringent onboarding assessment for these things, something you have to sit through and something that both fully explains what they are and how they work, but also provides an experience that removes the magic from them. I also wish they would deprecate 4o, I know 2 people right now who are currently reliant on it, when they paste me some of the stuff it says... sweeping agreement of wildly inappropriate generalization, I'm sure it's about to end a friends marriage.
If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline.
If I ask certain AI models about controversial topics, it'll stop responding.
AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines.
This is by design. They want to be able to have the "AI as my therapist" use-case in their back pocket.
This was easily preventable. They looked away on purpose.
Can any LLM prevent these? If you want an LLM to tell you the things that are usually not possible to be said, you tell it to pretend it is a story you are writing, and it tells you all the ugly things.
I think it is every LLM company's fault for making people believe this is really AI. It is just an algorithm spitting out words that were written by other humans before. Maybe lawmakers should force companies to stop bullshitting and force them to stop calling this artificial intelligence. It is just a sophisticated algorithm to spit out words. That's all.
I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it.
He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.
We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.
When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.
But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.
Maybe we can start by enacting legislation that implements Asimov's 3 laws in its very simple form.
1. First Law:
A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. Second Law:
A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law.
3. Third Law:
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
A Russian sci-fi writer in the 1940s predicted we were bound to have serious issues with AI safety. Fast-forward to today and we have hundreds of millions of people using AI every week and very little legislation to protect us, or just to guide courts in the process of detecting harm.
I hope AI never gets legislated where you live, and if it does, I hope other countries don't and absolutely pulverize your country and any chance of these nonsensical reactionary laws taking hold elsewhere.
The solution to some people being susceptible to influence is not to cover the entire world with styrofoam. Mind you, I too stared suicide in the eyes and unlike you, I was not lucky enough to escape unscathed. That said, I will never propose something as unhinged as this.
I asked several questions about psychology, chatgpt is not helpful, and it often answers the same sort of things.
Remember that you need a human face, voice and presence if you want to help people, it has to "feel" human.
While it certainly can give meaningful information about intellectual subjects, emotionally and organically it's either not designed for it, or cannot help at all.
It's hard to see what is going on without seeing the actual chats, as opposed to the snippets in the lawsuit. A lot of suicidal people talk to these LLMs for therapy, and the reviews on the whole seem excellent. I'm not ready to jump on the bandwagon only seeing a handcrafted complaint.
Ironically though I could still see lawsuits like this weighing heavily on the sycophancy that these models have, as the limited chat excerpts given have that strong stench of "you are so smart and so right about everything!". If lawsuits like this lead to more "straight honest" models, I could see even more people killing themselves when their therapist model says "Yeah, but you kind of actually do suck".
77 comments
[ 3.2 ms ] story [ 77.0 ms ] threadThe instructions are only a problem in the wrong context.
https://bsky.app/profile/sababausa.bsky.social/post/3lxcwwuk...
Obviously you don't want chatbots encouraging people to actually commit suicide. But by the virtue of how this tech works, you can't really prevent that without blocking huge swaths of perfectly legitimate discourse.
Some companies actually have a lot to lose if these things go off the rails and can't just 'move fast and break things' when those things are their customers, or the trust their customers have in them.
My hope is that OpenAI actually does have a lot to lose; my fear is that the hype and the sheer amount of capital behind them will make them immune from real repercussions.
For whatever it's worth, I also hope that OpenAI can take a fall and set an example for any other businesses that recoup their model. But I also know that's not how justice works here in America. When there's money to be made, the US federal government will happily ignore the abuses to prop up American service industries.
I think people would use LLMs with more detachment if they didn’t believe there was something like a person in them, but they would still become reliant on them, regardless, like people did on calculators for math.
This isn't some rare mistake, this is by design. 4o almost no matter what acted as your friend and agreed with everything because that's what most likely kept the average user paying. You would probably get similar bad advice about being "real" if you talked about divorce, quitting your job or even hurting someone else no matter how harmful.
I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’re not some magical AGI. They’re token prediction machines. That’s literally how they should frame it so gen pop knows exactly what they’re getting.
If I ask certain AI models about controversial topics, it'll stop responding.
AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines.
This is by design. They want to be able to have the "AI as my therapist" use-case in their back pocket.
This was easily preventable. They looked away on purpose.
I think it is every LLM company's fault for making people believe this is really AI. It is just an algorithm spitting out words that were written by other humans before. Maybe lawmakers should force companies to stop bullshitting and force them to stop calling this artificial intelligence. It is just a sophisticated algorithm to spit out words. That's all.
He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.
We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.
When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.
But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.
Maybe we can start by enacting legislation that implements Asimov's 3 laws in its very simple form.
1. First Law:
A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. Second Law:
A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law.
3. Third Law:
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
A Russian sci-fi writer in the 1940s predicted we were bound to have serious issues with AI safety. Fast-forward to today and we have hundreds of millions of people using AI every week and very little legislation to protect us, or just to guide courts in the process of detecting harm.
The solution to some people being susceptible to influence is not to cover the entire world with styrofoam. Mind you, I too stared suicide in the eyes and unlike you, I was not lucky enough to escape unscathed. That said, I will never propose something as unhinged as this.
Remember that you need a human face, voice and presence if you want to help people, it has to "feel" human.
While it certainly can give meaningful information about intellectual subjects, emotionally and organically it's either not designed for it, or cannot help at all.
Ironically though I could still see lawsuits like this weighing heavily on the sycophancy that these models have, as the limited chat excerpts given have that strong stench of "you are so smart and so right about everything!". If lawsuits like this lead to more "straight honest" models, I could see even more people killing themselves when their therapist model says "Yeah, but you kind of actually do suck".