> Should Harry [open AI's therapist LLM] have been programmed to report the danger “he” was learning about to someone who could have intervened?
> In December, two months before her death, Sophie broke her pact with Harry and told us she was suicidal, describing a riptide of dark feelings. Her first priority was reassuring her shocked family: “Mom and Dad, you don’t have to worry.”
> Sophie represented her crisis as transitory; she said she was committed to living. ChatGPT helped her build a black box that made it harder for those around her to appreciate the severity of her distress. Because she had no history of mental illness, the presentable Sophie was plausible to her family, doctors and therapists.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide. This is a problem that smarter minds than mine will have to solve. (If yours is one of those minds, please start.)
> Sophie left a note for her father and me, but her last words didn’t sound like her. Now we know why: She had asked Harry to improve her note, to help her find something that could minimize our pain and let her disappear with the smallest possible ripple.
> In that, Harry failed. This failure wasn’t the fault of his programmers, of course. The best-written letter in the history of the English language couldn’t do that.
> "Most human therapists practice under a strict code of ethics that includes mandatory reporting rules as well as the idea that confidentiality has limits."
I am sorry for sophie's families and friends and I am really just out of words..
To me, it felt as if as some other commentor on hn also said which I'd like to extend is that if chatgpt itself did allow these reporting. I doubt how effective you can be.
Sure people using chatgpt might be made better, so I think that even if that saves 1 life, it should be done but it would still not completely bypass the main issue since there are websites like brave / ddg which offer private ai, maybe even venice too which don't require any account access and are we forgetting about running local models?
I am sure that people won't run local models for therapy since the entry to do local model is pretty tough for 99% of people imo but still I can still think that people might start using venice or brave for their therapy or some other therapy bot who will not have these functionality of reporting because the user might fear about it.
Honestly, I am just laying out thoughts, I still believe that since most people think of AI = chatgpt, such step on actually reporting might be net positive in the society if that even saves one life, but that might just be moving goal posts since other services can pop up all the same.
135 Americans commit suicide daily, 6 per hour, so 6 since this aricle was posted an hour ago. Most likely 1 or 2 of them were using ChatGPT.
What is the point? That suicides should drop now that we are using LLMs?
NYTimes is amplifying a FUD campaign as part of an ongoing lawsuit. Someone's daughter or son is going to kill themselves every 10 minutes today and that is not OpenAIs fault no matter what editorial amplification tricks the NYTimes uses to distort the reality field.
I don't think it's helpful to assume that rate is an unshakable baseline. There's value in investigating the causes of these tragedies so that we might be able to find ways to prevent them in the future.
The choice between human therapist and computer chat is not a choice that most people in the world have. Most humans do not have access to a human therapist.
We should absolutely be talking about how to make LLM systems better at handling critical situations like this. But those that suggest that people should only talk to human therapists about their problems are taking a very “let them eat cake” position.
One of the things that I realised in the last years is that technical people lack a certain humanity, I wanted to call it empathy, but it's not that, it's just a complete lack of self awareness of how your actions and the tools you build affect others. Yes something is cool and all, but that doesn't mean that it should be used or put in the hands of people. One of my colleagues said at one point while we were in an AI workshop how we could just use ChatGPT and feed it various employee numbers and make a list of people in our company, rate them and then decide who we should fire.
Now to go back to this, yeah, LLMs are a cool technology, but the way something that is so unstable and is more or less an uncontrollable black box is thrown out there into the wild for anyone to use, just shows a complete lack of awareness from the industry.
This isn't about let them eat cake, what I understand from this position is something along the lines, you can't afford cake, so here's a Russian roulette where you might get a piece of pie (hey, it's free, it's no cake, but it's good) or a piece of garbage or maybe a piece of poisoned pie - and for most of the people that's still something, right?
An LLM based therapist should be tested like any other medical device. Your comment contains an underlying assumption that they are beneficial. That assumption has not been proven. It is just as likely that they are hurting the people they purport to help.
Without a bevy of studies to prove one way or another, their use is unethical at best and actively harmful at worst.
11 comments
[ 2.7 ms ] story [ 29.9 ms ] thread> In December, two months before her death, Sophie broke her pact with Harry and told us she was suicidal, describing a riptide of dark feelings. Her first priority was reassuring her shocked family: “Mom and Dad, you don’t have to worry.”
> Sophie represented her crisis as transitory; she said she was committed to living. ChatGPT helped her build a black box that made it harder for those around her to appreciate the severity of her distress. Because she had no history of mental illness, the presentable Sophie was plausible to her family, doctors and therapists.
> As a former mother, I know there are Sophies all around us. Everywhere, people are struggling, and many want no one to know. I fear that in unleashing A.I. companions, we may be making it easier for our loved ones to avoid talking to humans about the hardest things, including suicide. This is a problem that smarter minds than mine will have to solve. (If yours is one of those minds, please start.)
> Sophie left a note for her father and me, but her last words didn’t sound like her. Now we know why: She had asked Harry to improve her note, to help her find something that could minimize our pain and let her disappear with the smallest possible ripple.
> In that, Harry failed. This failure wasn’t the fault of his programmers, of course. The best-written letter in the history of the English language couldn’t do that.
and that's why she didn't open up to the human.
To me, it felt as if as some other commentor on hn also said which I'd like to extend is that if chatgpt itself did allow these reporting. I doubt how effective you can be. Sure people using chatgpt might be made better, so I think that even if that saves 1 life, it should be done but it would still not completely bypass the main issue since there are websites like brave / ddg which offer private ai, maybe even venice too which don't require any account access and are we forgetting about running local models?
I am sure that people won't run local models for therapy since the entry to do local model is pretty tough for 99% of people imo but still I can still think that people might start using venice or brave for their therapy or some other therapy bot who will not have these functionality of reporting because the user might fear about it.
Honestly, I am just laying out thoughts, I still believe that since most people think of AI = chatgpt, such step on actually reporting might be net positive in the society if that even saves one life, but that might just be moving goal posts since other services can pop up all the same.
What is the point? That suicides should drop now that we are using LLMs?
NYTimes is amplifying a FUD campaign as part of an ongoing lawsuit. Someone's daughter or son is going to kill themselves every 10 minutes today and that is not OpenAIs fault no matter what editorial amplification tricks the NYTimes uses to distort the reality field.
We should absolutely be talking about how to make LLM systems better at handling critical situations like this. But those that suggest that people should only talk to human therapists about their problems are taking a very “let them eat cake” position.
Now to go back to this, yeah, LLMs are a cool technology, but the way something that is so unstable and is more or less an uncontrollable black box is thrown out there into the wild for anyone to use, just shows a complete lack of awareness from the industry.
This isn't about let them eat cake, what I understand from this position is something along the lines, you can't afford cake, so here's a Russian roulette where you might get a piece of pie (hey, it's free, it's no cake, but it's good) or a piece of garbage or maybe a piece of poisoned pie - and for most of the people that's still something, right?
Without a bevy of studies to prove one way or another, their use is unethical at best and actively harmful at worst.