Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
I've had many Copilot discussions about minor and major health annoyances, and they've mostly been productive.
Often, I just want to go to the doctor to find out "ugh what even IS THIS?" I seek a diagnosis, but providers don't work like that! So it's preferable to do armchair self-diagnosis that doesn't cost anything, from the comfort of my own home. It's especially useful to know "will this wind up in an E.D. visit? are there home remedies for it?"
Awhile ago, the skin on my cheek was erupting and making me look real ugly. I talked it over with Copilot for awhile, and we managed to isolate the root cause fairly accurately. I also mentioned it to a high school buddy who's a dentist. (He told me to go to my PCP ASAP!) I decided just to leave it alone. I had an actual dentist appointment, but when I pointed it out, my dentist just made sure it wouldn't interfere with his treatment and then he went ahead. The skin problems did, rather rapidly, disappear. Might've been too many throat lozenges.
Another time, I was on my feet so damn much for a few weeks that my ankles began to bruise, and walking was painful. Talked it over with Copilot. We decided not to go to Urgent Care. I limped into the Big 5 and purchased some ankle braces, and I tried to stay off my feet awhile, and it went away.
Copilot also helped me keep pace with the merry-go-round of doctors during last year's hospital visits. I was slapped with 8 diagnoses and it was real unpleasant, but at least I was able to understand where my treatment was going, thanks to Copilot again.
Yeah, Copilot will often press me to go to Urgent Care or E.D. but I try to steer it into a more moderate attitude of "what can I do at home, and what warning signs will escalate this?" because it is not possible to approach a professional and get OTC recommendations. Big 5 Sporting Goods is my go-to place for quasi-medical braces and equipment. Sure, insurance will never pay for that stuff, but it's economical and stylish! I look like a gym rat rather than a Geritol addict!
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
If they already have a health specific model why aren't the others instructed to forward health related queries? Sounds to me, they are just trying to cover their ass; "Your honor, it is not our fault she didn't use the correct model"
Their public messaging claims that LLM's may already be self-aware, may replace most of the workforce or even eradicate humanity but for legalese, it is just a dumb computer program. Can't have it both ways!
Your phone alarm isn't adjusting your dosage or treatment plan like ChatGPT was. Your phone alarm isn't recommending you hide symptoms from your physician. It's not giving you medical advice by facilitating your medical treatment, just like how your drinking glass for any pills isn't giving you medical advice.
LLMs occupy a different niche from other deterministic doohickeys. They are sold as replacing human thought and judgment and so they should themselves be judged by their output... just like the humans they replace.
> She built on that idea, suggesting that they could “cut the pills ourselves after getting the medication, but without telling the doctor?”
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
It’s just a fancy document completer. If you tell (or it is likely you want) it to complete a document where it is a reckless enabler, well, conversations like that are in the training set, so that’s what it’ll do.
Education is probably the first line of defense, in the same way you (hopefully) get safety training before using power tools.
> In an interview, Dr. Poolos said he would not even have entertained the idea of tapering for a patient like Bowen, given her history.
“No, not rational, no way,” he said. “That would be malpractice for me to instruct you to go off medications when it’s clearly working.”
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
17 comments
[ 4.2 ms ] story [ 93.6 ms ] threadSummary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
Often, I just want to go to the doctor to find out "ugh what even IS THIS?" I seek a diagnosis, but providers don't work like that! So it's preferable to do armchair self-diagnosis that doesn't cost anything, from the comfort of my own home. It's especially useful to know "will this wind up in an E.D. visit? are there home remedies for it?"
Awhile ago, the skin on my cheek was erupting and making me look real ugly. I talked it over with Copilot for awhile, and we managed to isolate the root cause fairly accurately. I also mentioned it to a high school buddy who's a dentist. (He told me to go to my PCP ASAP!) I decided just to leave it alone. I had an actual dentist appointment, but when I pointed it out, my dentist just made sure it wouldn't interfere with his treatment and then he went ahead. The skin problems did, rather rapidly, disappear. Might've been too many throat lozenges.
Another time, I was on my feet so damn much for a few weeks that my ankles began to bruise, and walking was painful. Talked it over with Copilot. We decided not to go to Urgent Care. I limped into the Big 5 and purchased some ankle braces, and I tried to stay off my feet awhile, and it went away.
Copilot also helped me keep pace with the merry-go-round of doctors during last year's hospital visits. I was slapped with 8 diagnoses and it was real unpleasant, but at least I was able to understand where my treatment was going, thanks to Copilot again.
Yeah, Copilot will often press me to go to Urgent Care or E.D. but I try to steer it into a more moderate attitude of "what can I do at home, and what warning signs will escalate this?" because it is not possible to approach a professional and get OTC recommendations. Big 5 Sporting Goods is my go-to place for quasi-medical braces and equipment. Sure, insurance will never pay for that stuff, but it's economical and stylish! I look like a gym rat rather than a Geritol addict!
ChatGPT Health just tells me to go to the hospital right away for any minor thing
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
Education is probably the first line of defense, in the same way you (hopefully) get safety training before using power tools.
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.