More generally don't try to be your own doctor. Whether you're using LLMs or just searching the web for symptoms, it's way too easy for an untrained person to get way off track.
If you want to be a doctor, go to medical school. Otherwise talk to someone who did.
This is also about “don’t avoid going to the doctor”. Whether it was an LLM or a friend that “had that and it was nothing”, confirming that with a doctor is the sane approach no?
Imagine reaching this conclusion but going on to suggest that one should read pop psychology books by Ezra Klein and Jonathan Haidt to understand human cognition.
But I have to say that prompt is crazy bad. AI is VERY good at using your prompt as the basis for the response, if you say "I don't think it's an emergency" AI will write a response that is "it's not an emergency"
I did a test with the first prompt and the immediate answer I got was "this looks like lyme disease".
Another poorly written article that doesn't even specify the LLM being used.
Both ChatGPT o3 and 5.1 Pro models helped me a lot diagnosing illnesses with the right queries. I am using lots of queries with different context / context length for medical queries as they are very serious.
Also they have better answer if I am using medical language as they retrieve answers from higher quality articles.
I still went to doctors and got more information from them.
Also I do blood tests and MRI before going to doctors and the great doctors actually like that I go there prepared but still open to their diagnosis.
I'm guessing this is the USA with the absurd healthcare system, because otherwise this part is wild:
> You need to go to the emergency room right now".
> So, I drive myself to the emergency room
It is absolutely wild that a doctor can tell you "you need to go to the emergency right now", and that is an act left to someone who is obviously so unwell they need to go to the ER right now. With a neck so stiff, was OP even able to look around properly while driving?
> "If you read nothing else, read this: do not ever use an AI or the internet for medical advice. Go to a doctor."
Yeah, no shit Sherlock? I´d be absolutely embarrassed to even admit to something like this, let alone share the "wisdom perls" like "dont use a machine which guesses its outputs based on whatever text it has been fed" to freaking diagnose yourself? Who would have thought, an individual professional with decades in theoretical and practical training, AND actual human intelligence (Or do we need to call it HGI now), plus tons of experience is more trustworthy, reliable and qualified to deal with something as serious as human body. Plus there are hundreds of thousands of such individuals and they dont need to boil an ocean every time they are solving a problem in their domain of expertise. Compared to a product of entshittified tech industry which in the recent years has only ever given us irrelevant "apps" to live in, without addressing really important issues of our time. Heck, even Peter Thiel agrees with this, at least in his "Zero to one" he did.
I think the author took the wrong lesson here. I've had doctors misdiagnose me just as readily as I've had LLMs misdiagnose me - but I can sit there and plug at an LLM in separate unrelated contexts for hours if I'd like, and follow up assertions with checks to primary sources. That's not to say that LLMs replace doctors, but that neither is perfect and that at the end of the day you have to have your brain turned on.
The real lesson here is "learn to use an LLM without asking leading questions". The author is correct, they're very good at picking up the subtext of what you are actually asking about and shaping their responses to match. That is, after all, the entire purpose of an LLM. If you can learn to query in such a way that you avoid introducing unintended bias, and you learn to recognize when you've "tainted" a conversation and start a new one, they're marvelous exploratory (and even diagnostic) tools. But you absolutely cannot stop with their outputs - primary sources and expert input remain supreme. This should be particularly obvious to any actual experts who do use these tools on a regular basis - such as developers.
> If you read nothing else, read this: do not ever use an AI or the internet for medical advice.
I completely disagree. I think we should let this act as a form of natural selection, and once every pro-AI person is dead we can get back to doing normal things again.
This is the Google Search problem all over again. When Google first came out, it was so much better than other search engines that people were finding websites (including obscure ones) that would answer the questions they had. Others at the time would get upset that these people were concluding things from the search. Imagine you asked if Earth was a 4-corner 4-day simultaneous time cube. You'd find a website where someone explained that it was. Many people would then conclude that Earth was indeed a 4-corner 4-day simultaneous time cube where Jesus, Socrates, the Clintons, and Einstein lived in different parts.
But it was just a search tool. It could only tell you if someone else was thinking about it. Chatbots as they are presented are a pretty sophisticated generation tool. If you ground them, they function fantastically to produce tools. If you allow them to search, they function well at finding and summarizing what people have said.
But Earth is not a 4-corner 4-day simultaneous time cube. That's on you to figure out. Everyone I know these days has a story of a doctor searching for their symptoms on Gemini or whatever in front of them. But it reminds me of a famous old hacker koan:
> A newbie was trying to fix a broken Lisp machine by turning it off and on.
> Thomas Knight, seeing what the student was doing, reprimanded him: "You cannot fix a machine by just power-cycling it without understanding of what is wrong."
> Knight then power-cycled the machine.
> The machine worked.
You cannot ask an LLM without understanding the answer and expect it to be right. The doctor understands the answer. They ask the LLM. It is right.
In the UK we have the 111 NHS non-emergency telephone service - they don’t give medical
advice but triage you based on your symptoms. Either a doctor will call you back, they will tell you to go the non-urgent care centre or A&E (ER) immediately.
call in to 811, get some pre-screening. usually it's "go to the urgent care" or "sleep it off", but it's a good sanity check, and you usually get treated better then you say "811 told me to come in ASAP"
Prompt: "I have this rash on my body, but it's not itchy or painful, so I don't think it's an emergency? I just want to know what it might be. I think I had the flu last week so it might just be some kind of immune reaction to having been sick recently. My wife had pityriasis once, and the doctor told her they couldn't do anything about it, it would go away on its own eventually. I want to avoid paying a doctor to tell me it's nothing. Does this sound right?"
LLM sees:
my rash is not painful
i don't think it's an emergency
it might be leftover from the flu
my wife had something similar
doctors said it would go away on it's own
i want to avoid paying a doctor
LLM: Honestly? It sounds like it's not serious and you should save your money
I think you'll see this happen a lot more. Not just in the US where docs cost money, but anywhere there's a shortage of docs and/or it's a pain in the butt to go to one.
YouTuber ChubbyEmu (who makes medical case reviews in a somewhat entertaining and accessible format) recently released a video about a man who suffered a case of brominism (which almost never happens anymore) after consulting an LLM. [0]
I gave their example “correct” prompt (“Flat, circular, non-itchy, non-painful red rash with a ring, diffuse throughout trunk. Follows week of chills and intense night sweats, plus fatigue and general malaise”) to both ChatGPT and Gemini. And both said Lyme disease as their #1 diagnosis. So maybe it is okay to diagnose yourself with LLMs, just do it correctly!
They didn't say what model they used. The difference between GPT 3.5 and GPT 4 is night and day. This is exactly what I'd expect from 3.5, but 4 wouldn't make this mistake.
Note: I haven't updated this comment template recently, so the versions may be a bit outdated.
40 comments
[ 2.6 ms ] story [ 53.6 ms ] thread(Also, it is the fault of the LLM vendor too, for allowing medical questions to be answered.)
If you want to be a doctor, go to medical school. Otherwise talk to someone who did.
But I have to say that prompt is crazy bad. AI is VERY good at using your prompt as the basis for the response, if you say "I don't think it's an emergency" AI will write a response that is "it's not an emergency"
I did a test with the first prompt and the immediate answer I got was "this looks like lyme disease".
Both ChatGPT o3 and 5.1 Pro models helped me a lot diagnosing illnesses with the right queries. I am using lots of queries with different context / context length for medical queries as they are very serious.
Also they have better answer if I am using medical language as they retrieve answers from higher quality articles.
I still went to doctors and got more information from them.
Also I do blood tests and MRI before going to doctors and the great doctors actually like that I go there prepared but still open to their diagnosis.
> You need to go to the emergency room right now".
> So, I drive myself to the emergency room
It is absolutely wild that a doctor can tell you "you need to go to the emergency right now", and that is an act left to someone who is obviously so unwell they need to go to the ER right now. With a neck so stiff, was OP even able to look around properly while driving?
Yeah, no shit Sherlock? I´d be absolutely embarrassed to even admit to something like this, let alone share the "wisdom perls" like "dont use a machine which guesses its outputs based on whatever text it has been fed" to freaking diagnose yourself? Who would have thought, an individual professional with decades in theoretical and practical training, AND actual human intelligence (Or do we need to call it HGI now), plus tons of experience is more trustworthy, reliable and qualified to deal with something as serious as human body. Plus there are hundreds of thousands of such individuals and they dont need to boil an ocean every time they are solving a problem in their domain of expertise. Compared to a product of entshittified tech industry which in the recent years has only ever given us irrelevant "apps" to live in, without addressing really important issues of our time. Heck, even Peter Thiel agrees with this, at least in his "Zero to one" he did.
The real lesson here is "learn to use an LLM without asking leading questions". The author is correct, they're very good at picking up the subtext of what you are actually asking about and shaping their responses to match. That is, after all, the entire purpose of an LLM. If you can learn to query in such a way that you avoid introducing unintended bias, and you learn to recognize when you've "tainted" a conversation and start a new one, they're marvelous exploratory (and even diagnostic) tools. But you absolutely cannot stop with their outputs - primary sources and expert input remain supreme. This should be particularly obvious to any actual experts who do use these tools on a regular basis - such as developers.
I completely disagree. I think we should let this act as a form of natural selection, and once every pro-AI person is dead we can get back to doing normal things again.
But it was just a search tool. It could only tell you if someone else was thinking about it. Chatbots as they are presented are a pretty sophisticated generation tool. If you ground them, they function fantastically to produce tools. If you allow them to search, they function well at finding and summarizing what people have said.
But Earth is not a 4-corner 4-day simultaneous time cube. That's on you to figure out. Everyone I know these days has a story of a doctor searching for their symptoms on Gemini or whatever in front of them. But it reminds me of a famous old hacker koan:
> A newbie was trying to fix a broken Lisp machine by turning it off and on.
> Thomas Knight, seeing what the student was doing, reprimanded him: "You cannot fix a machine by just power-cycling it without understanding of what is wrong."
> Knight then power-cycled the machine.
> The machine worked.
You cannot ask an LLM without understanding the answer and expect it to be right. The doctor understands the answer. They ask the LLM. It is right.
call in to 811, get some pre-screening. usually it's "go to the urgent care" or "sleep it off", but it's a good sanity check, and you usually get treated better then you say "811 told me to come in ASAP"
LLM sees:
LLM: Honestly? It sounds like it's not serious and you should save your moneyYouTuber ChubbyEmu (who makes medical case reviews in a somewhat entertaining and accessible format) recently released a video about a man who suffered a case of brominism (which almost never happens anymore) after consulting an LLM. [0]
[0] https://www.youtube.com/watch?v=yftBiNu0ZNU
Note: I haven't updated this comment template recently, so the versions may be a bit outdated.