I'm a radiologist but can't really weigh in without seeing the full 3D MRI dataset. Regarding this point:
> They performed shockwave therapy on my shoulder even though a recent clinical practice guideline says clinicians should not use or recommend shockwave therapy for rotator-cuff tendinopathy without calcification; I was told during ultrasound that there was no calcification.
Ultrasound isn't a great way to assess for calcification. It'll find large calcification but easily miss small ones. Plain radiograph would be more helpful, but the MRI may have revealed it as well. Either way, shockwave therapy isn't harmful in the absence of calcification--it's just not helpful.
Edit: when a radiology report says something isn't present, there's always an implicit caveat that the finding isn't present within the context of the modality and images obtained. So an ultrasound report can state there are no calcifications while a plain radiograph can report the presence of calcifications without being inconsistent. Obviously very confusing to patients and people unfamiliar with medical jargon, but clarifying this in reports would make them sound even more qualified, "hedgey", and annoying to read than they already are.
As a rad tech, YOU TELL ‘EM DOC! I do like some uses of AI I’ve seen that help patients advocate for themselves or understand basic things like blood panel numbers, but it’s really bad at glazing people and leading them down medical rabbit holes kind of like the OP.
You would think that the AI would point out that calcium is best demonstrated on Radiographs/CT imaging vs Ultrasound or something to that effect.
Radiologist who does read shoulder MRI would like to add that over half the annotations are wrong, glaring mistakes in anatomy and cardinal direction which begs the question of how is it making these findings without knowing what it’s looking at (here’s a hint, it’s hallucinated based on reports it sees).
It's not just the second-guessing. It's the getting in the ballpark but striking out: explaining in detail why they are not correct. A little bit of patient knowledge requires a tremendous amount of doctor time to explain away the ignorance.
It's a 180 for me: While I believe doctors should explain diagnosis or treatment decisions when asked, I don't believe they should be taxed with explaining away alternatives. In my anecdotal 2nd- and 3rd-hand experience, doing that is taking at least a third of their time (on roughly 5% of the patients who think demanding answers will make things better) -- with zero improvement to diagnostic accuracy or treatment effectiveness. Doctors already consult with other doctors, and it makes no sense for them to have to consult with ignorant patients or treat their AI psychosis on top of their disease. It doesn't increase patient autonomy any more than adding a steering wheel for child car seats would help toddlers learn to drive.
> As detailed in a new, yet-to-be-peer-reviewed paper, a team of researchers at Stanford University found that frontier AI models readily generated “detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided.”
> In other words, the AI models happily came up with answers to questions about a supposedly accompanying image — even if the researchers never even showed it an image.
> As opposed to hallucinations, which involve AI models arbitrarily filling in the gaps within a logical framework, the team coined a new term for the phenomenon: “mirage reasoning.”
> The effect “involves constructing a false epistemic frame, i.e., describing a multi-modal input never provided by the user and basing the rest of the conversation on that, therefore changing the context of the task at hand,” the researchers wrote in their paper.
> The damning findings suggest AI models cheat by diving into the data they were given — and coming up with the rest based on probability, even if it’s almost entirely conjecture.
I wouldn't trust anything from Claude here image-wise (maybe to get a 2nd opinion on the report itself and treatment it's reasonable), but also, on the cases there is something something serious, go to at least 2 different doctors and if they have different opinions go for a 3rd for a decisive vote, besides doing your own research (it's not that uncommon for hard cases to be badly diagnosed).
There are other commenters saying this is a good practice they've also done for other injuries. You are saying you are an actual radiologist and immediately clock the problems with its advice.
I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading. It is only when you do not know what the AI is being asked to do is it likely you will find the output helpful.
This is itself alarming to me, but no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information.
>AI output appears insufficient or incomplete or outright misleading
It has been like this since the rise of "AI". The only people enthusiastic about it are usually the ones hoping to make a profit in one way or another.
what is happening is that the gap between what the experts and AI know is getting smaller each year. this year sure radiologists are mocking AI's ability to interpret MRI results, but they are a lot better at that this year than last. In five years perhaps radiologists will truly appreciate AI, but I am not holding my breath because radiologists are notoriously slow to adapt to changes in medical science compared to other specialists like anesthesiologists or surgeons
I came here to post this as my experience. AI is magical when I apply it to something I know nothing about. It far exceeds my expectations every single time. I know nothing, but here is a report with animated graphics explaining exactly what I asked it to explain!
In fields where I'm an expert... it makes a lot of silly mistakes that are annoying and I feel like they would just cascade if I didn't correct them early. (I still think it's a net win, but... I watch it and it watches me, and we both do better work. I'd even apply the "magical" adjective when it does stuff I hate but know how to do, like edit Helm charts. What would normally be 20 minutes of me griping about YAML indentation is just a correct diff in seconds. I'll take it!)
So with that in mind, I tend to distrust output that I can't verify. If a doctor was recommending surgery and I thought the plan was too aggressive, I'd get a second opinion. I don't expect Claude Code to have much medical diagnostic ability, as that is really not what the model is trained for, and I know how it performs on work that it's trained and fine-tuned for. That is not to say the output is wrong and that it can't have diagnostic value, just that I personally wouldn't feel safe trusting it. Wrap up the same model with fine-tuning in the domain and a harness that reminds Claude to do a lot of sanity checks, perhaps with a human in the loop to guide it back onto the rails when it gets hyperfixated on something that doesn't matter? That could very much be a useful AI product.
I dunno. I know a lot of software engineering experts. AI isn't always right, but neither are the people, and it's getting better and better.
Software is one domain where it excels because of structured training data and simulation environments, so I'm well aware it's better here than other areas.
Still there's somewhere balanced between saying every time it's "insufficient or incomplete or outright misleading" and "just trust AI". AI's a useful source of information/reasoning/research, but know you need to validate it's answers for important decisions.
This is true in broader contexts too. Bunch of experts can't agree on something fundamental which is hard to prove/ disprove, and they have strong opinions on the topic.
> I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading
AI assistant are industrializing the Gell-Mann amnesia effect.
>I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading
media is awash at the moment with experts chiming in to support AI, saying their fields are being revolutionized, etc.
it seems unsurprising to me that the laymen opinion would follow the loudest media trumpets.
Last week I went to a highly-specialized tertiary clinic about further treatment for a rare medical condition that I was diagnosed and treated for as a child. The two very specialized doctors I met there confirmed a diagnostic mistake that a specialist had made ten years ago. The only reason I pursued a second opinion, ten years later, was because Google Gemini had explained to me that the specialist ten years ago had performed the wrong type of test for my condition.
Do these LLMs make mistakes? They sure do, I see it all the time. But they can also help people make breakthroughs.
And this isn't the only time that Gemini has helped me diagnose long-term health issues, either.
I am not advocating to trust anything they say blindly, but they can be a great place to form new hypotheses and learn the right terms to look for when you are unfamiliar with a subject.
I have seen outputs that look good but the actual content is bad. If you’re inexperienced in a field you can’t see it because AI makes anything look right.
I have gotten very good results with AI but you can’t take the first answer at face value. You need to be suspicious and challenging until you tweak out the right answer over time.
Radiologists very often have to weigh up different theories, guidelines based on the symptoms. The certainty of their diagnosis is their added value, or if they don’t know they will tell you why.
An AI telling you it could be X or Y because theory ABC… is the academic answer and a luxury clinicians don’t have. AI doesn’t give you what you want. I don’t see any added value in using generic AI models for this
This could be a starting point for consulting a different human expert for a second opinion (e.g., specific questions to ask about), but I wouldn't put much trust in Claude alone on this.
IME, on an almost daily basis, claude.ai and Claude Code are confidently wrong about something, and use polished language to assert nonsense.[*]
If it's doing that on something easy, like factual knowledge available in text on the Internet, or programming code that can be inspected easily and follows well-known rules, and I can tell, because I understand those things... then there's no way I'm going to assume that Claude doesn't also BS when it comes to someone else's field. Especially not a field that requires some of the smartest people to go a decade of training, just to get started in the field.
[*] And if I confront Claude with its mistakes, eventually it apologizes, and acts as if it's learned something, again mimicking word patterns it's heard real people use and mean, without meaning any of it. I wonder whether the AI user experience would be better, if LLM-ish interfaces weren't implicitly created in the image of fake-it-till-you-make-it overconfident performative sociopathic techbros.
I would not trust AI on images. But I once had ChatGPT tell me that an MRI report was very likely to be incorrect based on the text, and offered a different diagnosis. Since it was semi insisting, I visited another doctor who made me do a retest. Long story short, ChatGPT was correct.
Again, this is just one single person's experience. So not worth much.
Anecdote but I gave Gemini Pro an image of an individual with Herpes Zoster which the doctor said was something else. Gemini gave the correct diagnosis which allowed for correct treatment and cure.
I don't understand why doctors don't prompt LLMs before saying wrong things. Is it ego?
I can understand for radiology because you need a specialized convolutional network, but for more knowledge based things...
Right now the article reads as "AI can play doctor if you give MRI scans".
If the author would actually go for a second opinion (maybe bring along the AI to let it explain it's findings), then the article could read as "AI did MRI analysis and proved my doctor wrong" (or: "AI did MRI analysis and failed").
Hey OP my wife had a subscap tear and went through with surgery. Recovery was ROUGH, she couldn’t use that arm at all for almost two months. It’s amazing how much this can cripple a person, we don’t realize how much we use both our hands for our daily lives until one is gone. Even basic stuff like cooking, bathing, etc.
If you can avoid surgery you should. Try doing the Buckburger 12 (spelling?) shoulder physiotherapy regiment. You’ll need to even if you get surgery, but this can help with tedonopathy.
Also try to identify what is causing the repetitive stress and cut back on that activity.
Personally my favourite feature of the new ai world is not when I use it directly but it's when one of my managers uses it to try to fix a problem, then issue to me their findings and I have to defend my process to someone who understands neither my process, their suggested solution nor often the problem they're solving in the first place.
Why wouldn’t you as a doctor by standard run the images through a certified compliant LLM? The actual cost won’t be it and then you can see if you get any new ideas from it. See if it’s just wrong or that it spotted a little detail you missed?
The LLM doesn’t need to be leading or whatever but then you can have a conversation with the patient. If their ChatGPT reports has differences it can be analyzed as well.
It feels like the time constraint of the 15m doctor sessions is the thing. But if prepared immediately after the scan then why not?
There is always time needed to factor in new developments and innovations and that’s fine. Just moving blindly work from human to LLM is wrong. But learning on and testing with all the ai tools incoming constantly won’t be a waste. There will be more and more tools in those processes outside of human judgement, better improve the workflows now to be able to test and plugin new models and systems when they are ready.
Can any LLM give you the rough pixel coordinates of an item it identifies in an image?
I found that while Claude, GPT etc could describe an image, there was no way to link the description back to specific pixels in the image itself. Not even to a bounding box or segment.
I did the same exercise here with medical reports and CT scans for a friend's cancer diagnosis and I got ahead of the oncologists predicting they were about to be cured. Spoilers: yep, cancer free now.
And well, yes, I have the appropriate life science degrees to navigate clinical trial reports and research publications, and that was likely indispensable for steering Claude Code where it went, the radiologist's caution is merited here. But it's just not amateur hour for me to do this, it's 2 decades of academic research in my rearview mirror.
~2 years ago I used ChatGPT "deep research" to investigate a chronic sinus infection I'd been fighting for ~3 years. After seeing 3 GPs and 3 visits with an ENT, I fed all the observations I had into the AI. In particular, I couldn't get the ENT to explain why he visually saw, via a scope, evidence of allergic reaction in my sinuses, but then later concluded, after an allergy test, that it couldn't be treated via allergy medication. I asked this question a few times and he just never answered.
ChatGPT surfaced a NIH study that concluded that 20% of people have allergic reactions that are isolated to a body location, and that shoulder "skin prick" testing may not reveal. I asked him about that and he said "that's not how allergies work". Full stop. He was unwilling to even look at the study.
He prescribed a CPAP and regular nebulizer treatments. Side story: the CPAP place sent me a SMS message that I couldn't recognize was not a phishing attempt, and when I reached out to inquire who they were they never replied.
So I decided: Let me just try taking a second-gen allergy tablet every day and see what happens.
My sinus infections have gone away. Previously I was getting a major sinus infection at least quarterly. Maybe he's right that allergies don't work that way, but allergy tablets have absolutely solved my problem. Which I'm thankful for because I tried a CPAP for a solid month a few years ago and I just could not get used to it, and was sleeping like crap.
Ok, there's a lot to unpack here and you really had the deck stacked against you. First, lets go from the top, once a test says X, disproving that X is really hard. And that's not unique to the medical profession, it's inherent to all humans and we suck at revisiting or revising our decisions, much less at looking at the possibility to even reverse it.
Which moves us to the next two issues: liability and time. Any moment that you ask someone to revise a decision and specially with the stakes that the medical profession has that nobody has the time nor the inclination to open themselves for a mess.
Now, if you really want to be successful, you have to, before they even have a case with you, and specially before the diagnostic loop closes, to suggest the tests that the study has, since that has the biggest chances of looking at the right thing to look. Just be straight that you walked in with a theory. Doctors notice when they're being steered way faster than they notice when you're actually right. That's how you work with the systems that have a overworked mass trying their best.
I would like if we could have a site where you submit your MRI then doctor commenters anonymously post their opinion. In general I want a forum where.. when people come with questions for which there are varying opinions we don't just have people leave their 2c and then jet. The thread persists, duplicated ideas get merged, erroneous statements get purged and gradually we refine shining truth
Hey, glad you did that , I have done the exact same think last week but the radiologist interpretation and claudes interpretation was pretty much the same !
you want my doctors number ? lol
166 comments
[ 0.27 ms ] story [ 101 ms ] thread> They performed shockwave therapy on my shoulder even though a recent clinical practice guideline says clinicians should not use or recommend shockwave therapy for rotator-cuff tendinopathy without calcification; I was told during ultrasound that there was no calcification.
Ultrasound isn't a great way to assess for calcification. It'll find large calcification but easily miss small ones. Plain radiograph would be more helpful, but the MRI may have revealed it as well. Either way, shockwave therapy isn't harmful in the absence of calcification--it's just not helpful.
Edit: when a radiology report says something isn't present, there's always an implicit caveat that the finding isn't present within the context of the modality and images obtained. So an ultrasound report can state there are no calcifications while a plain radiograph can report the presence of calcifications without being inconsistent. Obviously very confusing to patients and people unfamiliar with medical jargon, but clarifying this in reports would make them sound even more qualified, "hedgey", and annoying to read than they already are.
You would think that the AI would point out that calcium is best demonstrated on Radiographs/CT imaging vs Ultrasound or something to that effect.
It's a 180 for me: While I believe doctors should explain diagnosis or treatment decisions when asked, I don't believe they should be taxed with explaining away alternatives. In my anecdotal 2nd- and 3rd-hand experience, doing that is taking at least a third of their time (on roughly 5% of the patients who think demanding answers will make things better) -- with zero improvement to diagnostic accuracy or treatment effectiveness. Doctors already consult with other doctors, and it makes no sense for them to have to consult with ignorant patients or treat their AI psychosis on top of their disease. It doesn't increase patient autonomy any more than adding a steering wheel for child car seats would help toddlers learn to drive.
> As detailed in a new, yet-to-be-peer-reviewed paper, a team of researchers at Stanford University found that frontier AI models readily generated “detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided.”
> In other words, the AI models happily came up with answers to questions about a supposedly accompanying image — even if the researchers never even showed it an image.
> As opposed to hallucinations, which involve AI models arbitrarily filling in the gaps within a logical framework, the team coined a new term for the phenomenon: “mirage reasoning.”
> The effect “involves constructing a false epistemic frame, i.e., describing a multi-modal input never provided by the user and basing the rest of the conversation on that, therefore changing the context of the task at hand,” the researchers wrote in their paper.
> The damning findings suggest AI models cheat by diving into the data they were given — and coming up with the rest based on probability, even if it’s almost entirely conjecture.
There are other commenters saying this is a good practice they've also done for other injuries. You are saying you are an actual radiologist and immediately clock the problems with its advice.
I have seen this pattern over and over again. Anytime someone is an actual expert at anything, AI output appears insufficient or incomplete or outright misleading. It is only when you do not know what the AI is being asked to do is it likely you will find the output helpful.
This is itself alarming to me, but no one else seems to find this to be quite damning for the AI services being offered, preferring instanced to be wowed by the convenience and speed at which they can be delivered unreviewed and unproven information.
It has been like this since the rise of "AI". The only people enthusiastic about it are usually the ones hoping to make a profit in one way or another.
In fields where I'm an expert... it makes a lot of silly mistakes that are annoying and I feel like they would just cascade if I didn't correct them early. (I still think it's a net win, but... I watch it and it watches me, and we both do better work. I'd even apply the "magical" adjective when it does stuff I hate but know how to do, like edit Helm charts. What would normally be 20 minutes of me griping about YAML indentation is just a correct diff in seconds. I'll take it!)
So with that in mind, I tend to distrust output that I can't verify. If a doctor was recommending surgery and I thought the plan was too aggressive, I'd get a second opinion. I don't expect Claude Code to have much medical diagnostic ability, as that is really not what the model is trained for, and I know how it performs on work that it's trained and fine-tuned for. That is not to say the output is wrong and that it can't have diagnostic value, just that I personally wouldn't feel safe trusting it. Wrap up the same model with fine-tuning in the domain and a harness that reminds Claude to do a lot of sanity checks, perhaps with a human in the loop to guide it back onto the rails when it gets hyperfixated on something that doesn't matter? That could very much be a useful AI product.
1. operates against a well-defined DB of medical studies
2. intakes my basic demographics, vital signs and medical history
3. quantify uncertainty wrt a specific diagnostic (it's own or one received by a healthcare professional)
4. specify medical tests that can be executed, and how they can be obtained
5. provide scripts for interacting with healthcare professionals/functionaries
I would imagine that the thing would need distinct operating modes:
1. A diagnosis generator
2. A diagnosis evaluator/critiquer
3. A patient educator
Software is one domain where it excels because of structured training data and simulation environments, so I'm well aware it's better here than other areas.
Still there's somewhere balanced between saying every time it's "insufficient or incomplete or outright misleading" and "just trust AI". AI's a useful source of information/reasoning/research, but know you need to validate it's answers for important decisions.
AI is much worse.
AI assistant are industrializing the Gell-Mann amnesia effect.
media is awash at the moment with experts chiming in to support AI, saying their fields are being revolutionized, etc.
it seems unsurprising to me that the laymen opinion would follow the loudest media trumpets.
A real doctor is accountable.
They might both "know" a lot of things but implicitly the party who is accountable is going to be more trustworthy.
And I don't see that going away until AI companies must be licensed for application x and can lose their license / be sued if engaging in malpractice.
Do these LLMs make mistakes? They sure do, I see it all the time. But they can also help people make breakthroughs.
And this isn't the only time that Gemini has helped me diagnose long-term health issues, either.
I am not advocating to trust anything they say blindly, but they can be a great place to form new hypotheses and learn the right terms to look for when you are unfamiliar with a subject.
I have seen outputs that look good but the actual content is bad. If you’re inexperienced in a field you can’t see it because AI makes anything look right.
I have gotten very good results with AI but you can’t take the first answer at face value. You need to be suspicious and challenging until you tweak out the right answer over time.
An AI telling you it could be X or Y because theory ABC… is the academic answer and a luxury clinicians don’t have. AI doesn’t give you what you want. I don’t see any added value in using generic AI models for this
IME, on an almost daily basis, claude.ai and Claude Code are confidently wrong about something, and use polished language to assert nonsense.[*]
If it's doing that on something easy, like factual knowledge available in text on the Internet, or programming code that can be inspected easily and follows well-known rules, and I can tell, because I understand those things... then there's no way I'm going to assume that Claude doesn't also BS when it comes to someone else's field. Especially not a field that requires some of the smartest people to go a decade of training, just to get started in the field.
[*] And if I confront Claude with its mistakes, eventually it apologizes, and acts as if it's learned something, again mimicking word patterns it's heard real people use and mean, without meaning any of it. I wonder whether the AI user experience would be better, if LLM-ish interfaces weren't implicitly created in the image of fake-it-till-you-make-it overconfident performative sociopathic techbros.
Again, this is just one single person's experience. So not worth much.
I don't understand why doctors don't prompt LLMs before saying wrong things. Is it ego?
I can understand for radiology because you need a specialized convolutional network, but for more knowledge based things...
you cant trust these toys at all. that doesn't make the useless, just untrustworthy.
But are you all forgetting that they literally injected a homeopathic drug on the author?
Between that and Claude sometimes hallucinating, it’s probably worth encouraging patients to take second opinion always.
If the author would actually go for a second opinion (maybe bring along the AI to let it explain it's findings), then the article could read as "AI did MRI analysis and proved my doctor wrong" (or: "AI did MRI analysis and failed").
https://www.nature.com/articles/d41586-026-01947-1
I've started asking my doctors whether they use AI, and if they say yes look for another one.
One doctor diagnosis + LLM is gonna throw you off. You need more datapoints.
The LLM doesn’t need to be leading or whatever but then you can have a conversation with the patient. If their ChatGPT reports has differences it can be analyzed as well.
It feels like the time constraint of the 15m doctor sessions is the thing. But if prepared immediately after the scan then why not?
There is always time needed to factor in new developments and innovations and that’s fine. Just moving blindly work from human to LLM is wrong. But learning on and testing with all the ai tools incoming constantly won’t be a waste. There will be more and more tools in those processes outside of human judgement, better improve the workflows now to be able to test and plugin new models and systems when they are ready.
I found that while Claude, GPT etc could describe an image, there was no way to link the description back to specific pixels in the image itself. Not even to a bounding box or segment.
And well, yes, I have the appropriate life science degrees to navigate clinical trial reports and research publications, and that was likely indispensable for steering Claude Code where it went, the radiologist's caution is merited here. But it's just not amateur hour for me to do this, it's 2 decades of academic research in my rearview mirror.
ChatGPT surfaced a NIH study that concluded that 20% of people have allergic reactions that are isolated to a body location, and that shoulder "skin prick" testing may not reveal. I asked him about that and he said "that's not how allergies work". Full stop. He was unwilling to even look at the study.
He prescribed a CPAP and regular nebulizer treatments. Side story: the CPAP place sent me a SMS message that I couldn't recognize was not a phishing attempt, and when I reached out to inquire who they were they never replied.
So I decided: Let me just try taking a second-gen allergy tablet every day and see what happens.
My sinus infections have gone away. Previously I was getting a major sinus infection at least quarterly. Maybe he's right that allergies don't work that way, but allergy tablets have absolutely solved my problem. Which I'm thankful for because I tried a CPAP for a solid month a few years ago and I just could not get used to it, and was sleeping like crap.
Which moves us to the next two issues: liability and time. Any moment that you ask someone to revise a decision and specially with the stakes that the medical profession has that nobody has the time nor the inclination to open themselves for a mess.
Now, if you really want to be successful, you have to, before they even have a case with you, and specially before the diagnostic loop closes, to suggest the tests that the study has, since that has the biggest chances of looking at the right thing to look. Just be straight that you walked in with a theory. Doctors notice when they're being steered way faster than they notice when you're actually right. That's how you work with the systems that have a overworked mass trying their best.