I mean, it's a well known fact that LLMs do badly in the "long tail" of the distribution: anything they haven't seen many many times (e.g. relatively obscure specialist knowledge), they are definitely very likely to get wrong. The more "popular" the thing you're asking from them is, the more likely it is they'll answer correctly.
I disagree with the idea that this is basic human anatomy, basic at the level of a practitioner, likely, basic at the level of a lay person like myself? Not certain. Had you asked me "Hey Ian, at what level is the caval hiatus?" I would either tell you I had no clue or look it up.
If anything this demonstrates that LLMs aren't great with niche knowledge, which makes sense since the training corpuses are constructed from common knowledge for the most part.
This is however, a potentially great case for retrieval augmented generation.
Fair enough, It's still more specialized knowledge and I'd guess Anthropic's training corpus is light on it. If I recall correctly OpenAI included a lot of textbook knowledge.
No you're right. He's using an incredibly obscure term (caval hiatus rather than caval opening/foramen/aperture) for a mild-moderately obscure question.
Nevertheless I would wager most clinicians (except for surgeons, radiologists or cardiologists) wouldn't know the answer to this question. You learn this in first year of medical school and then forget it.
FWIW, Pi returns: "The caval hiatus is located at the level of the T8 vertebra, which is near the middle of the spine. It's a small opening in the diaphragm that allows the inferior vena cava, a major vein, to pass through. There's also a nearby opening called the esophageal hiatus, which is where the esophagus passes through the diaphragm. The caval hiatus is an important anatomical landmark, and it's also sometimes called the caval opening or the caval orifice. Is this something you're studying for a class or just out of curiosity?"
As a cardiologist I will translate this into software: saying that the name of the portion of the diaphgragm that opens to the IVC is "basic human anatomy" is like saying that "monoids in the category of endofunctors" is a basic tenet of software development.
As a radiologist who looks at thoracoabdominal anatomy for a living I completely agree. That was also my initial reaction, I don't think I've ever read "caval hiatus" used in clinical text other than maybe during medical school.
It's not even correct, anatomically it's most correctly described as the caval foramen. Hiatus refers to a slit/gap, foramen refers to a circumscribed orifice. Caval hiatus is either misremembering or an archaic text. Clinically people use aperture, foramen or opening.
Other problems with this post:
I assume the incredibly poor way of prompting is attempting to simulate a user using a chat interface? If so, are they even using the right LLaMA model (at the very least chat fine-tuned, ideally also on the open medical conversation dataset)? What GPT model are we comparing with?
Is there system context provided for the prompt? If you were deploying this you wouldn't only pass the inputed user text in the prompt. ChatGPT's interface adds this so if you're going to make a fair comparison you should be doing the same.
It's still going to favor OpenAI for this specific use case (vague and poorly phrased short prompts of esoteric trivia) as they've had much more training in this domain.
As a non-native speaker I wonder if this could still be of use because it's maybe better than using a dictionary. I think getting medical terms right is one of the hardest things to do in a hurry (like when visiting a doctor on vacation, not if you're living in an English-speaking country). You're only guessing anyway.
Can't believe I'm somehow defending bad LLM output, but here I am, even with this huge caveat,
Potentially, but no patient or physician is going to be using "caval hiatus".
If this was the intention, the author should disclose that he's intentionally using very obscure antiquated wording not "basic human anatomy". If he is instead unaware that "caval hiatus" is not accepted terminology it is further evidence supporting how obscure this fact is.
Which means nothing, you have a pre-clerkship anatomy course director using archaic and obscure terminology that is extremely rarely used (and arguably incorrect, but we don't have to debate etymology). You will never see this used clinically or in any major anatomy textbook.
As you are a medical student you should be familiar with accepted references for anatomy. In case you're not here are some:
Netter's anatomy (most commonly used clinical anatomy text): caval foramen.
Gray's anatomy: inferior vena cava opening.
eAnatomy: caval foramen.
Your other clue is in ChatGPT's response:
>The term "caval hiatus" is typically used in reference to human anatomy, specifically the anatomical structure known as the "caval opening" or "caval foramen."....
If you ask Claude the same question using 'caval foramen':
>The caval foramen, also known as the foramen of vena cava, is located at the level of T8 vertebra in the thoracic region of the vertebral column. Specifically...
Some friendly advice as you seem to be early in your medical training:
1. When an attending (in this case two) corrects you, you're probably wrong. Your next step should be to confirm your knowledge not retort with "it's what I was taught".
2. Never say a "major medical school" in Canada (where I also graduated from) which implies there are tiers. Unlike the US, all Canadian medical schools and (almost all) residency programs are excellent.
ChatGPT uses a 1.7 trillion parameter model IIRC. I wouldn't expect a 70b model to compare well. There are medical finetunes that might be better to try for this, like https://huggingface.co/wanglab/ClinicalCamel-70B
Edit/Correction: as mentioned below, free tier used by OP is in the neighborhood of 175b
I'm not sure, I've tried looking it up and couldn't find an official answer. But that site says GPT3 is 175b and GPT4 is 170 trillion. That's insane if true
30 comments
[ 1.8 ms ] story [ 141 ms ] threadIt's caval foramen, esophageal hiatus and aortic hiatus. Clinically, everyone says opening, foramen or aperture.
If anything this demonstrates that LLMs aren't great with niche knowledge, which makes sense since the training corpuses are constructed from common knowledge for the most part.
This is however, a potentially great case for retrieval augmented generation.
> I disagree with the idea that this is basic human anatomy
When I say basic, I mean something taught in second or third year anatomy or physiology courses; or first year medical school.
Nevertheless I would wager most clinicians (except for surgeons, radiologists or cardiologists) wouldn't know the answer to this question. You learn this in first year of medical school and then forget it.
I ask because using plain medical language seems to elicit the correct answer from Claude and ChatGPT3.5 (I didn’t test llama from my phone.)
It's not even correct, anatomically it's most correctly described as the caval foramen. Hiatus refers to a slit/gap, foramen refers to a circumscribed orifice. Caval hiatus is either misremembering or an archaic text. Clinically people use aperture, foramen or opening.
Other problems with this post:
I assume the incredibly poor way of prompting is attempting to simulate a user using a chat interface? If so, are they even using the right LLaMA model (at the very least chat fine-tuned, ideally also on the open medical conversation dataset)? What GPT model are we comparing with?
Is there system context provided for the prompt? If you were deploying this you wouldn't only pass the inputed user text in the prompt. ChatGPT's interface adds this so if you're going to make a fair comparison you should be doing the same.
It's still going to favor OpenAI for this specific use case (vague and poorly phrased short prompts of esoteric trivia) as they've had much more training in this domain.
Can't believe I'm somehow defending bad LLM output, but here I am, even with this huge caveat,
If this was the intention, the author should disclose that he's intentionally using very obscure antiquated wording not "basic human anatomy". If he is instead unaware that "caval hiatus" is not accepted terminology it is further evidence supporting how obscure this fact is.
As you are a medical student you should be familiar with accepted references for anatomy. In case you're not here are some:
Netter's anatomy (most commonly used clinical anatomy text): caval foramen.
Gray's anatomy: inferior vena cava opening.
eAnatomy: caval foramen.
Your other clue is in ChatGPT's response:
>The term "caval hiatus" is typically used in reference to human anatomy, specifically the anatomical structure known as the "caval opening" or "caval foramen."....
If you ask Claude the same question using 'caval foramen':
>The caval foramen, also known as the foramen of vena cava, is located at the level of T8 vertebra in the thoracic region of the vertebral column. Specifically...
Some friendly advice as you seem to be early in your medical training:
1. When an attending (in this case two) corrects you, you're probably wrong. Your next step should be to confirm your knowledge not retort with "it's what I was taught".
2. Never say a "major medical school" in Canada (where I also graduated from) which implies there are tiers. Unlike the US, all Canadian medical schools and (almost all) residency programs are excellent.
Edit/Correction: as mentioned below, free tier used by OP is in the neighborhood of 175b
[1]: https://ecoagi.ai/articles/chatgpt-parameters
Edit: and this one says 1.7, idk https://levelup.gitconnected.com/gpt-4-parameters-explained-...
Haven't used chatgpt in a bit, but looks like the free tier is GPT3 still, so 175b may be correct if OP isn't a subscriber
A 180b Falcon model very recently came out https://huggingface.co/tiiuae/falcon-180B A finetune of this might be competitive, but I haven't tried it.