"Half" here is the 10 year cumulative probability of at least one false positive if you get a screening each year.
The status quo ten-year false positive rate with the test used before 3d mammograms (digital mammography) is 56.3, and the false positive rate with 3d mammograms (aka tomosynthesis) is 49.6.
So the study is actually reporting an improvement vs the previous state of the art! It's weird that the article is written to convey that the false positive rate of the new technique is a drawback when it's actually a benefit.
(This comment has been heavily edited thanks to a correction I received below.)
Yeah, I mixed up which name applied to the new technique. To be clear, I don't know the space, I just read the study and assumed that "digital mammography" sounded more 3d-ish than "tomosythesis". I'll update my comment accordingly.
Edit: oh man. Now I see that I mixed up the direction too! What in the heck. The article was written like there was a big problem with 3d mammograms so I assumed it was performing worse on this metric than the baseline. But in fact it's better?
The title is technically accurate, but you have to take into account uninformed people's priors in order to efficiently convey information that might update their beliefs. Just my personal opinion on this issue. If you say "new technique [X] has [high] false positive rate," you're going to make people who don't know the status quo think there is a deficit in the new technique. Also, I'm not just talking about the title.
But isn't that millions of fewer false positive results over 10 years?
Assumptions:
* US Women: ~150e6
* Fraction getting annual mamograms: ~1/3 (life expectancy of 80yrs, and mamograms during 30 of those years, minus some with poor access to health care)
* Reduction in cumulative FP rate over 10 year period: 6%
150e6 x 0.33 * 0.06 = ~3e6 fewer false positive results over a 10 year period. That seems like a significant effect size to me.
My wife got a false positive a few years back, her doctor called her personally to discuss and reassure her that it was not necessarily a problem, just an area of suspicion. She wasn't all that anxious or worried about it. The worst thing she was worried about was the chance of needing a biopsy, not the chance of actual breast cancer.
She went in for a followup ultrasound scan and was cleared.
Given the survivability of early stage breast cancer versus late stage, a high false positive rate doesn't seem worse than the alternative of missing early stage breast cancer. Obviously, better imaging or other tools would be even better.
This is really the tradeoff we face in the binary classification space. Do we want false positives or false negative reduced? There are more sophisticated approaches, but if the costs in one direction are magnitudes more than in another, it becomes a clear preference.
There's also a subtle clinical communication aspect to this. I once had a doctor message me shortly after a routine test and say "I need you to come back tomorrow", and my pregnant wife was losing it until we could get through to him and find out that he just wanted to ensure I hadn't gotten an infection during the test. My wife wasn't wrong to be upset! Several hours of suffering could have been avoided with a good pamphlet.
Yep, I've got family members who are doctors and they always tell me that bedside manner is the most important part of their job. They have to go to continual learning conferences every year and the biggest part of them is usually bedside manner stuff.
Yeah, communication is key - no doubt it would have been a different story if she got a voice-mail from her doctor's office saying "We found a suspicious lump in your mammogram, we're going to need you to come in for a followup tomorrow". She probably would have had a sleepless night wondering how serious it was.
Unfortunately we treat lots of false positives, which leads to unnecessary chemotherapy and other treatments. These unnecessary treatments harm ones health so much that no study could show an increase in life expectancy by doing regular mammograms.
No one just treats a mammography or tomography positive without other follow up tests. It's not the standard of care. The next step is to get an ultrasound and then a biopsy.
No one receives chemotherapy from a screening mammogram alone, there are always further diagnostic tests before treatment. Though there are probably unneccessary biopsies.
Screening mammograms do reduce breast cancer mortality:
Annual mammographic screening at the age of 40-49 years resulted in a relative reduction in mortality, which was attenuated after 10 years. It is likely that digital mammography with two views at all screens, as practised now, could improve this further. There was no evidence of overdiagnosis in addition to that which already results from the National Programme carried out at later ages.
My wife got a false negative screening and then a few months later was diagnosed with stage 4 metastatic breast cancer. The cost of dealing with false positives seems much less than the consequences of a false negative in our experience, so I agree with your sentiment.
My wife is a survivor of stage 3 metastatic. We deal with false positive scares around once a year at this rate, but I'll take them every time over not finding something in time.
> Given the survivability of early stage breast cancer versus late stage, a high false positive rate doesn't seem worse than the alternative of missing early stage breast cancer
I stand by this sentiment wholeheartedly. 3D mammo caught a 0.6mm diameter tumor for my wife and she never even felt the lump. I felt it once and then couldn't find it again. I had to drag her to the doctors office as she didn't believe me. Oddly enough, my college years were spent in a pathology lab and I knew exactly what a tumor felt like and knew her family history and my heart sank the first time I felt it, so I was quite persistent in forcing her to see the doctor.
They rushed her surgery on the schedule and still the tumor was 3.5mm by the time they removed it. Time is absolutely of the essence. Had she waited for her annual exam or for some convincing evidence that she needed to go to the doctor, I fear it would have metastasized and I'd be a widower writing this comment.
I feel like adding, in the US, at the time, we had to fight tooth and nail for the 3D mammo to be covered by insurance and it was still pretty pricey copay. This really should be more widely available. Especially for certain demographic traits. My wife was 30ish (dense tissue), family history, known BRCA+, and should not have to fight for this technology that is the obvious best option.
The absolute worst thing is false negatives. No tests should allow those. We can live with false positive to some degree. But we should nonetheless try and minimize the false positives since high false positives might do more collective harm than no test at all. What the false positive limit is, I guess it's up to society and the medical profession to decide.
Something that should be kept in mind is that false positives should always be talked about alongside false negatives. There is no perfect test, and often in medicine there is an increase in one of those in order to decrease the other.
"A false-negative mammogram looks normal even though breast cancer is present. Overall, screening mammograms miss about 1 in 8 breast cancer"
It isn't discussed in the OP article, and I could not easily find something that differentiates false-negative rates in 2D vs 3D mammograms, but the cost of over-diagnosis might be worth it if that means less cancers go unnoticed in screenings.
Note that this is not due to bad tests, just a counter-intuitive probabilistic result happening when one tries to detect an a priori very rare condition.
It follows from a straightforward application of Bayes formula:
- Suppose a person has 1% chance of getting cancer, so p(cancer) = 0.01
- Suppose that the test has 99% sensitivity and specificity, i.e., p(positive | cancer) = p(negative | no cancer) = 0.99
- To use Bayes we first need the probability of a positive result regardless of disease status, i.e. p(positive) = p(positive | cancer) * p(cancer) + p(positive | no cancer) * p(no cancer) = 0.99 * 0.01 + 0.01 * 0.99 = 0.0198
- Then by Bayes we have p(cancer | positive) = p(positive | cancer) * p(cancer) / p(positive) = 0.99 * 0.01 / 0.0198 = 0.5
And things get worse the rarer the condition is. When p(cancer) = 0.5 the computations above give p(cancer | positive) = 0.99, and when p(cancer) = 0.001 then p(cancer | positive) = 0.09.
In other words, the rarer the condition, the more precise tests have to be.
I think there's one nuance to add: the test is based on your base rate expectation for how likely the given individual is to have the rare disease.
The more evidence you have that this person is more likely to have the disease, the more useful the test becomes. Some examples:
- if the disease is more common among people over age 65, it's more useful on people in that age group
- if the person displays more symptoms associated with the disease, and not also associated with more common diseases, then it's more useful for someone with those symptoms
- if a disease is common in an area of the world, and the person has traveled there, the test is more useful
The more factors you have, the less likely a false positive. This is why it's often better to avoid medical tests unless there's a reason to suspect someone is at elevated risk for a disease: a positive result is more likely to be a false positive.
There's a really interesting mathematical 'paradox' with false positives along the same line as the birthday paradox. It's an important one also that many doctors get wrong. Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate.
Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.
This is, incidentally, the same basic reason why so many knee-jerk policies produce systems that can never do anything but fail. When trying to find needles in hay stacks, since the needle (let's say terrorists trying to board airplanes) is so tiny relative to the hay (the millions of other people doing same), the people responsible for reacting are essentially trained only on false positives and still miss the real events when they (highly infrequently) occur.
Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
Going a level further, the claim that the systems fail is itself tough to support!
Let's say two million people have been seriously inconvenienced by airport screening. We have to ask if that's an acceptable price to pay for the benefits? But do we know the benefits? Can we know the benefits? We actually don't know if the policy deterred anything, or even if someone who wanted to do something bad had their knife or liquid explosive or whatever confiscated and just abandoned the plan. It's purely a hypothetical so you're going to support the view you want: "we don't need to do all this because we don't actually have a big threat" or "we have successfully prevented any more attacks."
After all, we often see discussions here of foolish companies that underfund their security teams because "why are we paying so much for this when nobody has cyberattacked us?" ;)
But tests are not generally described in terms of accuracy. They're described in terms of sensitivity and specificity, because those quantities are both "accuracy" but they're not closely related to each other.
A test that is always negative has 0% sensitivity and 100% specificity. Nobody thinks that sounds good.
The point isn’t that you can’t create a single number that can represent the effectiveness, just that you can’t get a single number that the public will understand. People have enough trouble with simple percentages.
It looks like an F1 score is calculated from precision and recall. Sensitivity and recall are the same thing, but precision and specificity are different things.
Precision is not a good way to describe a test because, as I mentioned elsewhere, it relies on information about the population being tested.
My favorite is the 50% test case. If you flip a coin as a test and ascribe heads to true, 50% of the time it will describe what's happening accurately.
So if you want to know "Has the Sun exploded?" you can flip a coin, assign heads to "Yes the Sun has exploded", and 50% of the time it will be correct.
You flip a coin, you get heads. You decide to flip the coin again, and you get heads again! What are the odds the Sun has exploded?
Yeah this is why in practice the followup to a mammogram is never going to be another mammogram. It's a different test which is more invasive but also produces far less false positives: taking a tissue sample.
> Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.
This conclusion assumes that running the test twice is equivalent to running two independent tests for the same condition. For many real tests, getting a false positive once predicts a higher chance of getting another false positive in a repeated test.
Also depends a lot on the nature of the screening and what is being screened for.
Prostate cancer being somewhat of an outlier as at least to my understanding, essentially all men will end up with some kind of tumor or another if they live long enough, but some large fraction of those won't become something that requires medical intervention before the individual would die from any number of other age-related causes. Outside my area of expertise but the relative size of the tissue that produces various breast cancers as well as the proximity to lymph nodes change the calculus on that a bit.
But I would definitely expect overall screening protocols to evolve as methodologies are developed that can potentially exclude the false positives from another metholodgy. While test A + A may still produce false positives due the nature of the test failing the same way, if we have A + B + C that are all relatively low-impact and not horribly expensive, that may end up being a fairly good program if some combination of A, B, and C generally exclude the parameters that the other produces a false positive for.
Not 100% sure. But I think the proximity means that its easy for the cancer cell to spread to the lymph nodes, then they’ll get fast tracked around the body, spreading cancer everywhere. Once that happens you’re basically dead, all targeted forms of cancer treatment don’t work anymore, because the target area is basically your entire body.
Again outside my area of expertise here, but to my understanding one of the significant risks to breast cancer is that if it spreads into the lymphatic tissue, it can subsequently metastasize across the across multiple other organs while compromising the lymphatic immune response that would normally help limit such a spread.
Prevalence of prostate cancer increases with age >70. Of confirmed cases ~2/3 are low grade, require only "active surveillance". Around 20% are intermediate grade tumors and ~10% are high-grade, aggressive cancer. The basic screening tool is the high-specificity PSA lab test, with biopsy considered the definitive diagnostic procedure.
Except for the highly malignant variety prostate cancer is generally an "indolent" tumor, that is slow-growing. So in the majority of cases chances are good that the patient would die of something else before the tumor becomes the cause of death. However prostate cancer is a leading cause of cancer death in men. In the US ~30k/yr men die of prostate cancer, so it's not a trivial condition.
Yes exactly. In fact mammograms are already double-read and agreement is quite high, so this is definitely one of those types of tests. There are benign masses that are radiologically impossible to distinguish until you get a biopsy.
Yes, and this is how the standard of care treats things. You never proceed from positive mammo screen to treatment. You always first do some other steps like ultrasound or biopsy.
In the general case, this is the point where medicine arrive at both a medical and ethical problem. The biopsy and further tests both involve some non-insignificant risk to the patient, as does overtreatment. If 99% or even higher number of patients are false positive, and the biopsy has a risk of causing significant negative sides, at what point is testing doing more harm that good?
This is the general argument against testing for prostate cancer in non-symptomatic patients. If you only test patients that has symptoms then the rate of false positives goes down. The downside is that you only catch late stage patients where treatment might be too late. The alternative (which is used for mammography) is that for every person saved, an order of magnitude more will be permanent harmed by unnecessary surgery.
Yes it's not trivial to balance these things and this is why there have been randomized control trials that look at long term end points where both the costs of false negatives and false positives come into play.
It's also why breast cancer screening is not recommended for everyone, just women over 50.
For women over 50 there has been a decades long sincere effort by the medical community to take this all into account and the consensus is that screening makes sense in that population.
Yes, but every positive mammogram means a percentage of people are going to get treatment, including surgery and radiation/chemotherapy, who never would have developed cancer -- and screening more women, especially those who are likely to generate false positives (younger women) is a very big problem in preventative medicine, especially when introducing a new modality like "3d" mammograms. Orac / David Gorski, MD (Surgical Oncologist (Breast)) has blogged extensive. A good if old starting place is https://sciencebasedmedicine.org/a-holiday-round-in-the-mamm...
> For many real tests, getting a false positive once predicts a higher chance of getting another false positive in a repeated test.
Almost all of them, in fact. If improving accuracy was as easy as repeating the test, the doctors would just be told that the correct procedure for this test is to take three readings.
You can expect that this is untrue only if the tests are especially expensive or invasive.
I believe repeating the test is standard procedure for many tests - if the first one is positive. However the repetitions are usually temporally separated (in some cases intentionally, in others because the test takes time.) This is effective in those cases because although there are a lot of temporary causes for a false positive to spike, there are fewer where it's persistent.
You misunderstood the scenario: 10,000 people would receive a test. Out of those, ~100 would be expected to test positive. Only those ~100 people would receive a second test.
Thus, the population for tests 1..10,000 is different than the population for tests 10,001..10,100.
In this example? Because we know that one is a true positive. We tested 100 people with a false positive, and 1 with a true positive.
If we test again, and the test is "fair", then of these 101 people, we will retest 99% (or 99) true negative, 1% (or 1) false positive, and 1 true positive. So two positives, one false positive and one true positive. 1 out of 2 is 50%.
(Tho, there is a chance also that the true positive might come back as a false negative, then 1 out of 1 or 100% of positive results would be false positive)
The second population contains 100 people without the disease. Of them we expect one person to have a false positive. In addition the person who actually has the disease will test positive. So on average 2 people will test positive, one of whom has the disease and one of whom doesn't. If you are one of those two people you have a 50% chance of your (positive) diagnosis being false.
There is a 1/10000 (the OP said 0.01%) chance of having the disease. That is, in 10000 people, one person will have the disease. But also, if you test everyone with a 99% accurate test, 100 people will test positive. If you test them again, 1 person will get another positive test. So, at the end of the day, your test will identify the one person with the disease with a 99% certainty (or 98.01 if you require two tests to be positive). But also identify another individual as being doubly positive. So, after two rounds of tests, 50% of the marked individuals truly have the disease.
The silly thing with this demonstration is that accuracy is poorly defined. Tests are normally conditional on the actual presence of the disease or not.
It forgets to consider false negatives, which will not really change much for this example as the accuracy at 99% is pretty high and the incidence is pretty low, and false negatives thus would not really change much.
However, when it comes to mammograms, where the false negative rate is up to 12.5% and the false positive rate is up to 50%, and the incidence of breast cancer in women (12.5% over a lifetime) is rather high, false negatives would play more of a role.
Well, if we assume there is a 0% chance of false negatives (which seems implicit in the post, even though the definition of "test accuracy" doesn't make that assumption), then we don't need to consider it.
If we are not making that assumption, then the 50% chance claim at the end is incorrect, since there's a chance that both of the two positive results are false positives and there is a false negative somewhere else in the remaining 99 (out of 101) people.
The 50% chance of it being a false positive (conditional on two positive results being observed) is actually a lower bound, and will increase as a function of the false negative rate.
No, they didn't say that was the definition in their one post.
Their math shows that those are the same number, and they didn't elaborate on definitions.
The least misleading definition of accuracy is that accuracy = 1-false_positive_rate = 1-false_negative_rate. So unless told otherwise, I will use that definition.
Yes they did. They say it twice. I have italicised both places where they say it.
"Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100"
Then they conclude by saying:
"So there will be 100 false positives, and 1 real positive."
The fact there is one real positive necessarily implies they are assuming the false negative rate is exactly 0, given that the disease prevalence is in 0.01 percent of the population.
> The fact there is one real positive necessarily implies they are assuming the false negative rate is exactly 0, given that the disease prevalence is in 0.01 percent of the population.
Or that they rounded .99 people to 1 person.
And given that they did round the false positives, I don't see why that's a strange idea.
First, I'm assuming that top comment meant the false-positive rate was 1%, even though it was poorly stated as "accuracy." If you're commenting on this original ambiguity, then the whole thing is ambiguous.
But if we accept a 1% false-positive rate, then it makes no difference at all what the false-negative rate is, once we have a positive test.
Out of a 100,000 tests, on average 1000 will be falsely positive. That's what a 1% false-positive means. It makes no difference how many are truly positive, how many are truly negative, and how many are falsely negative. The definition of 1% false-positive means that, out of 100,000 tests, on average 1000 will show a positive test when the condition isn't actually there.
So if 100,000 people each took two tests, on average 10 will have two false-positive tests. Again, this is independent of the number of false-negatives, and is simply the definition of false-positive. (Some tests may have correlated false-positives, i.e. you're more likely to get two in a row, but that's not standard and not part of the definition.)
So if you have a disease whose incidence is 10 people out of 100,000, and you took two tests which are both positive, you are equally-likely to be in the group of 10 people who really have the disease, and the group of 10 people sitting there with two positive tests who don't have the disease.
You seem to be saying that the percentages will be affected by whether any of the 10 out of 100,000 who really do have the disease got a negative test. But it doesn't. 1% false-positive means that, on average, 10 people who don't have the disease will get two false positive tests. It doesn't matter what the people who really have it get.
Positive predictive value is the likelihood that, if you have gotten a positive test result, you actually have the disease. It’s calculated as TP/TP+FP.
TP, ie true positives, is a function of false negatives.
Here's a common sense explanation. Assume the false negative rate is 100% and so you have zero true positives. Then every positive test result will be a false positive, and therefore a positive test result shows that you don't have the disease.
On the other hand, assume the false negative rate is 0. Then a positive test result will mean that you probably do have the disease as long as the false positive rate isn't too high.
Remember your post I was replying to was all about the probability that you actually have the disease conditional on a positive test result. That's the question I am addressing. For this question, you must consider the false negative rate.
I'm going to join you down at the bottom here. I agree with you it isn't quite right, and I _think_ the reason comes from a couple sleights of hand. One: healthy people don't go in testing for the presense of a rare disease. That is, if you are administering this test to 10,000 people and only one truly has the disease, then any statistic that you back out of this trial is well below any significance. Two: the colloquial use of 'accuracy' is not the statistical use of 'accuracy'. This post uses the former for our 'intuition' and the latter to demonstrate that our intuition is wrong.
> healthy people don't go in testing for the presense of a rare disease.
Well that’s precisely the point. There’s frequently the idea raised among lay people that it would be beneficial to test or screen for all kinds of rare or even just uncommon diseases that have high mortality in the general populace. This is the counter argument.
And many tests, likely the more invasive ones used to confirm these initial tests, carry risks. Even something as "routine" as a colonoscopy carries risks.
Funnily enough, accuracy is a somewhat inaccurate term when it comes to medical testing. What we are interested in are sensitivity, specificity [0] and prevalence of the disease in the population being tested. With those three we can get positive prediction value (PPV) and negative prediction value (NPV) [1]. But PPV and NPV always depend on prevalence! That's why we don't want to test healthy people for all kinds of exotic diseases.
However, despite the high false positive rate, breast cancer screening is a net beneficial procedure (though not without controversy) in that when applied in a population, long term outcomes like mortality improve.
(edit yes of course cancer is not beneficial thank you)
> However, despite the high false positive rate, breast cancer screening is a net beneficial procedure (though not without controversy) in that when applied in a population, long term outcomes like mortality improve.
I've read that's also a statistically tricky statement. IIRC, they measure survival time from when the cancer was detected. So early detection by definition improves the survival stats even when there's no meaningful treatment.
If I've read about this, I'm sure medical practitioners are well aware of it, but even so, I don't think it's easy to correct for it. How do you determine when a late-detected cancer actually started?
Thanks! Paraphrasing to check my understanding: you don't use the stat I mentioned, instead measuring whether the patients ultimately die from cancer or something else. You use a control, so after like detection method some patients get treated and some don't. But not treating may be unethical, so you use a "synthetic control", which I had to look up. Basically an invented group of not-treated patients based on statistics from other populations. [0] It sounds interesting but tricky to get right. "A healthy degree of scepticism on the use of synthetic controls is thus expected from the scientific community."
[edit: or, looking at the NLST you mentioned [1], maybe "treatment" doesn't always mean what I think of as treatment. They are actually comparing two different detection methods, and they aren't using a synthetic control.]
Do they ever use the "survival time after detection" stat I mentioned anymore, or has it been (/ should it be) abandoned?
Yes, NLST was actually a real randomized control trial for lung cancer screening, done before it was known whether or not it was helpful. The trial eventually stopped once they had enough outcome based evidence.
The statistic you mentioned is not used in assessing the effectiveness of screening. It does closely relate to what is though to be the causal mechanism that leads to effectiveness.
You can still try to estimate this using something called QALYs which are life years adjusted for quality of life.
A breast cancer scare, is certainly harmful, but it's important to remember that it is going to be a short episode as it becomes clear that the tumor is not growing. So you have maybe a few months of scariness.
On the other hand cancer is brutal, and death is permanent. Years of life cut short. The years leading up that are extremely sad. The impacts of death extend beyond the person to their family and friends.
When people try their best to measure these things on balance, breast cancer screening is overall a good intervention.
Just to expand/add nuance to your comment about controversy:
It's pretty uncontroversial that cancer screening for women above 50 years old or women with high risk (e.g. familial or genetic evidence or risk, history of cancer/treatment) is a net beneficial procedure.
For asymptomatic, low-risk women aged 40-49, the recommendations are a bit more mixed. While there is some evidence that overall mortality does improve, there are also considerable harms (physical, emotional, financial) associated with false positives and "overdiagnosis" (that is, identifying and treating low-grade non-invasive lesions that are unlikely to progress to breast cancer).
Checking the Wikipedia article, for every woman saved there is 10 women who will get unnecessary surgery and 200 that will get to go through extended testing.
It would be interesting to know the mortality rate of unnecessary breast surgery. I would hope the number is less than 10%, but I suspect it is not an insignificant number of people. Net beneficial, but definitively controversial.
PPV is absolutely something that's tracked. In fact, if you read the instructions for use (equivalent to "package insert") for lab diagnostics, you'll find the manufacturer's truth tables and can calculate the PPV if it's not in there. FDA keeps them online, so you don't even need to buy the diagnostic. Look the up.
I'm not sure 99% is considered the gold standard you think it is. Sure, a lot of our tests can't even do that well, but we know that 99% isn't a lot for large populations. We even have jargon like "five nines", "six sigma" around the fact that 99% isn't a lot.
> So why isn't this ratio part of the major test criteria?
Because it has nothing to do with the test. Sensitivity and specificity are facts about the test, and conceptually they will remain whatever they are if you start testing a different population.
Whether doing a test has any value is a fact about the population ("olympic athletes are never anemic, so there's no point in testing them for anemia") that has nothing to do with the test. Why would you evaluate the test based on that?
A good example is screening men for breast cancer. It's possible for men to get breast cancer, but it's so vanishingly rare that the overwhelming majority of positive results will be false.
Note that the basic Bayesian analysis relies on the samples being independent, so if the same test is re-run on the same patient, equipment, process and facility, the error is likely correlated to the previous test, so the chance that it's still a false positive is even higher than an independent analysis would suggest.
Imo you are exaggerating a bit: What you are describing is specificity vs sensitivity [1] which I am fairly certain doctors get educated about pretty early on in their studies. Once you know about that your examples shouldn't be surprising to anyone.
Any younger doctor has most certainly had sensitivity vs specificity hammered into them in med school.
I remember reading studies about doctors not understanding the stats years ago, but my wife assures me that medical education has over corrected in the last decade or 2.
> Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
The correct answer is that the question doesn't give enough information to know.
I mean, a test that always said no would be 99.99% accurate. A mere 99% accuracy could be no false negatives and 0.99% of all tests false positives, in which case the answer is that a positive result is 1% likely to be correct, and 99% likely to be a false positive. Or it could be, on the other extreme, 0.01% false negative and 0.98% of all tests false positives, in which case the chance that a positive result represents an actual case would be 0%.
Presumably, though, approval processes for medical tests will constrain the actual distribution of results to be such that, a patient with positive result is not more likely to be clear of the condition than one with a negative result, so it won't be as bad as the latter extreme, but it's worth noting that your assumption (essentially that the false/true ratio is equal for negative and positive results) is not certain at all.
Nitpick here. There's nothing wrong with the math of Bayesian probability, but the setup is usually something like an STD test, and in those cases the prior is wrong.
Our prior probability shouldn't be "0.01% (1 in 10,000) of the population is infected," we really want % of infection for the population that get tested.
This is really obvious with disease tests because people don't typically get tests without reasons (symptoms, contact with infected persons, etc.). The math as-is would only work if we're sampling people randomly.
Who defines “99% accurate”? Couldn’t it mean zero false positives and 1/100 positives missed?
Couldn’t it also alternatively be defined as 99% accurate for a given person?
Maybe there are reasons/conventions it is always as you originally describe, but if that’s the case wouldn’t it be more about poor communication to the public than a paradox?
I think it’s correct that “99% accurate” here means “99% true positive and 1% false positive”. Agreed that “accurate” is ironically an awfully imprecise word to use.
As for why it is the case, I presume it is just the usual focus on counting your hits and not on your misses.
This is even more terrifying when one considers the possibility that the result of the second test is not independent from the result of the first. That is, some trait within the population increases the probability of a false positive (and this is likely in any heterogeneous population).
Now the conditional probability of infection, given TWO positive tests is still far below 50%.
This is why a second test of a different type, even if it is less accurate, if it is more likely to be independent, is much preferred!
I feel like the general use of accuracy assumes no false positives. When I hear "accuracy" in general conversation I assume it's 99% percent chance of a true positive and a 1% chance of false negative and ignore chance of a false positive and true negative when considering accuracy. Of course everyone already mentioned the various ratios that scientifically actually make up accuracy.
Ignoring the difference between recall and precision - off the bat, it seems weird to rely on a test with some 1% error rate for a disease that has a 0.01% occurrence rate. Without doing the math, it seems like it would be mostly “wrong” in some sense (
Can someone describe the core reason the second test is a false positive is not 1%? Why does the outside world (number of people being tested) affect my own test error probability?
It's the same reason that the first test doesn't get you 1%, but the testing pool has changed. Instead of testing 10000 people without and 1 person with, you're testing 100 people without (and the same person with).
It's a Bayesian update. Every (independent) positive result increases your confidence of having the condition by 100x. At first you have no information so you assume the base rate, 1:10000. At the first test, that goes to 100:10000. Second test gets you to 10000:10000 or even odds. Third positive result finally gets you to 1000000:10000, or a 99% confidence of having the condition.
A maths teacher once told me that if I'm struggling to work through a logical problem in my head, I should look at the extreme numbers, because that often gives a clearer picture of what's happening. This has helped me a lot, and is how I would approach this:
Let's say we've got a test that:
* If you've got cancer, detects it 100% of the time (to make the maths easy)
* If you haven't got cancer, gives you a (true) negative 99% of the time, and a (false) positive 1% of the time.
For starters, consider a population of 100 people, all of whom have cancer. Our magical test always finds these cases, so we have an 100% positive rate, and a 0% false positive rate. (None of these can be false positives because we know they all really do have cancer.)
Now consider a group of 200 people, 50% of whom have cancer, and 50% of whom don't. The group who have cancer will still all get detected, but if the 100 people who don't have cancer, one of them will be marked as having cancer even though they don't. In total, that's 101 people who will get a positive result. So our overall positivity rate is 101/200 (~50%), but our false positivity rate is the number of false positives divided by the number of all positives, so 1/101 (~1%)
Now consider the case where we have 101 people, of whom only one has cancer. If we run our test a third time, we of course detect the person with cancer, but of the remaining 100 who don't have cancer, we will detect a second person. That's two positive cases in 101 people, so our positivity rate is ~2%. However, of those two positive cases we know only one really has cancer - the false positivity rate is 1/2, or 50%.
So that's three situations where the only thing we've varied is the proportion of the population who have the condition we're scanning for. (Technically we also changed how big the population was in the first place but that was just to make the maths easier - it doesn't affect the results at all.) However, we get a different rate of false positives in every test, because the number of false positives is dependent on the ratio between the number of people with cancer and the number of people without.
It's also worth noting that this obviously doesn't just apply to cancer, but also other situations. Most relevant to us in the last couple of years is probably Covid tests - you might have seen over the summers some scepticism about how valuable it is to do lots and lots of tests, and that's because during those times, the number of affected people was lower, and so these tests were more likely to produce less useful results. However, in winter, even relatively poor tests become more useful, because there are more true positives, and therefore a lower rate of false positives.
First, it assumes that a prior false positive does not increase the likelihood of a false positive on testing again (that is, a retest's result is independent from the first test's result).
Given that, from the first test, we have 100 people who were false positives, and 1 person who is truly positive. We would expect that 1 truly positive person to be positive again (we have a 99% chance of it from the test, after all), and 1 of the 100 who were false positive before to also test positive this second time (that 1% inaccuracy rate). Given 1 true positive, and 1 false positive, in that sample, you would expect a positive rate to mean you have a 50% chance of being the true positive.
If we did not have information from the first test, yes, it would be ~1% chance of a false positive. We have nearly a 0% chance of having a true positive, so of the 101 people, and a 99% accurate test, we would expect one to come out positive, despite not having the disease. But, because we have that first test, from a much larger sample size, we likely have one person who is a true positive, and so could expect two people to test positive, one with the disease, and one without (the 1/100 of the remaining), giving us a 50% chance a positive result = has the disease.
Wow, breaking down the percentages like that with a real example was eye opening to me. If I got a positive on a 99% accurate test I'd be convinced I was dying. This really puts things into perspective...thanks.
I'm still pissed that I have to get a pap smear every couple years in spite of being with the same man since I turned 18. Doctors won't prescribe birth control without one. I had a lovely biopsy of cervical tissue a couple years ago. Again, monogamous relationship with a man that started when we were both virgins. But I got tissue ripped out of my most sensitive region anyway.
Why am I forced to submit to these screening tests for issues I'm not at risk for in order to receive tangentially related medical care?
If your gyno won't prescribe BC without a pap smear, look to get a new doctor. Insurance isn't mandating it; it's just the doc deciding they know better than you.
I may end up going through planned parenthood if that improves things. I really, really hate speculums up my cooch and have no predispositions to any issues in that area and there's absolutely no reason for me to be required to have a yearly exam to get my BC.
I wonder how much difference there really is with the older tech based on looking at outcomes. For standard surveillance, does the new 3D dental xray actually benefit me? Granted the cost is low (<10% normal yearly exposure), but I still wonder.
I remember almost fainting when my doctor showed me the MRI scans of my brain. It doesn’t bother me to see someone else’s scan, but I just couldn’t stomach looking at mine at all, seeing my optic nerve and my eyeballs and everything.
And I got my MRI brain scan and I had absolutely zero emotional reaction to it, except for curiosity. But then again I'm a schizoid so that was very likely anyway...
Well according to the article in this case 3D imaging is reducing false positives by about 6% which seems like an actual benefit. Would it be a benefit if it is easier to maintain or easier for the doctor to read?
It will take a long time to sort this out. I think that eventually, the ability for AI to review 3D in greater detail will improve on false positives and false negatives.
For CT vs. XRay there have already been 2 long term randomized controlled trials that look at outcomes for lung cancer screening. PLCO, the XRay study, was not effective in improving long term outcomes like mortality. NLST, the CT study, showed a 20% reduction in mortality. With more modern techniques that actually look at the tumor in 3D NELSON in Europe has seen even better mortality reduction.
DBT is newer so A) Radiologists are not as good at reading it yet B) the long term evidence hasn't accumulated yet, but I am definitely optimistic that it will present a big improvement over the previous state of the art.
As mentioned elsewhere in the thread, this false positive number is already better than the existing state of the art for 2D.
Kind of a non sequitur but anyone else working on this stuff? I am writing a mammo rendering engine for the browser and it would be nice to have people to talk shop with. This shit is hard.
This is all certainly true but it's important to look at the bigger picture. What is the overall impact of catching cancer earlier and saving lives? That is enormous as well. One way this is sometimes talked about in public health literature is by looking at the QALYs of an intervention. On the whole, breast cancer screening does seem to be an effective intervention
That's why it is so important medical doctors comminicate this right. Unfortunately, most of them have a false understanding of the probability of cancer based on a positive (other than B5) mammography. But this is a solveable problem.
The "4B$" is a rather useless number. The same people would probably argue it's much cheaper to go to your local shaman instead of a medical doctor.
It's good to be publishing stats like this so that the psychological harm of false positives is lower. It's scary to get a false positive.
However, in my opinion, there has been a systemic resistance to screening because of statistics like this, and that is misguided. This is not the right number to use for determining whether or not screening is useful.
A 3D mammogram will have false positives, yes. And some women will need to either get an uneccesary ultrasound (annoying, but not harmful) or worse, an uneccessary tissue sample (biopsy). This type of biopsy is done with a needle and is certainly uncomfortable, but carries little to no risk. This is all true. But it needs to be balanced against the fact that catching cancer early gives you an immensely better chance of surviving.
The right numbers to look at are more like:
How much does a screening program reduce overall mortality rates? How much does the screening program reduce breast cancer mortality rate? What are the impacts of this in terms of QALY (quality adjusted life years)? What are the impacts of the false positives in quality adjusted life years?
I believe on balance the answer to these questions is that screening for breast cancer is on the whole beneficial. Here is one example study
https://pubmed.ncbi.nlm.nih.gov/31098856/
Now the question is, how will 3D affect these questions? That is as of yet unanswered as far as I know but we shouldn't let numbers like this influence the conversation prematurely.
You're totally right. However, the study you link does not discuss mortality while taking overdiagnosis into account from what I can tell. Other studies (first one I found: https://pubs.rsna.org/doi/full/10.1148/radiol.11110210 ) can't confirm a significant reduction in mortality.
Yeah I just linked that study because it was looking at QALY which is an aggregate metric so it should in some sense take into account the effects of overdiagnosis even if overdiagnosis is not explicitly handled.
Yes, the problem is mostly one of presentation. To most people a positive test means it is likely they have it, which is often a statistically invalid conclusion.
We should instead present it as 'you currently have breast cancer with 13% probability' (population avg with no tests), a positive test means you have it with 25% probability, a negative test with 3% probability.
That makes the test a lot less scary, after all you're only gaining 10% certainty either direction.
I don't believe we should present it as a statistics. Most people would not understand or feel comforted by that explanation. 'you currently have breast cancer' is all the person will see and the numbers and implications of will be lost on them.
I feel like this is where bedside manor comes into play and the human component of being a physician. Any doctor ordering a 3D mammogram, then receiving a positive result, should communicate to their patient that imaging is just a screening tool and not diagnostic. Many doctors will not even use the word "Positive" in the test result conversation. They'll use vague wording like, "we did happen to see something we'd like to investigate further to be sure, it's really hard to see exactly what's going on in your body on a camera so while I was hoping to avoid it I do believe a biopsy should be the next step". Yes, a biopsy is now warranted, but the patient should try to keep up their spirits high and not stress to much at this point. The biopsy will tell more information about what is going on inside the body and we can come up with a plan for whatever that is once we know more.
It's natural/unavoidable for the patient to be worried/anxious but should not yet be scared and that is best conveyed in a human interaction with a caring physician.
I looked into this recently. From what I can tell, no study could show a reduction in deaths from doing mammograms. This is because they detect many cancers that would go away on their own thus unnecessarily exposing you to chemotherapy which brings other health risks. A good starting point on this topic is this video: https://youtu.be/_sg14En-Z7A
I was doubtful about the validity of that video at first, but everything he says is backed up by studies.
"Long-term follow-up of the UK Age RCT"
"There was a statistically significant 25% reduction in mortality from breast cancers diagnosed during the intervention phase at 10 years' follow-up"
This is specifically for an annual screening of women 40-49yrs. The reason why this article was written is because there is a scientific debate about whether the advantages outweigh the disadvantages in this age group. I don't think this article really settles this discussion. When you look at the CI you'll see it's rather big and the upper bound is only slightly below 1.
But this is about screening of rather young women. Most established screening programs start at an age of 50. Still young. :-)
Cancer usually doesn't go away just like that. There is a chance in situ tumours or benigne tumours "go away" and there is some discussion about what you should do with this type of tumour. Not every tumour is cancer.
Anyway, even with cancer you can of course bet on dying of another cause before the cancer gets you and thus ignore medical therapy. Some people win this bet ... somehow.
Here is a sample bayes rule calculcator made in my favorite web tool of all time, instacalc. This feels like the right way to do the math here. Random numbers provided, fill in your own as you deem appropriate.
This isn't quite as big of a problem as they're portraying and there is a lot of misinformation in these comments. For context I'm a radiologist at one of the academic centres involved in the ongoing TMIST (3D mammography) trial.
For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis.
Every concern that has been raised in these comments is factored into our reporting system. All radiologists use the BI-RADS reporting system which is internationally validated and expresses probabilities of malignancy to guide further management based on specific imaging. features that have been validated against pathologic data from excised surgical specimens and biopsies. This was done to homogenize reporting practices.
For example a BI-RADS 5 lesion, which is based on specific imaging features not a gestalt/expert opinion, denotes a 95% probability of malignancy and would generally result urgent biopsy + surgical consultation. A BI-RADS 3 lesion has a 2% chance of malignancy and would generally be followed with serial imaging for a specified interval.
BI-RADS 4 is the middle, this is broken down into A/B/C but essentially anything with 3%+ probability goes for biopsy. That means we expect up to 97% of the lesions we biopsy will have been false positives.
This is intentional and desired, this is because the harms of missing breast cancer is horrible, early detection results in treatment with lumpectomy + radiation vs advanced breast cancer needing systemic therapies and radical resections.
Second, what happens when there is an abnormality on a screening mammogram is not straight to biopsy. We use additional mammography views and ultrasound to help us find out what's going on before making that decision. We also have MRI to troubleshoot and increasingly for screening, certainly for high risk and extremely dense breasts.
Tomosynthesis or 3D mammography is not a cure-all for false positives, it's also not intended for dense breasts. The point of Tomosynthesis is to reduce callbacks for overlapping breast tissue which can sometimes look like a real cancer, and to increase sensitivity for architectural distortion (an occult or infiltrative cancer distorting the fibroglandular architecture) which can be really hard to pick up on 2D.
It works really great for that. Patients with extremely dense breasts should really be getting screening MRIs (US as a lesser alternative) in today's age and this is happening with increasing frequently.
For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis.
I totally get this and it's technical terminology.
I think my takeaway from trying to explain why this isn't a big deal on this thread is that calling a tomo or mammo "positive" is just a recipe for confusion.
I understand why it's done this way historically, but the idea of a mammo or tomo being "positive for cancer" makes no sense since there will almost always be either an ultrasound a biopsy or something else before a diagnosis is made. It's a test for whether a more invasive/expensive test is worth it, not a test for cancer.
Curious what your opinion is on AI for DBT? I think there is great long term potential here, even more so than the potential that we have seen so far in mammo (such as https://www.nature.com/articles/s41586-019-1799-6), because it's easier for AI to thoroughly look at things in 3d and spot new patterns that are not obvious to human eyes.
AI in medical imaging has been very limited and ineffective thus far.
My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.
There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).
AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity.
What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same).
I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.
AI in medical imaging has been very limited and ineffective thus far.
A) I think it's too early to make a call here. The first big paper for AI in medical imaging with modern methods came out in 2016, and we are just now getting large scale prospective studies where algorithms outperform specialists:
https://www.thelancet.com/journals/landig/article/PIIS2589-7...
My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.
B) Totally agree that old CAD has not been effective, and this has created a huge trust gap for the new generation of technology. The new generation of technology using deep learning is also easier for anyone to create so it will take some time to separate the good from the bad.
There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).
C) This claim seems overly broad. If we can improve sensitivity it can certainly be a good thing. That being said, improving specificity in an assisted read scenario seems almost impossible in my experience, because if a doctor is convinced something is cancer, no output from an algorithm is going to get them to downgrade. We did an analysis on exactly what was going on when our algorithm would give a lower, non-recall score vs. a panel of radiologists, and we found that the most common situation was missing that a lesion was a scar. https://www.nature.com/articles/s41591-019-0447-x
It turns out that radiologists will in general not look at a lesion from all different possible reconstructions in 3D, because it's too time consuming. But sometimes there is only one reconstruction/view where the scarring is obvious. AI obviously does not have this issue, so in principle it is certainly capable of improving on specificity and this has been seen in many peer reviewed retrospective studies.
As I mentioned, the issue is how to make this happen in clinical practice where assisted read is the only option that will be tolerated for deployment for at least the next 5-10 years. In that time period, I think the only thing we can really hope for is a boost to sensitivity without impacting specificity. I believe eventually enough strong evidence (prospective observational multi site studies) will accumulate, and the algorithms will improve enough that for something like mammography, at least a portion of the reads can be done without a radiologist. This is probably the only way we will see improvements to specificity AND sensitivity in my opinion.
Increased sensitivity (I.e. in CTPA) often leads to over diagnosis and bad outcomes.
We don’t routinely look at all three planes because most imaging and size criteria were developed and validated on axial sections (I.e. RECIST). Research done on volumetric measurements is in its infancy and of questionable added benefit.
Show me an example where sensitivity is a problem in modern day radiology and I might buy this argument.
Otherwise this is a solution in search of a problem
One thing engineers often don’t appreciate about medical interpretation is that you may THINK you’ve designed a better wheel but unless you can show better OUTCOMES (not increased detection) this is probably irrelevant.
"The findings of this study suggest that digital breast tomosynthesis is associated with a lower cumulative probability of false-positive results compared with digital mammography"
Which is an important finding since today such screenings rely on digital mammography. So, tomography is good.
For whatever reason, the cited article turns this upside down into:
"Half of all women getting 3D mammograms will experience a false positive over a decade of annual screening"
Which suggests that tomography performs really bad.
If you do a mammography screening every two years, you do 5 mammographies in 10 years. With a false positive, you do 5 and an MRI. So what. Doctors should always tell their patients that a positive screening mammography doesn't necessarily imply they actually have cancer.
I am always (jadedly) suspicious with articles on 'false positives' within the medical field.
Sure, statistics say that there are false positives. Personally, I rather have three false positives, than one false negative. Taken to a bit of an extreme, the outcome of first is money spent and stress, the outcome of the later is death. I am not a statistics to myself.
I am sorry for the inconvenience to the medical field by having to re-test, and certainly feel for the insurance company to have to pay for a second, or third opinion. I care, I really do. Just, not that much.
That said, it took me decades to recognize the need and add sympathy into my calculations.
Of course. It's a high sensitivity test. It's meant to screen patients for more specific tests such as biopsies. You want it to be positive for anything that even remotely resembles breast cancer, the result can be confirmed by additional testing.
Seems to me like the article is mostly pointing out the stress and out of pocket costs that come with all these false positives.
An ultrasound is not intrusive at all, a biopsy however is.
If I was a conspiracy theorist I'd say that a high amount of false positives are in fact an economic incentive for the medical community.
And anyone having seen the hell metastatic breast cancer is, up close and personal, will insist that 1000 false positives is worth it to catch one case early on.
Cancer awareness spam combined with obsessive screening and mild hostility towards women who do not frequently line up for ovarian/breast cancer testing (I don't see men being targeted nearly as much as women) is probably less about saving lives and more about making money imho.
A biopsy is not at all harmless neither in the breast nor vagina. Do a quick internet search on prostate biopsy to see some gruesome numbers.
All are pretty painful, prone to infection and a high chance of side effects.
Some pretty severe, and they are not astronomically rare. The severe ones are in the 1% range.
The fear of the brutality that is metastatic cancer is enough for few object to being routinely subjected and even coerced into this pretty dangerous and intrusive procedure.
Those who do object are generally dismissed or even ridiculed.
There is even a radiologist in the commentary here who sees no issue with the high rates of false positives as a potentially dangerous biopsy will quickly clear up the cancer scare...
For vaginal scraping which is actually recommended as a screening measure for my age group
2-4 out of 100 women will have side effects.
About 1 out of 100 will have severe side effects.
Anecdotally. I have even had one doctor ( female if that matters ) tell me that after 50 it is probably best to have both my breasts and ovaries removed.
As a preventative measure.
Medical doctors tend to terrify me. I really do avoid them unless I have something really concrete to show them that needs fixing and is not up for interpretation.
EDIT: typos
Every person has to decide for themselves whether he/she wants to do a screening examination. It's not for everyone and it is okay to say no. But I think you should try to find peace with medical doctors. Most of them just try to be helpful.
168 comments
[ 3.7 ms ] story [ 222 ms ] thread"Half" here is the 10 year cumulative probability of at least one false positive if you get a screening each year.
The status quo ten-year false positive rate with the test used before 3d mammograms (digital mammography) is 56.3, and the false positive rate with 3d mammograms (aka tomosynthesis) is 49.6.
So the study is actually reporting an improvement vs the previous state of the art! It's weird that the article is written to convey that the false positive rate of the new technique is a drawback when it's actually a benefit.
(This comment has been heavily edited thanks to a correction I received below.)
Edit: oh man. Now I see that I mixed up the direction too! What in the heck. The article was written like there was a big problem with 3d mammograms so I assumed it was performing worse on this metric than the baseline. But in fact it's better?
[If you assume that all women have mammograms at the normal cadence, over a period of 10 years then] half of women will get a false positive.
* Half of all women will get false positives - that's a lot, it's not great, people will think they have cancer and we should set expectations
* Despite the introduction of a newer technology to improve false positives (3D scanning) we've only improved the situation from 56% to 50%
But isn't that millions of fewer false positive results over 10 years?
Assumptions:
* US Women: ~150e6
* Fraction getting annual mamograms: ~1/3 (life expectancy of 80yrs, and mamograms during 30 of those years, minus some with poor access to health care)
* Reduction in cumulative FP rate over 10 year period: 6%
150e6 x 0.33 * 0.06 = ~3e6 fewer false positive results over a 10 year period. That seems like a significant effect size to me.
She went in for a followup ultrasound scan and was cleared.
Given the survivability of early stage breast cancer versus late stage, a high false positive rate doesn't seem worse than the alternative of missing early stage breast cancer. Obviously, better imaging or other tools would be even better.
Screening mammograms do reduce breast cancer mortality:
https://pubmed.ncbi.nlm.nih.gov/33141657/
Annual mammographic screening at the age of 40-49 years resulted in a relative reduction in mortality, which was attenuated after 10 years. It is likely that digital mammography with two views at all screens, as practised now, could improve this further. There was no evidence of overdiagnosis in addition to that which already results from the National Programme carried out at later ages.
I'm very sorry about your wife.
I stand by this sentiment wholeheartedly. 3D mammo caught a 0.6mm diameter tumor for my wife and she never even felt the lump. I felt it once and then couldn't find it again. I had to drag her to the doctors office as she didn't believe me. Oddly enough, my college years were spent in a pathology lab and I knew exactly what a tumor felt like and knew her family history and my heart sank the first time I felt it, so I was quite persistent in forcing her to see the doctor.
They rushed her surgery on the schedule and still the tumor was 3.5mm by the time they removed it. Time is absolutely of the essence. Had she waited for her annual exam or for some convincing evidence that she needed to go to the doctor, I fear it would have metastasized and I'd be a widower writing this comment.
I feel like adding, in the US, at the time, we had to fight tooth and nail for the 3D mammo to be covered by insurance and it was still pretty pricey copay. This really should be more widely available. Especially for certain demographic traits. My wife was 30ish (dense tissue), family history, known BRCA+, and should not have to fight for this technology that is the obvious best option.
https://www.cancer.org/cancer/breast-cancer/screening-tests-...
Related articles on the subject.
Something that should be kept in mind is that false positives should always be talked about alongside false negatives. There is no perfect test, and often in medicine there is an increase in one of those in order to decrease the other.
"A false-negative mammogram looks normal even though breast cancer is present. Overall, screening mammograms miss about 1 in 8 breast cancer"
It isn't discussed in the OP article, and I could not easily find something that differentiates false-negative rates in 2D vs 3D mammograms, but the cost of over-diagnosis might be worth it if that means less cancers go unnoticed in screenings.
It follows from a straightforward application of Bayes formula:
- Suppose a person has 1% chance of getting cancer, so p(cancer) = 0.01
- Suppose that the test has 99% sensitivity and specificity, i.e., p(positive | cancer) = p(negative | no cancer) = 0.99
- To use Bayes we first need the probability of a positive result regardless of disease status, i.e. p(positive) = p(positive | cancer) * p(cancer) + p(positive | no cancer) * p(no cancer) = 0.99 * 0.01 + 0.01 * 0.99 = 0.0198
- Then by Bayes we have p(cancer | positive) = p(positive | cancer) * p(cancer) / p(positive) = 0.99 * 0.01 / 0.0198 = 0.5
And things get worse the rarer the condition is. When p(cancer) = 0.5 the computations above give p(cancer | positive) = 0.99, and when p(cancer) = 0.001 then p(cancer | positive) = 0.09.
In other words, the rarer the condition, the more precise tests have to be.
The more evidence you have that this person is more likely to have the disease, the more useful the test becomes. Some examples:
- if the disease is more common among people over age 65, it's more useful on people in that age group
- if the person displays more symptoms associated with the disease, and not also associated with more common diseases, then it's more useful for someone with those symptoms
- if a disease is common in an area of the world, and the person has traveled there, the test is more useful
The more factors you have, the less likely a false positive. This is why it's often better to avoid medical tests unless there's a reason to suspect someone is at elevated risk for a disease: a positive result is more likely to be a false positive.
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate.
Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.
Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
Let's say two million people have been seriously inconvenienced by airport screening. We have to ask if that's an acceptable price to pay for the benefits? But do we know the benefits? Can we know the benefits? We actually don't know if the policy deterred anything, or even if someone who wanted to do something bad had their knife or liquid explosive or whatever confiscated and just abandoned the plan. It's purely a hypothetical so you're going to support the view you want: "we don't need to do all this because we don't actually have a big threat" or "we have successfully prevented any more attacks."
After all, we often see discussions here of foolish companies that underfund their security teams because "why are we paying so much for this when nobody has cyberattacked us?" ;)
When you have a condition that affects only 1% of the population, a test that reads negative 100% of the time will be 99% accurate.
A test that is always negative has 0% sensitivity and 100% specificity. Nobody thinks that sounds good.
https://en.wikipedia.org/wiki/Fuzzy_logic
Precision is not a good way to describe a test because, as I mentioned elsewhere, it relies on information about the population being tested.
So if you want to know "Has the Sun exploded?" you can flip a coin, assign heads to "Yes the Sun has exploded", and 50% of the time it will be correct.
You flip a coin, you get heads. You decide to flip the coin again, and you get heads again! What are the odds the Sun has exploded?
It's even worse in practice, given that the false positive rate is probably not independent between both measurements
This conclusion assumes that running the test twice is equivalent to running two independent tests for the same condition. For many real tests, getting a false positive once predicts a higher chance of getting another false positive in a repeated test.
Prostate cancer being somewhat of an outlier as at least to my understanding, essentially all men will end up with some kind of tumor or another if they live long enough, but some large fraction of those won't become something that requires medical intervention before the individual would die from any number of other age-related causes. Outside my area of expertise but the relative size of the tissue that produces various breast cancers as well as the proximity to lymph nodes change the calculus on that a bit.
But I would definitely expect overall screening protocols to evolve as methodologies are developed that can potentially exclude the false positives from another metholodgy. While test A + A may still produce false positives due the nature of the test failing the same way, if we have A + B + C that are all relatively low-impact and not horribly expensive, that may end up being a fairly good program if some combination of A, B, and C generally exclude the parameters that the other produces a false positive for.
That sounds interesting. Why does lymph node proximity matter?
Except for the highly malignant variety prostate cancer is generally an "indolent" tumor, that is slow-growing. So in the majority of cases chances are good that the patient would die of something else before the tumor becomes the cause of death. However prostate cancer is a leading cause of cancer death in men. In the US ~30k/yr men die of prostate cancer, so it's not a trivial condition.
In fact, "mammogram indeterminate" is a better diagnosis.
This is the general argument against testing for prostate cancer in non-symptomatic patients. If you only test patients that has symptoms then the rate of false positives goes down. The downside is that you only catch late stage patients where treatment might be too late. The alternative (which is used for mammography) is that for every person saved, an order of magnitude more will be permanent harmed by unnecessary surgery.
It's also why breast cancer screening is not recommended for everyone, just women over 50.
For women over 50 there has been a decades long sincere effort by the medical community to take this all into account and the consensus is that screening makes sense in that population.
It's not, though.
Possible ≠ certain or even likely.
Screening tests are used to determine whether there are indications for more invasive but more conclusive diagnostic tests.
Yeah but "possible cancer", taken literally like that, is everyone.
Time to bring out the pigeons.
Having just read the paper again, It’s fascinating.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4651348/
Almost all of them, in fact. If improving accuracy was as easy as repeating the test, the doctors would just be told that the correct procedure for this test is to take three readings.
You can expect that this is untrue only if the tests are especially expensive or invasive.
If the math is wrong, show us.
Specifically, why would the 10,001st test carry a 50% chance of being a false positive?
Thus, the population for tests 1..10,000 is different than the population for tests 10,001..10,100.
Edit: I misunderstood the scenario.
If we test again, and the test is "fair", then of these 101 people, we will retest 99% (or 99) true negative, 1% (or 1) false positive, and 1 true positive. So two positives, one false positive and one true positive. 1 out of 2 is 50%.
(Tho, there is a chance also that the true positive might come back as a false negative, then 1 out of 1 or 100% of positive results would be false positive)
The silly thing with this demonstration is that accuracy is poorly defined. Tests are normally conditional on the actual presence of the disease or not.
However, when it comes to mammograms, where the false negative rate is up to 12.5% and the false positive rate is up to 50%, and the incidence of breast cancer in women (12.5% over a lifetime) is rather high, false negatives would play more of a role.
The math is, given the test shows a positive, what is the probability that you actually have the disease?
If we are not making that assumption, then the 50% chance claim at the end is incorrect, since there's a chance that both of the two positive results are false positives and there is a false negative somewhere else in the remaining 99 (out of 101) people.
The 50% chance of it being a false positive (conditional on two positive results being observed) is actually a lower bound, and will increase as a function of the false negative rate.
A 1% false negative rate has a negligible impact on the math. 50% isn't perfect down to the decimal point, but it's very close.
After the first round you have 99.99 false positives and .99 true positives.
After the second round, if independent, you have .9999 false positives and .9801 true positives.
And in fact the lower bound, if you had no false negatives, is slightly below 50%. .9999 vs 1.
Their math shows that those are the same number, and they didn't elaborate on definitions.
The least misleading definition of accuracy is that accuracy = 1-false_positive_rate = 1-false_negative_rate. So unless told otherwise, I will use that definition.
"Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100"
Then they conclude by saying:
The fact there is one real positive necessarily implies they are assuming the false negative rate is exactly 0, given that the disease prevalence is in 0.01 percent of the population.Or that they rounded .99 people to 1 person.
And given that they did round the false positives, I don't see why that's a strange idea.
First, I'm assuming that top comment meant the false-positive rate was 1%, even though it was poorly stated as "accuracy." If you're commenting on this original ambiguity, then the whole thing is ambiguous.
But if we accept a 1% false-positive rate, then it makes no difference at all what the false-negative rate is, once we have a positive test.
Out of a 100,000 tests, on average 1000 will be falsely positive. That's what a 1% false-positive means. It makes no difference how many are truly positive, how many are truly negative, and how many are falsely negative. The definition of 1% false-positive means that, out of 100,000 tests, on average 1000 will show a positive test when the condition isn't actually there.
So if 100,000 people each took two tests, on average 10 will have two false-positive tests. Again, this is independent of the number of false-negatives, and is simply the definition of false-positive. (Some tests may have correlated false-positives, i.e. you're more likely to get two in a row, but that's not standard and not part of the definition.)
So if you have a disease whose incidence is 10 people out of 100,000, and you took two tests which are both positive, you are equally-likely to be in the group of 10 people who really have the disease, and the group of 10 people sitting there with two positive tests who don't have the disease.
You seem to be saying that the percentages will be affected by whether any of the 10 out of 100,000 who really do have the disease got a negative test. But it doesn't. 1% false-positive means that, on average, 10 people who don't have the disease will get two false positive tests. It doesn't matter what the people who really have it get.
Positive predictive value is the likelihood that, if you have gotten a positive test result, you actually have the disease. It’s calculated as TP/TP+FP.
TP, ie true positives, is a function of false negatives.
Here's a common sense explanation. Assume the false negative rate is 100% and so you have zero true positives. Then every positive test result will be a false positive, and therefore a positive test result shows that you don't have the disease.
On the other hand, assume the false negative rate is 0. Then a positive test result will mean that you probably do have the disease as long as the false positive rate isn't too high.
Remember your post I was replying to was all about the probability that you actually have the disease conditional on a positive test result. That's the question I am addressing. For this question, you must consider the false negative rate.
https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
> healthy people don't go in testing for the presense of a rare disease.
Well that’s precisely the point. There’s frequently the idea raised among lay people that it would be beneficial to test or screen for all kinds of rare or even just uncommon diseases that have high mortality in the general populace. This is the counter argument.
[0] https://en.wikipedia.org/wiki/Sensitivity_and_specificity [1] https://en.wikipedia.org/wiki/Positive_and_negative_predicti...
(edit yes of course cancer is not beneficial thank you)
I've read that's also a statistically tricky statement. IIRC, they measure survival time from when the cancer was detected. So early detection by definition improves the survival stats even when there's no meaningful treatment.
If I've read about this, I'm sure medical practitioners are well aware of it, but even so, I don't think it's easy to correct for it. How do you determine when a late-detected cancer actually started?
I am most familiar with NLST which actually was a long term randomized trial.
[edit: or, looking at the NLST you mentioned [1], maybe "treatment" doesn't always mean what I think of as treatment. They are actually comparing two different detection methods, and they aren't using a synthetic control.]
Do they ever use the "survival time after detection" stat I mentioned anymore, or has it been (/ should it be) abandoned?
[0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7218288/
[1] https://www.cancer.gov/types/lung/research/nlst
Yes, NLST was actually a real randomized control trial for lung cancer screening, done before it was known whether or not it was helpful. The trial eventually stopped once they had enough outcome based evidence.
The statistic you mentioned is not used in assessing the effectiveness of screening. It does closely relate to what is though to be the causal mechanism that leads to effectiveness.
Living hapyy is more important than living for lots of people.
A breast cancer scare, is certainly harmful, but it's important to remember that it is going to be a short episode as it becomes clear that the tumor is not growing. So you have maybe a few months of scariness.
On the other hand cancer is brutal, and death is permanent. Years of life cut short. The years leading up that are extremely sad. The impacts of death extend beyond the person to their family and friends.
When people try their best to measure these things on balance, breast cancer screening is overall a good intervention.
It's pretty uncontroversial that cancer screening for women above 50 years old or women with high risk (e.g. familial or genetic evidence or risk, history of cancer/treatment) is a net beneficial procedure.
For asymptomatic, low-risk women aged 40-49, the recommendations are a bit more mixed. While there is some evidence that overall mortality does improve, there are also considerable harms (physical, emotional, financial) associated with false positives and "overdiagnosis" (that is, identifying and treating low-grade non-invasive lesions that are unlikely to progress to breast cancer).
It would be interesting to know the mortality rate of unnecessary breast surgery. I would hope the number is less than 10%, but I suspect it is not an insignificant number of people. Net beneficial, but definitively controversial.
In a population of 1,000,000:
TP=99
TN=989901
FP=9999
FN=1
Accuracy = TN + TP / (TP+TN+FP+FN) = 99%
Sensitivity = TP / (TP + FN ) = 99%
Specificity = TN / (TN + FP ) = 99%
Looks like a great test across the board!
However, the actual chance someone with a positive test result has the disease:
TP / ( TP + FP ) = ~1%
So why isn't this ratio part of the major test criteria?
Because it has nothing to do with the test. Sensitivity and specificity are facts about the test, and conceptually they will remain whatever they are if you start testing a different population.
Whether doing a test has any value is a fact about the population ("olympic athletes are never anemic, so there's no point in testing them for anemia") that has nothing to do with the test. Why would you evaluate the test based on that?
I think it's because this number depends on the percent of the population that is positive which can change after the trial is conducted.
https://github.com/khuyentran1401/Data-science/blob/master/s...
Is there a specific metric word or phrase for this less than 1% positive test result reliability?
Edit: Looks like hadmatter responded with the term I'm looking for!
[1] https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
I remember reading studies about doctors not understanding the stats years ago, but my wife assures me that medical education has over corrected in the last decade or 2.
The correct answer is that the question doesn't give enough information to know.
I mean, a test that always said no would be 99.99% accurate. A mere 99% accuracy could be no false negatives and 0.99% of all tests false positives, in which case the answer is that a positive result is 1% likely to be correct, and 99% likely to be a false positive. Or it could be, on the other extreme, 0.01% false negative and 0.98% of all tests false positives, in which case the chance that a positive result represents an actual case would be 0%.
Presumably, though, approval processes for medical tests will constrain the actual distribution of results to be such that, a patient with positive result is not more likely to be clear of the condition than one with a negative result, so it won't be as bad as the latter extreme, but it's worth noting that your assumption (essentially that the false/true ratio is equal for negative and positive results) is not certain at all.
Nitpick here. There's nothing wrong with the math of Bayesian probability, but the setup is usually something like an STD test, and in those cases the prior is wrong.
Our prior probability shouldn't be "0.01% (1 in 10,000) of the population is infected," we really want % of infection for the population that get tested.
This is really obvious with disease tests because people don't typically get tests without reasons (symptoms, contact with infected persons, etc.). The math as-is would only work if we're sampling people randomly.
Couldn’t it also alternatively be defined as 99% accurate for a given person?
Maybe there are reasons/conventions it is always as you originally describe, but if that’s the case wouldn’t it be more about poor communication to the public than a paradox?
As for why it is the case, I presume it is just the usual focus on counting your hits and not on your misses.
Now the conditional probability of infection, given TWO positive tests is still far below 50%.
This is why a second test of a different type, even if it is less accurate, if it is more likely to be independent, is much preferred!
It's a Bayesian update. Every (independent) positive result increases your confidence of having the condition by 100x. At first you have no information so you assume the base rate, 1:10000. At the first test, that goes to 100:10000. Second test gets you to 10000:10000 or even odds. Third positive result finally gets you to 1000000:10000, or a 99% confidence of having the condition.
Let's say we've got a test that:
* If you've got cancer, detects it 100% of the time (to make the maths easy) * If you haven't got cancer, gives you a (true) negative 99% of the time, and a (false) positive 1% of the time.
For starters, consider a population of 100 people, all of whom have cancer. Our magical test always finds these cases, so we have an 100% positive rate, and a 0% false positive rate. (None of these can be false positives because we know they all really do have cancer.)
Now consider a group of 200 people, 50% of whom have cancer, and 50% of whom don't. The group who have cancer will still all get detected, but if the 100 people who don't have cancer, one of them will be marked as having cancer even though they don't. In total, that's 101 people who will get a positive result. So our overall positivity rate is 101/200 (~50%), but our false positivity rate is the number of false positives divided by the number of all positives, so 1/101 (~1%)
Now consider the case where we have 101 people, of whom only one has cancer. If we run our test a third time, we of course detect the person with cancer, but of the remaining 100 who don't have cancer, we will detect a second person. That's two positive cases in 101 people, so our positivity rate is ~2%. However, of those two positive cases we know only one really has cancer - the false positivity rate is 1/2, or 50%.
So that's three situations where the only thing we've varied is the proportion of the population who have the condition we're scanning for. (Technically we also changed how big the population was in the first place but that was just to make the maths easier - it doesn't affect the results at all.) However, we get a different rate of false positives in every test, because the number of false positives is dependent on the ratio between the number of people with cancer and the number of people without.
It's also worth noting that this obviously doesn't just apply to cancer, but also other situations. Most relevant to us in the last couple of years is probably Covid tests - you might have seen over the summers some scepticism about how valuable it is to do lots and lots of tests, and that's because during those times, the number of affected people was lower, and so these tests were more likely to produce less useful results. However, in winter, even relatively poor tests become more useful, because there are more true positives, and therefore a lower rate of false positives.
Given that, from the first test, we have 100 people who were false positives, and 1 person who is truly positive. We would expect that 1 truly positive person to be positive again (we have a 99% chance of it from the test, after all), and 1 of the 100 who were false positive before to also test positive this second time (that 1% inaccuracy rate). Given 1 true positive, and 1 false positive, in that sample, you would expect a positive rate to mean you have a 50% chance of being the true positive.
If we did not have information from the first test, yes, it would be ~1% chance of a false positive. We have nearly a 0% chance of having a true positive, so of the 101 people, and a 99% accurate test, we would expect one to come out positive, despite not having the disease. But, because we have that first test, from a much larger sample size, we likely have one person who is a true positive, and so could expect two people to test positive, one with the disease, and one without (the 1/100 of the remaining), giving us a 50% chance a positive result = has the disease.
Why am I forced to submit to these screening tests for issues I'm not at risk for in order to receive tangentially related medical care?
Current medical advice does NOT stipulate the two should be related - https://www.acog.org/womens-health/experts-and-stories/ask-a...
If you can't find a doc who won't decouple the two, try Planned Parenthood.
I wonder how much difference there really is with the older tech based on looking at outcomes. For standard surveillance, does the new 3D dental xray actually benefit me? Granted the cost is low (<10% normal yearly exposure), but I still wonder.
For CT vs. XRay there have already been 2 long term randomized controlled trials that look at outcomes for lung cancer screening. PLCO, the XRay study, was not effective in improving long term outcomes like mortality. NLST, the CT study, showed a 20% reduction in mortality. With more modern techniques that actually look at the tumor in 3D NELSON in Europe has seen even better mortality reduction.
DBT is newer so A) Radiologists are not as good at reading it yet B) the long term evidence hasn't accumulated yet, but I am definitely optimistic that it will present a big improvement over the previous state of the art.
As mentioned elsewhere in the thread, this false positive number is already better than the existing state of the art for 2D.
these testing systems are just one of many indicators that are meant to be used . if used in isolation - very harmful outcomes will occur
False positives also have a major economic impact, to the tune of $4 billion USD [2].
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3601385/
[2] https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2014....
https://pubmed.ncbi.nlm.nih.gov/31098856/
The "4B$" is a rather useless number. The same people would probably argue it's much cheaper to go to your local shaman instead of a medical doctor.
However, in my opinion, there has been a systemic resistance to screening because of statistics like this, and that is misguided. This is not the right number to use for determining whether or not screening is useful.
A 3D mammogram will have false positives, yes. And some women will need to either get an uneccesary ultrasound (annoying, but not harmful) or worse, an uneccessary tissue sample (biopsy). This type of biopsy is done with a needle and is certainly uncomfortable, but carries little to no risk. This is all true. But it needs to be balanced against the fact that catching cancer early gives you an immensely better chance of surviving.
The right numbers to look at are more like: How much does a screening program reduce overall mortality rates? How much does the screening program reduce breast cancer mortality rate? What are the impacts of this in terms of QALY (quality adjusted life years)? What are the impacts of the false positives in quality adjusted life years?
I believe on balance the answer to these questions is that screening for breast cancer is on the whole beneficial. Here is one example study https://pubmed.ncbi.nlm.nih.gov/31098856/
Now the question is, how will 3D affect these questions? That is as of yet unanswered as far as I know but we shouldn't let numbers like this influence the conversation prematurely.
Here is another study looking at all-cause mortality, which again does implicitly handle overdiagnosis. https://www.ajronline.org/doi/full/10.2214/AJR.14.12666
We should instead present it as 'you currently have breast cancer with 13% probability' (population avg with no tests), a positive test means you have it with 25% probability, a negative test with 3% probability.
That makes the test a lot less scary, after all you're only gaining 10% certainty either direction.
I feel like this is where bedside manor comes into play and the human component of being a physician. Any doctor ordering a 3D mammogram, then receiving a positive result, should communicate to their patient that imaging is just a screening tool and not diagnostic. Many doctors will not even use the word "Positive" in the test result conversation. They'll use vague wording like, "we did happen to see something we'd like to investigate further to be sure, it's really hard to see exactly what's going on in your body on a camera so while I was hoping to avoid it I do believe a biopsy should be the next step". Yes, a biopsy is now warranted, but the patient should try to keep up their spirits high and not stress to much at this point. The biopsy will tell more information about what is going on inside the body and we can come up with a plan for whatever that is once we know more.
It's natural/unavoidable for the patient to be worried/anxious but should not yet be scared and that is best conveyed in a human interaction with a caring physician.
"Long-term follow-up of the UK Age RCT" "There was a statistically significant 25% reduction in mortality from breast cancers diagnosed during the intervention phase at 10 years' follow-up"
But this is about screening of rather young women. Most established screening programs start at an age of 50. Still young. :-)
Anyway, even with cancer you can of course bet on dying of another cause before the cancer gets you and thus ignore medical therapy. Some people win this bet ... somehow.
<https://instacalc.com/55494/embed?d=&c=QWN0dWFsX3Byb2JhYmlsa...>
For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis.
Every concern that has been raised in these comments is factored into our reporting system. All radiologists use the BI-RADS reporting system which is internationally validated and expresses probabilities of malignancy to guide further management based on specific imaging. features that have been validated against pathologic data from excised surgical specimens and biopsies. This was done to homogenize reporting practices.
For example a BI-RADS 5 lesion, which is based on specific imaging features not a gestalt/expert opinion, denotes a 95% probability of malignancy and would generally result urgent biopsy + surgical consultation. A BI-RADS 3 lesion has a 2% chance of malignancy and would generally be followed with serial imaging for a specified interval.
BI-RADS 4 is the middle, this is broken down into A/B/C but essentially anything with 3%+ probability goes for biopsy. That means we expect up to 97% of the lesions we biopsy will have been false positives.
This is intentional and desired, this is because the harms of missing breast cancer is horrible, early detection results in treatment with lumpectomy + radiation vs advanced breast cancer needing systemic therapies and radical resections.
Second, what happens when there is an abnormality on a screening mammogram is not straight to biopsy. We use additional mammography views and ultrasound to help us find out what's going on before making that decision. We also have MRI to troubleshoot and increasingly for screening, certainly for high risk and extremely dense breasts.
Tomosynthesis or 3D mammography is not a cure-all for false positives, it's also not intended for dense breasts. The point of Tomosynthesis is to reduce callbacks for overlapping breast tissue which can sometimes look like a real cancer, and to increase sensitivity for architectural distortion (an occult or infiltrative cancer distorting the fibroglandular architecture) which can be really hard to pick up on 2D.
It works really great for that. Patients with extremely dense breasts should really be getting screening MRIs (US as a lesser alternative) in today's age and this is happening with increasing frequently.
Further reading:
https://en.wikipedia.org/wiki/Receiver_operating_characteris...
https://www.acr.org/Clinical-Resources/Reporting-and-Data-Sy...
https://www.nejm.org/doi/full/10.1056/NEJMoa1903986
https://www.cancer.gov/about-cancer/treatment/clinical-trial...
I totally get this and it's technical terminology.
I think my takeaway from trying to explain why this isn't a big deal on this thread is that calling a tomo or mammo "positive" is just a recipe for confusion.
I understand why it's done this way historically, but the idea of a mammo or tomo being "positive for cancer" makes no sense since there will almost always be either an ultrasound a biopsy or something else before a diagnosis is made. It's a test for whether a more invasive/expensive test is worth it, not a test for cancer.
Curious what your opinion is on AI for DBT? I think there is great long term potential here, even more so than the potential that we have seen so far in mammo (such as https://www.nature.com/articles/s41586-019-1799-6), because it's easier for AI to thoroughly look at things in 3d and spot new patterns that are not obvious to human eyes.
My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.
There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).
AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity.
What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same).
I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.
A) I think it's too early to make a call here. The first big paper for AI in medical imaging with modern methods came out in 2016, and we are just now getting large scale prospective studies where algorithms outperform specialists: https://www.thelancet.com/journals/landig/article/PIIS2589-7...
My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.
B) Totally agree that old CAD has not been effective, and this has created a huge trust gap for the new generation of technology. The new generation of technology using deep learning is also easier for anyone to create so it will take some time to separate the good from the bad.
There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).
C) This claim seems overly broad. If we can improve sensitivity it can certainly be a good thing. That being said, improving specificity in an assisted read scenario seems almost impossible in my experience, because if a doctor is convinced something is cancer, no output from an algorithm is going to get them to downgrade. We did an analysis on exactly what was going on when our algorithm would give a lower, non-recall score vs. a panel of radiologists, and we found that the most common situation was missing that a lesion was a scar. https://www.nature.com/articles/s41591-019-0447-x
It turns out that radiologists will in general not look at a lesion from all different possible reconstructions in 3D, because it's too time consuming. But sometimes there is only one reconstruction/view where the scarring is obvious. AI obviously does not have this issue, so in principle it is certainly capable of improving on specificity and this has been seen in many peer reviewed retrospective studies.
As I mentioned, the issue is how to make this happen in clinical practice where assisted read is the only option that will be tolerated for deployment for at least the next 5-10 years. In that time period, I think the only thing we can really hope for is a boost to sensitivity without impacting specificity. I believe eventually enough strong evidence (prospective observational multi site studies) will accumulate, and the algorithms will improve enough that for something like mammography, at least a portion of the reads can be done without a radiologist. This is probably the only way we will see improvements to specificity AND sensitivity in my opinion.
Increased sensitivity (I.e. in CTPA) often leads to over diagnosis and bad outcomes.
We don’t routinely look at all three planes because most imaging and size criteria were developed and validated on axial sections (I.e. RECIST). Research done on volumetric measurements is in its infancy and of questionable added benefit.
Show me an example where sensitivity is a problem in modern day radiology and I might buy this argument.
Otherwise this is a solution in search of a problem
One thing engineers often don’t appreciate about medical interpretation is that you may THINK you’ve designed a better wheel but unless you can show better OUTCOMES (not increased detection) this is probably irrelevant.
"The findings of this study suggest that digital breast tomosynthesis is associated with a lower cumulative probability of false-positive results compared with digital mammography"
Which is an important finding since today such screenings rely on digital mammography. So, tomography is good.
For whatever reason, the cited article turns this upside down into:
"Half of all women getting 3D mammograms will experience a false positive over a decade of annual screening"
Which suggests that tomography performs really bad.
If you do a mammography screening every two years, you do 5 mammographies in 10 years. With a false positive, you do 5 and an MRI. So what. Doctors should always tell their patients that a positive screening mammography doesn't necessarily imply they actually have cancer.
Sure, statistics say that there are false positives. Personally, I rather have three false positives, than one false negative. Taken to a bit of an extreme, the outcome of first is money spent and stress, the outcome of the later is death. I am not a statistics to myself.
I am sorry for the inconvenience to the medical field by having to re-test, and certainly feel for the insurance company to have to pay for a second, or third opinion. I care, I really do. Just, not that much.
That said, it took me decades to recognize the need and add sympathy into my calculations.
An ultrasound is not intrusive at all, a biopsy however is.
If I was a conspiracy theorist I'd say that a high amount of false positives are in fact an economic incentive for the medical community. And anyone having seen the hell metastatic breast cancer is, up close and personal, will insist that 1000 false positives is worth it to catch one case early on.
Cancer awareness spam combined with obsessive screening and mild hostility towards women who do not frequently line up for ovarian/breast cancer testing (I don't see men being targeted nearly as much as women) is probably less about saving lives and more about making money imho.
A biopsy is not at all harmless neither in the breast nor vagina. Do a quick internet search on prostate biopsy to see some gruesome numbers. All are pretty painful, prone to infection and a high chance of side effects. Some pretty severe, and they are not astronomically rare. The severe ones are in the 1% range.
The fear of the brutality that is metastatic cancer is enough for few object to being routinely subjected and even coerced into this pretty dangerous and intrusive procedure.
Those who do object are generally dismissed or even ridiculed.
There is even a radiologist in the commentary here who sees no issue with the high rates of false positives as a potentially dangerous biopsy will quickly clear up the cancer scare...
For vaginal scraping which is actually recommended as a screening measure for my age group 2-4 out of 100 women will have side effects. About 1 out of 100 will have severe side effects.
Anecdotally. I have even had one doctor ( female if that matters ) tell me that after 50 it is probably best to have both my breasts and ovaries removed. As a preventative measure.
This is the link I received from my doctor in Iceland about vaginal scraping and this is where I took my side effect statistics from. https://www.landspitali.is/library/Sameiginlegar-skrar/Gagna...
Medical doctors tend to terrify me. I really do avoid them unless I have something really concrete to show them that needs fixing and is not up for interpretation. EDIT: typos