It appears that the FDA also has a data problem, as it refuses to respond to FOIA requests for data that is obviously in the public interest to be disclosed, and taxpayers end up paying $1.3 billion to stockpile a medicine that is probably not as effective as advertised.
They also actually put a gag order on at least one pharma company preventing from disclosing the fact that antidepressants were not effective in children and increased the risk of suicide.
The thing about medical studies though is that only the ones the pharma companies want to get published make it into journals, and there is no requirement that what they are saying in the journal articles matches the clinical trial data at all. And the FDA doesn't publish any of the raw clinical trial data, so you'll never know.
This is especially problematic since pharma companies only need two studies showing a drug plausibly works to get approval, regardless of how many trials fail to show any benefit. So usually what you see is one or two journal articles showing a drug works, which may or may not correspond to the findings of one or two clinical trials, and the rest of the trial data is buried forever. And then other authors publish metareviews based on this completely inaccurate data.
Essentially journal articles should only be ever be viewed as advertisements for the pharma industry, and should no way ever be taken as science. The one exception is the largescale trials funded by the NIH.
There have been some changes [1] in recent years attempting to address this. Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported. There could certainly be some loopholes or limitations on the data release that I am unaware of.
However, it is true that most of the controversial, blockbuster drugs that are discussed in the original article were approved long before these requirements came into effect.
> Since 2007, all drug, biologic, and device trials in the US must be registered with ClinicalTrials.gov after phase 1 (which, at least in principle, addresses safety only - not efficacy), and "results" must be reported.
- A substantial percentage of trials haven't reported their results within the required timeframe.
- The number of fines the FDA has levied for non-compliance is zero.
- The data isn't actually published on clinicaltrials.gov, only a brief summary of the results which doesn't really tell you anything.
- As you say, the requirements aren't retroactive.
- There are several large loopholes, e.g. phase IV studies aren't required to be registered.
I recently listened to an advanced statistics lecture. As an illustration the Professor said, medical science (still) uses 2 sigma statistics, jet engine builders use 6 sigma. That's why jet engines never (sic) fail.
Yes, and then there is the influence of the drug makers too.
The biggest difference is that jet engine builders can conduct an unlimited number proper controlled scientific experiments, while the fields that study humans (psychology, medicine, sociology, economy) can usually only conduct a limited number of field trials (which, even though they are "controlled" and possibly "double blind" are nowhere as rigorous as in proper sciences).
This is a very important point that seems to be forgotten frequently. Particularly when doing methodological comparison between disciplines.
The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
Yes. There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely.
I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome.
This is separate from the data publication issue that the article focuses on.
> There was the other article sitting around recently asking why the rate of blockbuster drugs have slowed down. Getting even the level of confidence required now is hugely expensive. Requiring higher confidence is an expensive choice that must be weighed closely.
Not just expensive but also potentially unethical.
Consider this: post-exposure prophylaxis for HIV. It's used both for sexual and non-sexual exposure to the HIV virus. It's a month-long series of antiretrovirals (the exact protocols used differ). It's effective if you start taking it within 72 hours of exposure.
The drugs themselves have been tested for use by people who are HIV+, but nobody has done clinical trials on using them as post-exposure prophylaxis. We know it works, but we don't know how well because testing would be unethical. You'd have to take half the people who are exposed and give them placebo and put them at risk of getting HIV.
At a certain point, you have to throw the dice and say "given the evidence we have, this is what we are going to do".
(The fact that such judgement calls are being distorted by evidence being intentionally withheld by drug companies is disgraceful though.)
"Requiring higher confidence is an expensive choice that must be weighed closely."
Confidence of effectiveness is very easy to get - you just need to find an effective drug.
e.g. when antibiotics were first tried - they are massively effective compared to previous treatments - you get high confidence of effectiveness from just a small sample size.
A reason it is hard for new drugs to get high confidence is because they are only marginally better than existing drugs.
What is a "blockbuster" drug? One that makes a lot of money for the various stakeholders or one that has a large impact on human health? I'm not sure I care too much about the former and it's not at all obvious how "level of confidence" would affect the latter. Yes, as a result of past medical progress it is more difficult to improve upon the current state of the art.
But new therapies have to compete with existing therapies in the market anyway. If a new therapy can't outperform existing ones, what actual value does it have? Perhaps drug companies should think about solving problems that aren't already solved rather than bemoaning their inability to reap massive profits while reinventing the wheel or pushing minor formulation tweaks to renew their patents.
The main difficulty is that it is unethical and illegal to test on humans to any degree of significance for treatments that have important ramifications. The moment results start leaning one way, researchers are obligated to move all the patients to the beneficial or non dangerous side. You are not allowed to let humans get sick or die in order to keep a clean control group.
Oh come on! How is this sigma counting relevant? What would you prefer, a drug that improved the condition of 1% tested patients (p = 10^{-6}) or a drug that improved the condition of 95% tested patients (p = 0.05)?
Take any drug, test in on millions of patients, here is your six sigma. What problem did you solve?
You name a problem. Many years ago, I used to teach physics and with it also error treatment and some basic statistics to medical students. My overall impression was, they never got it, nor did the majority even understand its importance, when results are interpreted. Most medical student appeared to be exceptionally smart, even smarter as physics students. But somehow statistics never appealed to them.
Drugs act on proteins which are little machines in the body, the instructions of which are encoded in our genome. The slight variations in our genetic code leads to some proteins having slight variations in their structure (and their function). Most of the time this is harmful, but sometimes it causes them to react differently when they interact with a drug or with each other. The manifestation of this can sometimes be harmful. Understanding "pharmacogenomic" effects will help us mitigate these side-effects going forward.
A person's genome, epigenetic profile, physical environment, social environment, microbiome, parasite-ome, neural connectome, metabolism, enteric connectome, and diet are all unique. Some of these factors regulate the others; some have two-way regulation.
There are far more combinations and variations of the above variables and sub-variables than can be tested for in clinical trials. This fact means that some obscure combination of these variables may result in unintended negative (or positive, less frequently) consequences upon consumption.
Theoretically, you could test for differences in the 1000th person-- with today's technology, it would be a test which described the person's genome. Once scientists had that genomic data in hand, they'd certainly find a profile of alleles which could feasibly describe why a negative reaction occurred.
Importantly, everyone's genome is unique, meaning that a correlation between genomes and negative drug reactions must be exceptionally rigorous in order to be believable. Currently, this correlation poses a problem, as our ability to quantify and understand genomes is far ahead of the other dimensions which I mentioned at the start of my post. As our understanding of the microbiome, enteric connectome, and epigenetics increases, it'll be more possible to predict adverse drug reactions beforehand, assuming the patients in question have been profiled.
Medical science does indeed have a data (or, rather, rigorous statistics problem), but it has a much more serious, intrinsic, problem: dimensionality. The number of variables is so high that proving something (clinically, when the biological process is unclear, which is most of the time) to a high degree of certainty is almost impossible.
Randomization lets you ignore dimensions that you're not interested in. It doesn't do much in the way of letting you attribute an effect to one or more dimensions from a large set of them.
There's a fundamental lower bound to the number of data-points you need to make in order to achieve statistical significance. This lower bound grows exponentially in the number of dimensions, particularly if you cannot make the simplifying assumption that the dimensions are statistically independent (and biological systems are often coupled in strange ways). Randomization is a good way of achieving that lower bound, but it cannot defeat it.
I read the article last evening before I turned in in this time zone, and have just read all the comments posted till now overnight. Readers of the writings of Dr. John P.A. Ioannidis[1] will already be familiar with many of these issues, which dog most research on human health and medical treatments. Several of the previous comments here are very helpful in pointing out that human physiology is based on much more complicated system interactions than most man-made machines, and subtle individual variations in genome or personal environment that are now poorly understood can make a different in how drugs work for different patients.
That said, an important point that hasn't been made here yet is that "alternative" treatments to the treatments developed by medical science often have a much worse data problem, because no one is gathering data about the alternative treatments for safety or for effectiveness in the first place. In the United States, "natural supplements" are regulated in a way that exempts billions of dollars of annual revenue from any review about whether the supplements do any good at all.[2] Chiropractic manipulation can kill, but chiropractors don't look into that issue carefully.[3] What you as a person who desires to stay healthy for many years have to do is turn in the first instance to medicine that works[4] and to continually ask your doctors for explanations of what they are doing and what the rationale is for what they are doing. Patients thinking more will encourage physicians to think more. Voters telling governments to regulate treatments more rationally will also help, as will voters supporting more funding of basic medical research.
I don't think that anyone outside the field can understand just how bad and pervasive this problem is, it is everywhere and harming patients every day. One more example - my wife is a surgeon working in a specialized field and in this field there are panels of 'experts' that get together and set the guidelines for how to treat the disease:
- The panel is stacked with visiting researchers working for the most senior researcher on the panel.
- The most senior experts on this panel recommend treating with a particular drug. It turns out that these experts also happen to receive money from these drug companies.
- These researchers then use the results of this panel when applying for NIH grants as proof that this is where the treatment is heading AND THEN, next time the panel happens, also use this research as proof that this is where the treatment is heading. Talk about self interest and circular logic.
What is incredibly sad about this is that the results of consensus panels are then quoted by doctors every day and used to lead their treatment of patients.
Clinical trials data is one piece of the puzzle and definitely a important one. One thing which in my opinion would dramatically change patient outcomes if care provider starts listening to the patient. In an anecdotal experience, my wife was wrongly diagnosed for almost a year impacting every day of her life to be filled with pain. Issue as it turned out was a simple infection and was fixed by an exceptional surgeon with tweezers in few minutes.
27 comments
[ 4.6 ms ] story [ 67.5 ms ] threadHow does this practice keep us safer?
The thing about medical studies though is that only the ones the pharma companies want to get published make it into journals, and there is no requirement that what they are saying in the journal articles matches the clinical trial data at all. And the FDA doesn't publish any of the raw clinical trial data, so you'll never know.
This is especially problematic since pharma companies only need two studies showing a drug plausibly works to get approval, regardless of how many trials fail to show any benefit. So usually what you see is one or two journal articles showing a drug works, which may or may not correspond to the findings of one or two clinical trials, and the rest of the trial data is buried forever. And then other authors publish metareviews based on this completely inaccurate data.
Essentially journal articles should only be ever be viewed as advertisements for the pharma industry, and should no way ever be taken as science. The one exception is the largescale trials funded by the NIH.
However, it is true that most of the controversial, blockbuster drugs that are discussed in the original article were approved long before these requirements came into effect.
[1] http://clinicaltrials.gov/ct2/manage-recs/fdaaa
- A substantial percentage of trials haven't reported their results within the required timeframe.
- The number of fines the FDA has levied for non-compliance is zero.
- The data isn't actually published on clinicaltrials.gov, only a brief summary of the results which doesn't really tell you anything.
- As you say, the requirements aren't retroactive.
- There are several large loopholes, e.g. phase IV studies aren't required to be registered.
Yes, and then there is the influence of the drug makers too.
The other point is complexity. The number of variables needed to create a useful model of a jet engine is a lot smaller than the number needed to model an organic system. This means that today we have useful models of jet engines and we can reliably test something with simulations - we may generate loads of valid data and easily apply "top-notch" statistics. The same, unfortunately, is not true for medicine.
I think it's becoming clearer each day that many drug side effects, and general drug efficacy is effected by the genetics of the drug taker. Without being able to tease out these genetic effects, blindly aiming for broad population safety or efficacy levels is always going to be troublesome.
This is separate from the data publication issue that the article focuses on.
Not just expensive but also potentially unethical.
Consider this: post-exposure prophylaxis for HIV. It's used both for sexual and non-sexual exposure to the HIV virus. It's a month-long series of antiretrovirals (the exact protocols used differ). It's effective if you start taking it within 72 hours of exposure.
The drugs themselves have been tested for use by people who are HIV+, but nobody has done clinical trials on using them as post-exposure prophylaxis. We know it works, but we don't know how well because testing would be unethical. You'd have to take half the people who are exposed and give them placebo and put them at risk of getting HIV.
At a certain point, you have to throw the dice and say "given the evidence we have, this is what we are going to do".
(The fact that such judgement calls are being distorted by evidence being intentionally withheld by drug companies is disgraceful though.)
Confidence of effectiveness is very easy to get - you just need to find an effective drug. e.g. when antibiotics were first tried - they are massively effective compared to previous treatments - you get high confidence of effectiveness from just a small sample size.
A reason it is hard for new drugs to get high confidence is because they are only marginally better than existing drugs.
But new therapies have to compete with existing therapies in the market anyway. If a new therapy can't outperform existing ones, what actual value does it have? Perhaps drug companies should think about solving problems that aren't already solved rather than bemoaning their inability to reap massive profits while reinventing the wheel or pushing minor formulation tweaks to renew their patents.
Take any drug, test in on millions of patients, here is your six sigma. What problem did you solve?
Your comment shows me, that not much has changed.
If there is something different biologically different about the 1,000th person, could it be tested for?
Drugs act on proteins which are little machines in the body, the instructions of which are encoded in our genome. The slight variations in our genetic code leads to some proteins having slight variations in their structure (and their function). Most of the time this is harmful, but sometimes it causes them to react differently when they interact with a drug or with each other. The manifestation of this can sometimes be harmful. Understanding "pharmacogenomic" effects will help us mitigate these side-effects going forward.
There are far more combinations and variations of the above variables and sub-variables than can be tested for in clinical trials. This fact means that some obscure combination of these variables may result in unintended negative (or positive, less frequently) consequences upon consumption.
Theoretically, you could test for differences in the 1000th person-- with today's technology, it would be a test which described the person's genome. Once scientists had that genomic data in hand, they'd certainly find a profile of alleles which could feasibly describe why a negative reaction occurred.
Importantly, everyone's genome is unique, meaning that a correlation between genomes and negative drug reactions must be exceptionally rigorous in order to be believable. Currently, this correlation poses a problem, as our ability to quantify and understand genomes is far ahead of the other dimensions which I mentioned at the start of my post. As our understanding of the microbiome, enteric connectome, and epigenetics increases, it'll be more possible to predict adverse drug reactions beforehand, assuming the patients in question have been profiled.
There's a fundamental lower bound to the number of data-points you need to make in order to achieve statistical significance. This lower bound grows exponentially in the number of dimensions, particularly if you cannot make the simplifying assumption that the dimensions are statistically independent (and biological systems are often coupled in strange ways). Randomization is a good way of achieving that lower bound, but it cannot defeat it.
That said, an important point that hasn't been made here yet is that "alternative" treatments to the treatments developed by medical science often have a much worse data problem, because no one is gathering data about the alternative treatments for safety or for effectiveness in the first place. In the United States, "natural supplements" are regulated in a way that exempts billions of dollars of annual revenue from any review about whether the supplements do any good at all.[2] Chiropractic manipulation can kill, but chiropractors don't look into that issue carefully.[3] What you as a person who desires to stay healthy for many years have to do is turn in the first instance to medicine that works[4] and to continually ask your doctors for explanations of what they are doing and what the rationale is for what they are doing. Patients thinking more will encourage physicians to think more. Voters telling governments to regulate treatments more rationally will also help, as will voters supporting more funding of basic medical research.
[1] https://med.stanford.edu/profiles/john-ioannidis?tab=publica...
[2] "Dietary supplement industry says “no” to more information for consumers (again)" http://www.sciencebasedmedicine.org/big-supp-resists-giving-...
[3] "Stroke Death from Chiropractic Neck Manipulation" http://www.sciencebasedmedicine.org/stroke-death-from-chirop...
[4] "There is no such thing as alternative medicine. There is only medicine that works and medicine that doesn't." http://books.google.com/books?id=zOh0t3QFXdoC&dq=There+is+no...
- The panel is stacked with visiting researchers working for the most senior researcher on the panel.
- The most senior experts on this panel recommend treating with a particular drug. It turns out that these experts also happen to receive money from these drug companies.
- These researchers then use the results of this panel when applying for NIH grants as proof that this is where the treatment is heading AND THEN, next time the panel happens, also use this research as proof that this is where the treatment is heading. Talk about self interest and circular logic.
What is incredibly sad about this is that the results of consensus panels are then quoted by doctors every day and used to lead their treatment of patients.