Quite thought-provoking, and connecting it to a related field it seems the (relative) success of LLMs and the likes are indications that enough data can at least learn you something without always needing to interfere with the world first(?)
In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input.
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
The reality that individuals and cultures create can be a dark and confining place. Empiricism is our only window to a universe full of possibility and light. Humanity is like a child, standing on our toes to peer through and wonder.
Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
That's not a "classic". It's just a silly, overwrought straw-man [1] that gets passed around by people who think they're being clever, and/or don't understand the topic with any degree of nuance.
Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment.
On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials.
[1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.
Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance.
We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".
12 comments
[ 2.3 ms ] story [ 51.2 ms ] threadThis is not happening.
> I was surprised to find that they usually discard papers based on observational evidence wholesale.
He he...welcome to the real world!
Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Many others.
I believe, without the tiniest shred of doubt, that the only trials drugs need to go through are initial safety/toxicity trials (phases 0/1) and that everything else would be much better left to access+observation.
This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually representing the phenomenon of interest, or the conditions under which they actually occur.
I often use the example of videogames and aggression. There were plenty of experimental studies of this but it was always questionable whether lab-induced anger is the same thing as, say, the sort of violence we generally are concerned about societally.
I generally have tried to teach students that experimental designs when done right provide powerful causal evidence of something, but often with limited generalizability; observational designs in contrast provide powerful generalizable evidence of some kind of association, but often with limited certainty about the causal pathways involved.
I've been in a department that was rabidly experimental in its focus and it always seemed sort of short-sighted, because people were idolizing RCTs with proxy manipulations that had questionable generalizability to the real-world phenomena they were trying to model.
Ideally you'd bring both experimental and observational evidence to bear on a question. Your conclusions should be robust to different types of designs.
_Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_
https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/
"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."
Nobody reasonable suggests that RCTs must be used for every problem. When the effect size is enormous, or there's obviously no alternative (as in the parachute example), it's silly to suggest the use of an experiment.
On the other hand, if you want to compare some subtle difference between two competing parachute designs, for example, then perhaps an RCT would actually be appropriate. This is much closer to what is actually being measured in real clinical trials.
[1] To be completely fair to the authors, it was intended as a joke, not an argument. The real joke ended up being that people took it seriously.
We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".