The basic research isn't what costs billions, of course.
If we've got "worthless cures" and a "waste [of] billions", it's because we've got:
1. pharmaceutical corporations incentivized to take any random chemical they might be able to make money off of by telling people it has a certain effect (and that isn't so useless that the FDA will literally tell them to stop selling it for being useless) and get them on the market; and
2. an FDA body incentivized to put billion-dollar requirements on corporations to research a chemical's safety by putting it through a gauntlet of trials, and to research said chemical's efficacy (i.e. whether it has some "statistically visible" effect on people), but without that efficacy test translating into anything like a chemical's marketable usefulness (i.e. it being the answer to any problem anyone actually has, such that a doctor would be independently motivated to prescribe it without the pharmacorp's encouragement.)
In other words, our system assumes that pharmacorps will only bother putting drugs on the market when they're useful. But that makes no sense: if peddling snake oil—snake oil the FDA has rubber-stamped as having "[statistically] noticeable effect" on some disease—is just as profitable as peddling cures, why would pharmacorps care about curing anything?
It's common knowledge that most nutritional supplements—especially the kind that are marketed as containing some named plant, rather than some named active ingredient—are useless, and often don't even contain what they claim to contain. We know exactly why the corporations who produce them produce them anyway: people are dumb enough to buy them, and the rules for things marketed as "nutritional supplements" don't prevent them from making a pill with literally no active ingredient.
Well, the logic for pharmacorps isn't any different. If the rules for things marketed as "drugs" allow them to get away with selling pills containing a chemical that just caused people to "show 1% improvement" along some symptom axis in studies, and doctors are dumb enough to [be nudged into] prescribing that pill—why wouldn't they sell them?
The research scientists who found out that the chemical might be beneficial in some way didn't cause any of this, any more than the people who first discovered coal in the ground caused coal power plant pollution. Corporate greed, and a government body with weak "efficacy" standards, are at fault here. The basic research—sloppy or not–is just (unintentional!) grist for their mill.
---
That being said: if you are a research scientist, and you want to do what you can to be a check on this system, making your basic research as rigorous as possible, so that they can make conclusions of no effect that the FDA can cite as reasons to reject a pharmacorp's pill—would be useful.
But, very likely, the pharmacorp has more money than your lab does, and so will be able to pay for a higher-powered study than you can afford, that will be sloppy, such that it could prove the opposite conclusion. Your voice, however clear and strong, might just be drowned out anyway.
Really, if we want to fix this problem, the fix needs to come from a different direction. Possibly FDA drug-approval reform, toward something more like a peer-review model (i.e. the pharmacorp gives the FDA money; the FDA uses that money to pay independent labs to rigorously replicate the efficacy study; and then the FDA trusts their independent labs over the pharmacorp's results.)
A tangent: interestingly, with anti-depressants in particular, doctors prescribe many of these "basically a placebo" drugs fully aware that they're basically-placebos, and indeed do so independently of any pharmacorp wining-and-dining. This is because the first-line treatment for depression is effectively "let's see if we can give you a placebo to get you to trick yourself out of being depressed." Which often works!
There are effective treatments for depression, mind you, but they have side-effects, while placebos are—unsurprisingly—very well-tolerated. So the "actually a drug" drugs get pushed back to the second-line or beyond.
Basically, doctors are treating depression the an ISP's phone support treats user complaints: they have a first-tier response that's effectively just there to help people who are temporarily confused and will resolve the problem themselves if you can talk to them long enough. You only get the "actual support" support once the phone-tree algorithm has verified that "placebo support" wasn't enough for you.
As a biomedical scientist at a solid US university, I'm getting pretty fed up with all the attacks on my field's credibility. It's similar to education where everyone moans about poor quality. While i cant speak to the average teacher I can honestly say the average researcher tries very hard to get it right. Remember most people don't become biomedical scientists unless they worked hard for many, many years in school and beyond for often quite terrible wages. So you can hate the game (drug companies, academic journals), but please think through any hate you might have on the player.
I don't think the blame can all be pinned on drug companies and academic journals, or whatever other third parties we can identify. They all have a role, but many problems with biomedical science come down to individual researchers making poor decisions about their experimental design, statistical methods, and computational tools.
I'm a statistician, and when I wrote my own book about bad statistics in science (see https://www.statisticsdonewrong.com/), I made sure to reference studies which quantify how often errors occur in real published research. The rate is stunningly high. The average biomedical experiment is conducted with (a) a sample size which is far too small to detect an effect of the expected size, (b) a vague analysis plan which leads to exploratory analyses with high false positive rates, (c) frequent copy-and-paste errors and math mistakes in presenting important results, and (d) an overreliance on statistical methods to make up for poor experimental design.
This means the average published paper is likely a false positive, likely an overestimate of the true effect if not, and is barely reproducible.
Is this the fault of individual scientists? Partly, yes -- these problems have been pointed out for years in leading journals, but nobody takes action to do better research. It's also the fault of the grant funding systems which incentivize salami-slicing of results instead of doing one big, rigorous, well-designed study, and of journals which prefer dramatic but unreliable results over mundane but well-executed results. (Of course, the journal editors and reviewers are usually active scientists themselves.) I think the average researcher would like to "get it right", but has to focus on getting a career instead.
I think the central issue is that in many subfields of biology, reproducibility is seen as infeasible or unnecessary, so even with the best of intentions researchers can go on fooling themselves about the reliability of their results for years. Plus of course, without the accountability afforded by reproducible results the bad money can end up driving out the good in funding decisions, and then the good intentions get driven out, too.
Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a...
On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.
I actually cited the seminal cancer-biology editorial in the GP. I suppose cancer biology is often a form of molecular biology, but I meant more fundamental research into molecular structure and biological function, and in-vitro methods for querying and modifying the structure. CRISPR is a good example.
CRISPR is a great example. Please see my other post on that topic in this thread, and if you are interested I will go find the refs that led me to that position.
I would certainly be interested to know the references, but I don't find your argument here convincing. Testing for insertion of a defensive gene by exposing the organism to a hostile environment is an extremely well-developed and reliable method in genetics. But I suppose there must be a million ways to screw it up so I can't rule your mutation scenario out based on what I currently know of the research.
On the other hand, I find your skepticism and independent thinking refreshing.
I see now I had an earlier discussion with tstactplsignore, back then they were seemingly incapable of understanding what I was saying (they kept thinking I denied the endonuclease activity).
http://www.nature.com/nchembio/journal/v10/n8/abs/nchembio.1... claims that resequencing of a CRISPR-targeted gene showed dramatic increases in indel mutations, consistent with non-homologous end joining repair of the CRISPR cuts. (Supplementary table 5.) I assume the resequencing happened before the drug screen, otherwise the paper is badly written.
Well, from your table S5 and table S12, as is usual we see that about 0.1 - 1% of the cells were mutants in the control group (ie they came pre-"modified").
Interestingly, from table S12 and my (selection for pre-existing mutants) model we can also explain a mysterious result they observed:
"Silent co-mutations in the repair oligonucleotides were introduced into >99% of D54H mutant alleles and ~3% of the F482S allele (Fig. 2c). Carryover of these silent SNPs indicates that the alleles are the product of HR and not de novo mutagenesis. The lower rate of coappearance of silent SNPs in F482S is presumably due to the larger distance between the two SNPs in the oligonucleotide and is rather common to see with single-stranded oligonucleotide donors19."
Rather than that ad hoc explanation, it is simply that the D54H cells had more silent only mutations to begin with (0.1% vs 0.0%). Regarding that 0.0%, an annoying thing is that they only report these percentages to one decimal place.
I'm not exactly clear on the number of cells present before the CRISPR-Cas9 treatment, but it sounds like 10^8, and then they let them grow for 72 hr + 7-12 days (total of 10-15 days) after the treatment. They also don't tell us how many cells were left at the end... but anyway if we assume these cells divide once a day, and 0.1% are preexisting mutants we could calculate the possible number of mutants thus:
Nt = 10^8
p = 0.001
d = 0:15
Nt*p*2^d
After 12 days we can get ~400 million cells from those initial pre-existing mutants, and after 15 days over as 3 billion. Of course other factors would probably come into play that limit this growth, I'm just saying it would be no problem for that small subset of the population to become dominant during the experiment. That is even if the 99,900,000 "WT" cells were just growth arrested rather than died.
So I find those results to favor the "selection for pre-existing mutants" explanation over the "gene modification" one.
The key observation from my perspective is that they seem to have done the resequencing before any resistance selection. Why would the indel frequencies rise only in the CRISPR-treated cells during drug-free growth?
>"The key observation from my perspective is that they seem to have done the resequencing before any resistance selection. Why would the indel frequencies rise only in the CRISPR-treated cells during drug-free growth?"
Because in cells that contain the target sequence (the complement to the guide RNA), Cas9 is damaging the DNA, leading to cell death and growth arrest. The small percentage of cells that already contained indels (thus reducing affinity for the guide RNA) are "immune", so they preferentially survive and divide to take over the population.
I assume the Cas9 endonuclease activity is the cause of the toxicity... The theory does not require that though, only that the treatment kills/damages the cells containing the target sequence.
I still don't understand. What is wrong with the mechanism of double stranded breaks leading to cell death and/or growth arrest?
Also, it is 100% possible to create a testable model of something at the level of toxicity without knowing the details of the toxicity. I mean, here would be a simple one (written in R) where the wild type cells divide at 1/10th the rate of the mutants for some reason, so the mutants take over the population:
# Basic parameters of the cell culture
Ntotal = 10^8 # Initial total number of cells
Pmut0 = 0.001 # Propotion pre-existing mutants
Rdiv_wt = 0.1 # Divisions/day
Rdiv_mut = 1.0 # Divisions/day
# Calculate initial numbers of WT and mutant cells
Nwt0 = Ntotal*(1-Pmut0)
Nmut0 = Ntotal*Pmut0
# Convert between days and divisions
t = 0:15
Dwt = Rdiv_wt*t
Dmut = Rdiv_mut*t
# Calculate number of cells at each timepoint
Nt_wt = Nwt0*2^Dwt
Nt_mut = Nmut0*2^Dmut
# Calculate proportion of mutant cells in population at each timepoint
Pt_mut = Nt_mut/(Nt_wt + Nt_mut)
# Plot proportion of mutants vs time
plot(t, Pt_mut, type = "b", panel.first = grid(),
xlab = "Days Since Treatment",
ylab = "Proportion of Mutant Cells")
If the parameters are known accurately enough (initial number of cells, initial proportion of mutants, division rates, etc) this is a perfectly testable quantitative model.
>"What is your proposed selection mechanism for pre-existing indels, given CRISPR treatment?"
This was explained earlier I believe, so I am not sure where the confusion lies.
1) You start with a mixed population of cells. From the literature it looks like about 99-99.9% will lack indels at the target site, the rest have them.
2) The Cas9 will cut the DNA of cells lacking indels at that locus (ie the wt cells containing a sequence complementary to the guide RNA), thus killing and/or growth arresting those cells.
3) Meanwhile the cells with indels will continue living and proliferating since they lack the target sequence. These are "immune" to the CRISPR-induced damage.
Thus the proportion of WT cells will decrease, while the "mutants" will increase. It may help to play with the code of the simple model I shared earlier.
> The Cas9 will cut the DNA of cells lacking indels at that locus
So you're stipulating the consensus understanding of Cas9's initial action, but arguing that this results in cell death rather than nonhomologous end joining repair?
In general, yes (except you left out "growth arrest"). I don't see how their experiments can tell the difference between the two mechanisms, or some combination of them. Also, since mine is simpler, it probably dominates.
This tells us the evidence for a given theory given the observations depends on the ratio of two things:
1) How well the observations fit a theory (you can substitute hypothesis/model/etc) and how plausible a theory would be without the observations in question.
2) The above for all other theories
In other words, the default for the scientist is to be skeptical of any explanation until the others have been rendered implausible (ie ruled out). There is a bit more to it (eg the Pr(O|T[i]) terms depend on the precision of the predictions, which remain vague in the case of NHEJ despite generous funding), but that is pretty much it.
It appears that your prior is uninformed by the extensive research into NHEJ, and the other successful genetic engineering methods which ate based on it.
>"It appears that your prior is uninformed by the extensive research into NHEJ, and the other successful genetic engineering methods which ate based on it."
I don't see how it is even possible for you to gather that from what I have said? All you could possibly have is a rough estimate of the ratio between the priors for NHEJ vs selection. Selection is a far more common and well studied process...
There also exist simple quantitative models of that process, which allow precise predictions, something lacking in the case of NHEJ models afaik, which must remain vague. So the likelihoods will also be narrower in the selection case.
Anyway, it was productive to discuss the specific paper and model, but is now getting philosophical and pointless. I only mentioned the scientific thought process because you asked why I would be skeptical.
I'm afraid you've fallen into the same kind of quantitative cargo-cult reasoning which led people astray in their applications of NHST.
Quantitative reasoning can give you a lot of leverage in systematizing and inferring from a body of knowledge, but if your reasoning doesn't start from that knowledge it will lead you nowhere. In this case, the necessary knowledge is the structure and mechanism of DNA repair.
When Dirichlet computed the probability that the Sun wouldn't rise tomorrow, given a flat two-event Dirichlet prior and the observation that it had risen every day for the last 6,000 years, he added that of course, for people who understood the workings of the solar system, the probability is far, far, lower. Your arguments here are like that. They just don't take into account the relevant facts of molecular biology.
1) I do indeed deny you need to know the exact mechanism of toxicity to model it. In fact, having any kind of quantitative model is better than the qualitative A increases/decreases B stuff that is usual. The reason being that it is too vague and many explanations can account for such an observation.
2) But please, let us skip the philosophy and talk directly about this topic, because that argument is not even necessary. It sounds to me like you do not believe that double strand breaks can lead to cell death and growth arrest? You find this implausible? I am really surprised that this is an objection.
Your links are talking about preclinical trials and attempts to replicate specific high profile, controversial studies from specific subfields.
We're talking about the entirety of basic research. The "evidence" for this is the incredible advances in our understanding of how the cell and the molecules of life work in the last 50 years. Every single experiment done in the modern life sciences would not be even possible to contemplate doing if all of the molecular biology it is based on was not extremely reproducible. Entire fields of modern biology like genomics, structural biology, genome editing, and more could not even be fathomed to exist if this were not true.
Nobody tallies up how often biology "works", only how often a few experiments attempting to prove specific hypotheses don't work. Basic science in molecular biology and microbiology are extremely reproducible most of the time and nobody writing these articles questions that.
No offense, but this is some creationist-tier science denial you have going on if you don't believe this, and it isn't the responsibility of the scientific community to educate people who've read one time many articles about reproducibility that science works.
>"Entire fields of modern biology like genomics, structural biology, genome editing, and more could not even be fathomed to exist if this were not true."
Yes, everything needs to be redone since at least the 1980s, and probably the 1940s (whenever NHST became the primary method of assessing what is correct or not). We have no idea what is actually going on. It is scary, I know. It took me years to accept.
>"this is some creationist-tier science denial you have going on if you don't believe this"
Ok, link to a specific paper and I will discuss it with you. In my experience, if it is biomed, no one will have ever published a replication of any results it contains. There will be no model capable of quantitative prediction of anything, and conclusions drawn will also almost certainly be fatally flawed due to the vagueness of their explanation.
This is ridiculous. Your views on NHST are extremist, absurd, and are not shared by pretty much all real statisticians, even those who are critical of frequentist methods.
Secondly, many biological findings are not quantitative or statistical in nature- they simply are observations that have been repeated tens of thousands of times.
Honestly I don't really care enough to respond over something that nobody else in the world really believes. Your claim that you were "trained in biomedicine" is cute but difficult to take seriously. It isn't worth my or anyone else's time to fight your bizarre and excessive ignorance on this topic. I just think it is important to point out to the community that you have zero credibility on this issue and are spouting nonsense that real statisticians, scientists, and computational biologists would laugh at.
Of course I have zero credibility, I am a random person on the internet. But since you know I am so wrong, it should have taken less effort for you to link a paper than to write that post...
Anyway, I agree it is pointless to continue a discussion at this level, which is why I tried to guide us to talking about specific findings (of your choice).
Lol, I just realized you were that guy who denies CRISPR-Cas9 genome editing really works. It may be against the rules, but I just have to tell you that you are an insufferable idiot who brings down the quality of discourse on this site with your bizarro pseudoscience nonsense.
This is a strawman. What I have seen in the data presented by the CRISPR-Cas9 researchers seems consistent with the idea that ~1% to 0.1% of cells in culture are mutants at any given site, the treatment selectively kills or growth arrests the non-mutants (at that site), and then the surviving (because they lacked the target sequence) pre-existing mutants divide and multiply to take over the population.
I believe this may account for the vast majority of the "editing" they are measuring, and claims of increased efficiency over earlier methods (eg TALEN) are due to this mechanism. They have not done proper due diligence of ruling out other explanations and prematurely starting running with their favorite, most hype-able, hypothesis.
Can you find one paper where they report the efficiency of inserting a cassette via HDR, or do they all only look at NHEJ when talking about that? I have only seen the latter.
Also, this alternative explanation has real-life consequences. If correct, the very mechanism via which CRISPR-Cas9 works is toxic, meaning all the hundreds of millions of dollars (billions?) currently being spent trying to make it less toxic will be wasted.
You do realize that the power of Cas9 is not it's ability to cut or even induce non-homologous-end-joining, but rather to co-localize with an exact sequence of DNA (that can be changed in a day for a few dollars).
Cutting DNA is certainly a great self-defense mechanism used by the bacteria that causes strep throat to attack other bacteria. But in the ways that Cas9 will eventually be used, likely its entire endonuclease activity will be stripped, leaving only its homing ability. And for that you can swap in any other useful protein functions that now happen at a particular sequence location. And that is what makes Cas9 so exciting. That NHEJ works at all is just a bonus to get us to some early results quickly.
fCas9 or dCas9, or a nickase (different endonuclease, and endonuclease-dead respectively) as well as a host of other variants actually hold great promise precisely and solely in their ability to locate a sequence.
That may be. My explanation also requires relatively precise co-localization. Assuming the toxicity is only due to the endonuclease activity, those plans shouldn't be affected either way.
I appreciate your comments but as the biologist hopefully you can understand that it's not like we are trying to keep our sample sizes small. We understand statistical power. It's simply not feasible logistically and financially to have the ideal numbers a lot of the time. Real life intervenes, so we do our best. For math/computer people which is probably many here in HN, I think they have a hard time with real-world messiness that is biomedical research. It's the same mentality of "oh I'll just apply my algorithms to this..." which works for them but not so much for us.
Given how infrequently we see power calculations in published papers, and the results of typical surveys of scientists, I don't think the typical scientist does understand statistical power or other statistical concepts. There are plenty of surveys (e.g. http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885...) showing that scientists just pick sample sizes based on what's usually done in their field, rather than working out what's necessary. There are plenty of other surveys showing practicing scientists don't understand the concepts of significance and power. Some of the authors of those studies don't understand them either -- I've seen at least one study with questions with no correct answer...
I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results.
I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.
>I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day.
A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive).
You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better.
The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.
I was trained in biomed and found it completely unsatisfactory. Simply put, I was trained to do pseudoscience (NHST, where you test a strawman "null" hypothesis rather than your own hypothesis and commit an egregious affirming the consequent error). There are also various other institutionalized satellite issues like the extreme difficulty of publishing replication results.
Consider these questions:
1) Can you define a p-value and articulate why you would perform a significance test?
2) How many replication studies have you published?
3) How often do you perform your experiments blinded to intervention?
4) Is there any quantitative prediction that has been made by anyone in your sub-field of research ever? I'm not even asking for an accurate one, just the publication of a point or small-interval prediction.
There are other ways to do biomedical science than p-hacking at large data sets.
Some of the best science I've seen were experiments where the result was guaranteed to be one of a few different possibilities - and therefore any concrete result was interesting. That's mostly attributable to good experimental design. Cell A does X under condition 1. Cell A does Y under condition 2. What is the protein/pathway/atom/molecule responsible for triggering condition Y under condition 2 (or conversely triggering X under condition 1). Pick your level of inquiry, discover the answer, remove the answer to test for necessity, and add the answer back to a null background to test for sufficiency. If all are true, you have no found something useful and true. Statistics was only a tool to help you accomplish the above and actually had little to no bearing on the knowledge being created. Good experimental design could create situations where the culminating experiment either presented an indicator or did not, confirming the questioned hypothesis true or false.
Those kinds of factual discoveries were the biomedical science I was taught to probe for, where whatever the answer was it would be significant, interesting, and useful. And there is no way to successfully conduct the above experiments in such a way that statistics even can be of much relevance to the scientific question itself. Either you discover a mechanism or you do not, and logically it can only be one of a very few options.
From this description I wouldn't buy this book. It is focusing on a bunch of minor issues rather than the main problem. Biomed researchers only come up with vague "A makes B go up/down" hypotheses. This is not because biology is more complicated than physics. It is 100% a culture and training issue.
Agree with that. The problem is not just the inaccurate answers, but the questions are often ill-thought as well. Life scientists often make correlations from molecules to behavior or phenotype, an inference that spans many many orders of spatial magnitude and a gazillion of complex processes. There is just not enough of incentive for systematic bottom-up approaches, as the average grant size is made to fit this kind of hypotheses. It's more culture than training, or rather, training is a consequence of the culture of the field.
It's more like: if you lost your keys on your way home, you go and look for them under the lamp post because you're drunk and you can only see where the light is.
49 comments
[ 2.0 ms ] story [ 143 ms ] threadIf we've got "worthless cures" and a "waste [of] billions", it's because we've got:
1. pharmaceutical corporations incentivized to take any random chemical they might be able to make money off of by telling people it has a certain effect (and that isn't so useless that the FDA will literally tell them to stop selling it for being useless) and get them on the market; and
2. an FDA body incentivized to put billion-dollar requirements on corporations to research a chemical's safety by putting it through a gauntlet of trials, and to research said chemical's efficacy (i.e. whether it has some "statistically visible" effect on people), but without that efficacy test translating into anything like a chemical's marketable usefulness (i.e. it being the answer to any problem anyone actually has, such that a doctor would be independently motivated to prescribe it without the pharmacorp's encouragement.)
In other words, our system assumes that pharmacorps will only bother putting drugs on the market when they're useful. But that makes no sense: if peddling snake oil—snake oil the FDA has rubber-stamped as having "[statistically] noticeable effect" on some disease—is just as profitable as peddling cures, why would pharmacorps care about curing anything?
It's common knowledge that most nutritional supplements—especially the kind that are marketed as containing some named plant, rather than some named active ingredient—are useless, and often don't even contain what they claim to contain. We know exactly why the corporations who produce them produce them anyway: people are dumb enough to buy them, and the rules for things marketed as "nutritional supplements" don't prevent them from making a pill with literally no active ingredient.
Well, the logic for pharmacorps isn't any different. If the rules for things marketed as "drugs" allow them to get away with selling pills containing a chemical that just caused people to "show 1% improvement" along some symptom axis in studies, and doctors are dumb enough to [be nudged into] prescribing that pill—why wouldn't they sell them?
The research scientists who found out that the chemical might be beneficial in some way didn't cause any of this, any more than the people who first discovered coal in the ground caused coal power plant pollution. Corporate greed, and a government body with weak "efficacy" standards, are at fault here. The basic research—sloppy or not–is just (unintentional!) grist for their mill.
---
That being said: if you are a research scientist, and you want to do what you can to be a check on this system, making your basic research as rigorous as possible, so that they can make conclusions of no effect that the FDA can cite as reasons to reject a pharmacorp's pill—would be useful.
But, very likely, the pharmacorp has more money than your lab does, and so will be able to pay for a higher-powered study than you can afford, that will be sloppy, such that it could prove the opposite conclusion. Your voice, however clear and strong, might just be drowned out anyway.
Really, if we want to fix this problem, the fix needs to come from a different direction. Possibly FDA drug-approval reform, toward something more like a peer-review model (i.e. the pharmacorp gives the FDA money; the FDA uses that money to pay independent labs to rigorously replicate the efficacy study; and then the FDA trusts their independent labs over the pharmacorp's results.)
---
A tangent: interestingly, with anti-depressants in particular, doctors prescribe many of these "basically a placebo" drugs fully aware that they're basically-placebos, and indeed do so independently of any pharmacorp wining-and-dining. This is because the first-line treatment for depression is effectively "let's see if we can give you a placebo to get you to trick yourself out of being depressed." Which often works!
There are effective treatments for depression, mind you, but they have side-effects, while placebos are—unsurprisingly—very well-tolerated. So the "actually a drug" drugs get pushed back to the second-line or beyond.
Basically, doctors are treating depression the an ISP's phone support treats user complaints: they have a first-tier response that's effectively just there to help people who are temporarily confused and will resolve the problem themselves if you can talk to them long enough. You only get the "actual support" support once the phone-tree algorithm has verified that "placebo support" wasn't enough for you.
I'm a statistician, and when I wrote my own book about bad statistics in science (see https://www.statisticsdonewrong.com/), I made sure to reference studies which quantify how often errors occur in real published research. The rate is stunningly high. The average biomedical experiment is conducted with (a) a sample size which is far too small to detect an effect of the expected size, (b) a vague analysis plan which leads to exploratory analyses with high false positive rates, (c) frequent copy-and-paste errors and math mistakes in presenting important results, and (d) an overreliance on statistical methods to make up for poor experimental design.
This means the average published paper is likely a false positive, likely an overestimate of the true effect if not, and is barely reproducible.
Is this the fault of individual scientists? Partly, yes -- these problems have been pointed out for years in leading journals, but nobody takes action to do better research. It's also the fault of the grant funding systems which incentivize salami-slicing of results instead of doing one big, rigorous, well-designed study, and of journals which prefer dramatic but unreliable results over mundane but well-executed results. (Of course, the journal editors and reviewers are usually active scientists themselves.) I think the average researcher would like to "get it right", but has to focus on getting a career instead.
Just a few papers on the problem of poor sample sizes in biomedicine: http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://rsos.royalsocietypublishing.org/content/4/2/160254 http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html
Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a...
On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.
Where is the evidence of this? There is plenty saying otherwise:
http://www.sciencemag.org/news/2015/06/study-claims-28-billi...
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/
http://www.slate.com/articles/health_and_science/future_tens...
http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm...
http://www.reuters.com/article/us-science-cancer-idUSBRE82R1...
http://www.nature.com/nature/journal/v483/n7391/full/483531a...
http://www.sciencemag.org/content/348/6242/1411
http://journals.plos.org/plosbiology/article?id=10.1371/jour...
http://www.nature.com/news/cancer-reproducibility-project-sc...
http://www.sciencemag.org/news/2017/01/rigorous-replication-...
On the other hand, I find your skepticism and independent thinking refreshing.
I see now I had an earlier discussion with tstactplsignore, back then they were seemingly incapable of understanding what I was saying (they kept thinking I denied the endonuclease activity).
Interestingly, from table S12 and my (selection for pre-existing mutants) model we can also explain a mysterious result they observed:
"Silent co-mutations in the repair oligonucleotides were introduced into >99% of D54H mutant alleles and ~3% of the F482S allele (Fig. 2c). Carryover of these silent SNPs indicates that the alleles are the product of HR and not de novo mutagenesis. The lower rate of coappearance of silent SNPs in F482S is presumably due to the larger distance between the two SNPs in the oligonucleotide and is rather common to see with single-stranded oligonucleotide donors19."
Rather than that ad hoc explanation, it is simply that the D54H cells had more silent only mutations to begin with (0.1% vs 0.0%). Regarding that 0.0%, an annoying thing is that they only report these percentages to one decimal place.
I'm not exactly clear on the number of cells present before the CRISPR-Cas9 treatment, but it sounds like 10^8, and then they let them grow for 72 hr + 7-12 days (total of 10-15 days) after the treatment. They also don't tell us how many cells were left at the end... but anyway if we assume these cells divide once a day, and 0.1% are preexisting mutants we could calculate the possible number of mutants thus:
After 12 days we can get ~400 million cells from those initial pre-existing mutants, and after 15 days over as 3 billion. Of course other factors would probably come into play that limit this growth, I'm just saying it would be no problem for that small subset of the population to become dominant during the experiment. That is even if the 99,900,000 "WT" cells were just growth arrested rather than died.So I find those results to favor the "selection for pre-existing mutants" explanation over the "gene modification" one.
Because in cells that contain the target sequence (the complement to the guide RNA), Cas9 is damaging the DNA, leading to cell death and growth arrest. The small percentage of cells that already contained indels (thus reducing affinity for the guide RNA) are "immune", so they preferentially survive and divide to take over the population.
Also, it is 100% possible to create a testable model of something at the level of toxicity without knowing the details of the toxicity. I mean, here would be a simple one (written in R) where the wild type cells divide at 1/10th the rate of the mutants for some reason, so the mutants take over the population:
If the parameters are known accurately enough (initial number of cells, initial proportion of mutants, division rates, etc) this is a perfectly testable quantitative model.This was explained earlier I believe, so I am not sure where the confusion lies.
1) You start with a mixed population of cells. From the literature it looks like about 99-99.9% will lack indels at the target site, the rest have them.
2) The Cas9 will cut the DNA of cells lacking indels at that locus (ie the wt cells containing a sequence complementary to the guide RNA), thus killing and/or growth arresting those cells.
3) Meanwhile the cells with indels will continue living and proliferating since they lack the target sequence. These are "immune" to the CRISPR-induced damage.
Thus the proportion of WT cells will decrease, while the "mutants" will increase. It may help to play with the code of the simple model I shared earlier.
So you're stipulating the consensus understanding of Cas9's initial action, but arguing that this results in cell death rather than nonhomologous end joining repair?
1) How well the observations fit a theory (you can substitute hypothesis/model/etc) and how plausible a theory would be without the observations in question.
2) The above for all other theories
In other words, the default for the scientist is to be skeptical of any explanation until the others have been rendered implausible (ie ruled out). There is a bit more to it (eg the Pr(O|T[i]) terms depend on the precision of the predictions, which remain vague in the case of NHEJ despite generous funding), but that is pretty much it.
I don't see how it is even possible for you to gather that from what I have said? All you could possibly have is a rough estimate of the ratio between the priors for NHEJ vs selection. Selection is a far more common and well studied process...
There also exist simple quantitative models of that process, which allow precise predictions, something lacking in the case of NHEJ models afaik, which must remain vague. So the likelihoods will also be narrower in the selection case.
Anyway, it was productive to discuss the specific paper and model, but is now getting philosophical and pointless. I only mentioned the scientific thought process because you asked why I would be skeptical.
Quantitative reasoning can give you a lot of leverage in systematizing and inferring from a body of knowledge, but if your reasoning doesn't start from that knowledge it will lead you nowhere. In this case, the necessary knowledge is the structure and mechanism of DNA repair.
When Dirichlet computed the probability that the Sun wouldn't rise tomorrow, given a flat two-event Dirichlet prior and the observation that it had risen every day for the last 6,000 years, he added that of course, for people who understood the workings of the solar system, the probability is far, far, lower. Your arguments here are like that. They just don't take into account the relevant facts of molecular biology.
2) But please, let us skip the philosophy and talk directly about this topic, because that argument is not even necessary. It sounds to me like you do not believe that double strand breaks can lead to cell death and growth arrest? You find this implausible? I am really surprised that this is an objection.
We're talking about the entirety of basic research. The "evidence" for this is the incredible advances in our understanding of how the cell and the molecules of life work in the last 50 years. Every single experiment done in the modern life sciences would not be even possible to contemplate doing if all of the molecular biology it is based on was not extremely reproducible. Entire fields of modern biology like genomics, structural biology, genome editing, and more could not even be fathomed to exist if this were not true.
Nobody tallies up how often biology "works", only how often a few experiments attempting to prove specific hypotheses don't work. Basic science in molecular biology and microbiology are extremely reproducible most of the time and nobody writing these articles questions that.
No offense, but this is some creationist-tier science denial you have going on if you don't believe this, and it isn't the responsibility of the scientific community to educate people who've read one time many articles about reproducibility that science works.
Yes, everything needs to be redone since at least the 1980s, and probably the 1940s (whenever NHST became the primary method of assessing what is correct or not). We have no idea what is actually going on. It is scary, I know. It took me years to accept.
>"this is some creationist-tier science denial you have going on if you don't believe this"
Ok, link to a specific paper and I will discuss it with you. In my experience, if it is biomed, no one will have ever published a replication of any results it contains. There will be no model capable of quantitative prediction of anything, and conclusions drawn will also almost certainly be fatally flawed due to the vagueness of their explanation.
Secondly, many biological findings are not quantitative or statistical in nature- they simply are observations that have been repeated tens of thousands of times.
Honestly I don't really care enough to respond over something that nobody else in the world really believes. Your claim that you were "trained in biomedicine" is cute but difficult to take seriously. It isn't worth my or anyone else's time to fight your bizarre and excessive ignorance on this topic. I just think it is important to point out to the community that you have zero credibility on this issue and are spouting nonsense that real statisticians, scientists, and computational biologists would laugh at.
Anyway, I agree it is pointless to continue a discussion at this level, which is why I tried to guide us to talking about specific findings (of your choice).
I believe this may account for the vast majority of the "editing" they are measuring, and claims of increased efficiency over earlier methods (eg TALEN) are due to this mechanism. They have not done proper due diligence of ruling out other explanations and prematurely starting running with their favorite, most hype-able, hypothesis.
Can you find one paper where they report the efficiency of inserting a cassette via HDR, or do they all only look at NHEJ when talking about that? I have only seen the latter.
Also, this alternative explanation has real-life consequences. If correct, the very mechanism via which CRISPR-Cas9 works is toxic, meaning all the hundreds of millions of dollars (billions?) currently being spent trying to make it less toxic will be wasted.
Cutting DNA is certainly a great self-defense mechanism used by the bacteria that causes strep throat to attack other bacteria. But in the ways that Cas9 will eventually be used, likely its entire endonuclease activity will be stripped, leaving only its homing ability. And for that you can swap in any other useful protein functions that now happen at a particular sequence location. And that is what makes Cas9 so exciting. That NHEJ works at all is just a bonus to get us to some early results quickly.
fCas9 or dCas9, or a nickase (different endonuclease, and endonuclease-dead respectively) as well as a host of other variants actually hold great promise precisely and solely in their ability to locate a sequence.
I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results.
I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.
A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive).
You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better.
The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.
Consider these questions:
1) Can you define a p-value and articulate why you would perform a significance test?
2) How many replication studies have you published?
3) How often do you perform your experiments blinded to intervention?
4) Is there any quantitative prediction that has been made by anyone in your sub-field of research ever? I'm not even asking for an accurate one, just the publication of a point or small-interval prediction.
Some of the best science I've seen were experiments where the result was guaranteed to be one of a few different possibilities - and therefore any concrete result was interesting. That's mostly attributable to good experimental design. Cell A does X under condition 1. Cell A does Y under condition 2. What is the protein/pathway/atom/molecule responsible for triggering condition Y under condition 2 (or conversely triggering X under condition 1). Pick your level of inquiry, discover the answer, remove the answer to test for necessity, and add the answer back to a null background to test for sufficiency. If all are true, you have no found something useful and true. Statistics was only a tool to help you accomplish the above and actually had little to no bearing on the knowledge being created. Good experimental design could create situations where the culminating experiment either presented an indicator or did not, confirming the questioned hypothesis true or false.
Those kinds of factual discoveries were the biomedical science I was taught to probe for, where whatever the answer was it would be significant, interesting, and useful. And there is no way to successfully conduct the above experiments in such a way that statistics even can be of much relevance to the scientific question itself. Either you discover a mechanism or you do not, and logically it can only be one of a very few options.
https://bml.bioe.uic.edu/BML/Stuff/Stuff_files/biologist%20f...
https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...