> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that.
This screams systematic error and error propagation to me. It's possible that we don't need to up the p-value, we just need to make sure researchers aren't stupid and can properly account for all sources of errors. The problem is that's often an acquired skill over time, not something younger researchers typically think about, especially those who aren't multidisciplinary.
Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of errors, up to and including blatantly cherry-picking or fabricating results, where the focus should be. There's been a series of recent posts encouraging scientists (and, moreover, journals) to publish negative results, which is probably a good starting point to promote greater transparency and honesty in the paper publishing game.
This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis.
For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data".
It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5.
Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.)
This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
> What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data.
The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data.
I can see why elementary mistakes are so pervasive.
exactly wrong, it's the multidisciplinary scientists who don't get enough depth or rigor to come close to understanding sources of error and error propagation.
The thing I fear is underexperienced multidisciplinary researchers coming up with convoluted procedures that put their hands and eyes on their work in so many places that small sources of error have compounded themselves and entrenched themselves into nooks and crannies.
I haven't seen any reason to believe that being multidisciplinary makes this more likely. The whole problem is caused by the fact that most of the single-disciplinary researchers are using bad statistics.
Very few degree programs give scientists enough background in statistics and experimental rigor to prevent these problems.
no doubt, the single-disciplinary researchers use bad statistics, too. My broader point is that multidisciplinary researchers are less likely to use statistics correctly, if for no other reason than that the fact that they are multidisciplinary means that they are less likely to be able to handle depth from a personality/self-habits point of view.
Perhaps that's a little bit of projection, but I don't use any but the most rudimentary statistics in my publications and I don't make claims about significance.
It would be interesting to look into this. My experience has been that multi-disciplinary researchers are more likely to be familiar with a broader range of research and familiar with statistical techniques across their different disciplines. Someone with a narrower perspective is more likely to get into bad habits related to their field.
I think your comment about them being less likely to be able to handle depth from a personality/self-habits point of view is really projecting and probably based on experiences with a few bad apples. My experience doesn't reflect that at all.
I feel like a lot of the problems with statistics come from overly insular journals and conferences where everyone does it wrong and nobody realizes that because they're all doing the same thing. I've seen this in some computer science fields.
Also, I'm in chemistry and biology (don't get me started about biophysics, which is populated by dilettantes who couldn't hack it in physics thinking that biophysics is "physics" - it's not. It's chemistry)... So what it means to me to be multidisciplinary may be different from what it means to you.
People seriously just take standard deviations and then plug it into the formula to spit out p values. Without thinking about things like: "should the values be normally distributed? or log-normal?" "How does error propagate through this formula I'm using?" "Is the major source of error in the replicates that I'm using (versus something I might be normalizing to, like a mass measurement)?" "Am I in the linear range of my calibration standards?" "Is using these data quantitatively (versus qualitatively) an honest thing to do?"
I agree with the diagnosis. The big problem is that it is not possible to be sure that some study accounts for all possible confounding factors, unless you have a correct theory that allows you to determine what factors might be confounding factors. Statistical methodology alone cannot tell you this.
Reproducibility by researchers who know no more about the experimental setup than what was published is a better way of guarding against this than upping the significance hurdle. Experienced researchers may have a long history of ignoring confounding factors - I am not sure maturity matters much. I think the best time to have your complacency about constructing statistical investogations torn apart is while you are doing your PhD.
>In most fields, if there's less than a five percent chance that you'd get the two numbers by random chance, then you can reject chance—the results are considered significant. In statistical terms, this is called having a p value of less than 0.05.
The Ars definition of a p value is not precisely worded but, I think, reasonably accurate. It doesn't include the bit about also including the probability of obtaining results more extreme than what you obtained.
The best definition I know is
> The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.
S. N. Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130:995–1004, 1999.
edit: oh dear, and then the Ars article says "Individual experiments may be wrong five percent of the time," but that's exactly what p values do not measure. Statistics is hard.
The problem with upping the standard is, as the PNAS article acknowledges, that you need a larger sample size to produce any given result.
Unfortunately, many studies -- particularly those in medicine -- are already conducted with samples that are too small to detect any effect you'd reasonably expect to see, because many researchers do not calculate in advance what sample size would be required. This has interesting paradoxical effects: the only published studies are those that overestimate the size of the true effect.
So there's a tradeoff. Do you want to eliminate false positives at the cost of more false negatives? It's a difficult balance. I suspect there are many areas where poor statistical practice can be remedied to produce better results without greater expense.
I still think the answer is not in any particular standard in any particular field, rather it's outputting the final, worked datasets. Making the numbers themselves available and usable to the 'reader' or scientist following up will immediately clear up issues of reproducibility or not meeting statistical thresholds. If I see the same data, run it through my own processes and it looks like noise I'm likely to discount the results moving forward. There's nothing inherently wrong with saying 'look, I did this once, and this is what I saw'. And many times it's useful. It's less useful, but still reasonable in certain circumstances to say, 'look, I did this a hundred times and I saw this once'. There are occasions when the experiment cannot reasonably have more than a statistically insignificant 'n' - but it still might be useful to see what happens. What is not reasonable is to say, 'this is what I (sometimes) see always (trust me)' and hide your data behind a 'representative' jpg and a 'statistically significant' p-value.
As I scientist who has worked with both rich, and poor datasets from standard and invented datatypes, I'd just say, "let me see the data".
The real problem is that effect size is rarely discussed. P-values only relate to variability, sometimes high variability is acceptable, sometimes it's not. Taken by itself, a p-value is worthless, no matter how small it is. For example, suppose you develop a fertilizer that you're 99.99999% certain will produce one additional ear of corn in 10,000 bushels. Who cares!? You see a lot of that in published papers. A lot of researchers stop once they get a p-value below 0.05 and neglecting effect size is the norm.
Effect size is important, because all events in the real world except for true random numbers have some correlation.
Eating an apple might help a broken leg heal faster or slower. There is certainly an extremely small correlation, and with a sufficiently large number of controlled experiments, a result rejecting the null hypothesis with p < 0.05 will be found. The required number of experiments might be astronomically large, but if any correlation exists, it can be found with enough samples.
But without looking at the effect size, the result is useless. Even if eating an apple helps your broken leg heal 2 seconds faster on average, it is pointless to suggest this as medical advice.
In most areas (social sciences, medicine) there is no "standard" for statistical significance. In economics, people try to deal with the issue by looking at the robustness of a result. If a result remains when you make various changes to the specification of your model, then it is less likely to be a statistical artifact.
Another step in the right direction is greater reproducibility, so that at least people can play with your analysis and see if what you did was the most direct and natural analysis, or if there are clear signs of playing with the parameters until you get the result you want.
If professional scientists do genuine and not so genuine mistakes, imagine what programmers (who are generally clueless about statistics and theory of experiment) do with benchmark numbers!
My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific.
It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough.
It is like heaving open-source, but without any source code.
While I think availability of data and code is important, note that it does not accomplish many of the functions of replication of experiments. If two people implement the described algorithm and get the same results, we can be a lot more confident than if two people run the same implementation, because the same implementation is more likely to have the same bugs.
By this criteria, there is very little science going on anywhere. We don't teach philosophy of science enough anymore. In most of the fields that I'm familiar with, reproducibility is very hard and rarely bothered about by either the original researcher or others in the field.
Agreed, but I'm not sure the solution is upping the statistical standards so much as educating researchers about what the statistical standards actually mean, how to use them, and how to construct a valid, repeatable experiment.
Many researchers in less mathematically rigorous fields lack basic statistical knowledge, such as how a p-value is calculated or what a chi-squared test is. And that isn't to be snarky towards them - they have a lot to absorb in their own field. It just means that they are slower to adopt or even know about various mathematical techniques.
This reminds me of the med researcher who rediscovered integration in 1993 and was cited many, many times:
- Methodology - If you're mining for answers, and then presenting only the statistically significant, you will still find false positives on larger datasets, it will just take more work.
- Reproduction - If false positives were more fiercely chased down, then this would force researchers to be more careful. There is improvement along these lines.
Changing the p value is arbitrary, and may miss important results. I believe that encouraging reproduction of results, and reducing blind data mining is a better solution.
Isn't the solution just to put funding into reproducing results? It's fine to have a wide-filter for exploratory research, but a big problem is that it can be decades before anybody goes back to reproduce.
Let's assume that 90% of results are false positives. Using a higher power experiment with even the same standard of p=0.05 would result in rejecting most of those, while hopefully keeping the majority of the true positives (due to the higher power of the new test). This would result in going from 90% false positives to less than half false positives, a considerable improvement.
It just seems that nobody ever advanced their career by trying to reproduce even a landmark result in their field.
The issue in day to day science is that people will work their numbers to get p=0.05. They'll have a hypothesis, that, say, lactating adenomas confer a protective effect against DCIS. And they'll pull, say, all lactating adenoma cases from 2001 to 2013. If that doesn't work, they might actually try dropping the 2001 data and just using the 2002 to 2013 data if it gets them to p=0.05. The result is more brittle (we often test findings by asking "How would the p value change if we added one negative case?") but the alternative (pun intended) wouldn't otherwise be published.
This search for p=0.05 also leads to a lot of hair-splitting studies: take two diagnoses, say, usual and atypical ductal hyperplasia. Now, if you can find some constellation of parameters that define a middle category, say, "borderline ductal hyperplasia", you have a wide-open field to all sorts of p=0.05's, even if there's no change in treatment or outcome. You can say "cases previously characterized as UDH with a <parameter x> greater than <x> are 67% more likely to have <parameter y> (p=0.002)" because you lumped together a bunch of stuff that people already mostly agreed on anyway.
43 comments
[ 3.9 ms ] story [ 101 ms ] threadThis screams systematic error and error propagation to me. It's possible that we don't need to up the p-value, we just need to make sure researchers aren't stupid and can properly account for all sources of errors. The problem is that's often an acquired skill over time, not something younger researchers typically think about, especially those who aren't multidisciplinary.
For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data".
It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5.
Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.)
This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
It's got priiiiooooooors.
The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data.
I can see why elementary mistakes are so pervasive.
The thing I fear is underexperienced multidisciplinary researchers coming up with convoluted procedures that put their hands and eyes on their work in so many places that small sources of error have compounded themselves and entrenched themselves into nooks and crannies.
Very few degree programs give scientists enough background in statistics and experimental rigor to prevent these problems.
Perhaps that's a little bit of projection, but I don't use any but the most rudimentary statistics in my publications and I don't make claims about significance.
I think your comment about them being less likely to be able to handle depth from a personality/self-habits point of view is really projecting and probably based on experiences with a few bad apples. My experience doesn't reflect that at all.
I feel like a lot of the problems with statistics come from overly insular journals and conferences where everyone does it wrong and nobody realizes that because they're all doing the same thing. I've seen this in some computer science fields.
People seriously just take standard deviations and then plug it into the formula to spit out p values. Without thinking about things like: "should the values be normally distributed? or log-normal?" "How does error propagate through this formula I'm using?" "Is the major source of error in the replicates that I'm using (versus something I might be normalizing to, like a mass measurement)?" "Am I in the linear range of my calibration standards?" "Is using these data quantitatively (versus qualitatively) an honest thing to do?"
Reproducibility by researchers who know no more about the experimental setup than what was published is a better way of guarding against this than upping the significance hurdle. Experienced researchers may have a long history of ignoring confounding factors - I am not sure maturity matters much. I think the best time to have your complacency about constructing statistical investogations torn apart is while you are doing your PhD.
No it isn't [1].
[1]: https://en.wikipedia.org/wiki/P-value
The best definition I know is
> The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.
S. N. Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130:995–1004, 1999.
edit: oh dear, and then the Ars article says "Individual experiments may be wrong five percent of the time," but that's exactly what p values do not measure. Statistics is hard.
For the purposes of understanding the definition, another way of looking at it is basically a 'statistical proof by contradiction':
1. Assume null hypothesis is true
2. Compute test statistic
3. Ask the question, "What is the probability of obtaining that test statistic or one more extreme?" (this probability is the p-value)
4. Pick a threshold (usually 0.05 but this is totally arbitrary)
5. If p < threshold then conclude the null hypothesis is false and reject it.
Reductio ad statistico absurdum.
If you're just feeling your way around in the dark, 2-sigma is a useful way to work, so we use that to guide exploration.
Why 2-sigma? Well, it's twice as big as 1-sigma.
Experiment didn't go well, but you need a more-impressive result? Use a 90% confidence interval instead of 95%.
Unfortunately, many studies -- particularly those in medicine -- are already conducted with samples that are too small to detect any effect you'd reasonably expect to see, because many researchers do not calculate in advance what sample size would be required. This has interesting paradoxical effects: the only published studies are those that overestimate the size of the true effect.
http://www.refsmmat.com/statistics/power.html http://www.refsmmat.com/statistics/regression.html#truth-inf...
So there's a tradeoff. Do you want to eliminate false positives at the cost of more false negatives? It's a difficult balance. I suspect there are many areas where poor statistical practice can be remedied to produce better results without greater expense.
As I scientist who has worked with both rich, and poor datasets from standard and invented datatypes, I'd just say, "let me see the data".
Eating an apple might help a broken leg heal faster or slower. There is certainly an extremely small correlation, and with a sufficiently large number of controlled experiments, a result rejecting the null hypothesis with p < 0.05 will be found. The required number of experiments might be astronomically large, but if any correlation exists, it can be found with enough samples.
But without looking at the effect size, the result is useless. Even if eating an apple helps your broken leg heal 2 seconds faster on average, it is pointless to suggest this as medical advice.
Another step in the right direction is greater reproducibility, so that at least people can play with your analysis and see if what you did was the most direct and natural analysis, or if there are clear signs of playing with the parameters until you get the result you want.
It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough.
It is like heaving open-source, but without any source code.
This reminds me of the med researcher who rediscovered integration in 1993 and was cited many, many times:
http://care.diabetesjournals.org/content/17/2/152.abstract
- Methodology - If you're mining for answers, and then presenting only the statistically significant, you will still find false positives on larger datasets, it will just take more work.
- Reproduction - If false positives were more fiercely chased down, then this would force researchers to be more careful. There is improvement along these lines.
Changing the p value is arbitrary, and may miss important results. I believe that encouraging reproduction of results, and reducing blind data mining is a better solution.
Let's assume that 90% of results are false positives. Using a higher power experiment with even the same standard of p=0.05 would result in rejecting most of those, while hopefully keeping the majority of the true positives (due to the higher power of the new test). This would result in going from 90% false positives to less than half false positives, a considerable improvement.
It just seems that nobody ever advanced their career by trying to reproduce even a landmark result in their field.
This search for p=0.05 also leads to a lot of hair-splitting studies: take two diagnoses, say, usual and atypical ductal hyperplasia. Now, if you can find some constellation of parameters that define a middle category, say, "borderline ductal hyperplasia", you have a wide-open field to all sorts of p=0.05's, even if there's no change in treatment or outcome. You can say "cases previously characterized as UDH with a <parameter x> greater than <x> are 67% more likely to have <parameter y> (p=0.002)" because you lumped together a bunch of stuff that people already mostly agreed on anyway.