Imagine if p <= 1.0 was considered "significant", wouldn't this cut down on publication bias? Shouldn't every study worth funding be worth publishing? They should all be considered significant results. So I think this proposal is:
1) Not really addressing the issues behind the replication crisis
2) Doing the exact opposite of what should be done (raise the significance level)
"Significance" is a social concept currently serving two roles: for determining what is considered publishable, and for determining what is considered true. This is meant to modify the second one, not the first. (Publishability should depend on quality of methodology, and be determined at the pre-registration phase before results are in.)
First, I agree that publishability should be independent of the results. Do you know anyone who disagrees with this though? That seems to be the standard opinion, yet the situation of filtering by p-value persists.
Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis.
I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
Don't be disingenuous. That would require someone to actually read their papers, which would already be a net loss and a massive waste of time for everyone involved. Not to mention the amount of department politics which could be involved in this issue.
Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good.
The only issue is that it is a much weaker constraint on publishing than it was previously thought.
And not publishing wastes everyone's time. When researches perform the same experiment again and again until someone gets a significant result to publish. There are entire fields of research supported by nothing but publication bias (e.g. Parapsychology and possibly most soft sciences.)
Eg. The Journal of Wacky Speculation publishes "The Moon is Made of Green Cheese". Using a highly sophisticated cheese detector, the authors measure results consistent with this hypothesis. They calculate it would happen only 4% of the time if the moon were not in fact made of green cheese, and therefore the result is significant. The cheese detector works perfectly, the calculations were accurate, but this does not mean the moon is made of green cheese! We've been there, and we're very certain that it's made of rock. Because the prior probability of the hypothesis was so low, the "significant" result was most likely a fluke. The Journal of Wacky Speculation specializes in this type of hypothesis, so they should use a much lower threshold for p.
Meanwhile, the Sensible and Boring Society publishes a study showing that round wheels roll better on ordinary roads than square wheels. We already had good reason to believe this, so p<0.05 shows it's most likely true. The Sensible and Boring Society publishes tests of hypotheses with higher prior probably, so they can use a higher p threshold.
> [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses.
Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1]
This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05.
> a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses
If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish.
Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
> it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
I agree that doing something is better than doing nothing, but I disagree that this will reduce bad results in a meaningful way. Setting a more stringent threshold for significance will elimnate studies that reach the wrong conclusion due to a statistical fluke, i.e. there's no effect but due to random chance, the noise in the data makes it look like there is. But IMO these are relatively rare. It seems to me that the vast majority of bad studies reach significance by virtue of methodological issues that enable one to reach any arbitrary level of significance just by trying hard enough. If Daryl Bem continued doing what he was doing, he would have eventually reached p<0.005.
edit: I just glanced over Bem's paper and he reports several p-values under 0.005 and several other very close to it.
There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.
I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery.
Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2].
Most of them are of the form "p > 0.05" or
p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good).
That is just whatever smoothing ggplot is doing not getting truncated. There were p-values greater than 1 in that dataset though, so it does need cleaning.
It's worth noting the datasets (except Brodeur for economics) include "less/greater than", e.g. p < .05
This is the reason for the bimodality, and also for economics missing the second peak
due to some difference in how economics does statistics, they don't use that specific statement, so it doesn't show up as a peak.
>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.
Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist.
Say my null hypothesis is that X=100. The alternative hypothesis is X>100.
What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result.
There are ways to fix this, but one should just be aware, though.
> Say my null hypothesis is that X=100. The alternative hypothesis is X>100.
> What if in reality X is really 100.5?
Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.)
> Depending on the problem domain, this may well be the same as the null hypothesis.
No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions.
> But a larger sample size is much more likely to give a significant result.
Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
>Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.)
Mathematically, yes. Practically, not always.
In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis.
But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it.
The math will be correct, and the conclusion will be wrong.
P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size.
>P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size
If you never extrapolate practical significance from statistical significance, then you have a problem. p values exist for the purpose of doing that. The problem in in research isn't the use of p values per se, but the misuse and misunderstanding of them.
An argument to do away with p-values demonstrates a basic misunderstanding of them.
If you want to prove that the new process is better on average ("Xnew > Xold") , you would have to use "Xnew <= Xold" as the null. When you determine Xold to be 1C using a sample size of 100, you also get a variance that you have to take into account instead of naively substituting it into the hypothesis.
If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
If you get wrong conclusions using correct math, your assumptions are wrong. In the case of your example, the incorrect assumption is treating an uncertain measurement as exact when comparing it to another uncertain measurement.
I think people are misreading my purpose. Your response comes closest:
>If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
That's for a single population. If you are trying to compare two populations, you have two empirical means and two variances. When you do a t-test [1] to compare your measurements, the test statistic has a variance that is approximately computed as
Adding more samples for only one of the populations means that this half of the sum vanishes, but the other half remains unchanged, thus guaranteeing that the variance never drops below a certain threshold.
> In many real world effects, we're not 100% accurate on what the null hypothesis is.
The null hypothesis is the simple negation of the affirmative hypothesis; anything else and you can't use p-values at all.
If you know what the hypothesis you are testing is (and if you don't, you have a fundamental problem), you also know what the null hypothesis is.
The proper affirmative and null hypotheses in the test you propose are “process 1 has a greater measure on the dimension of interest than process 0” and “process 1 has no better measure on the dimension of interest than process 0”. Normal statistics on parallel measures of both processes will give you results which may or may not allow you to reject the null hypothesis at a given p-values, but (except that the smallest p-values you can get goes up with sample size, so that you can't, rightly or wrongly, reject the null hypothesis to certain confidence with too small a sample size) the probability of rejecting the null hypothesis incorrectly does not go up with sample size.
About [2], I couldn't really tell the color of the graph and was confused about its story. But if you scroll down, it is revealed that there is no operator available for the data on the field of economics, because it hasn't been collected, and that is why there is no bimodal for economics. The quote is:
"It's confounding with the meta-analysis the data comes from: Brodeur 2016 is the only one that did econ; didn't collect <= or >= from papers"
I just want to buffer a little bit what you said about "the p-value is just checking a box more than being used as a tool for discovery.", but we are likely on the same page.
Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be "checking a box".
Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI's lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery.
At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.
The term "significant" is itself part of the problem. In statistics it is often used to describe an arbitrary threshold where a difference between datasets is assumed to be real. In common parlance this term implies "important" or "meaningful", but the statistical tests are not designed to draw such forward-looking conclusions.
It would be far easier, and more honest, to simply report the P-value obtained by performing a statistical test without adding the term "significant".
I think that a very stringent threshold for p values (it's going to be a lot harder to get to p < 0.005 than p < 0.05 and it's going to encourage a lot more gaming of the system) is not the solution. At best, it will lead to smaller numbers of much bigger studies with more authors and fewer interesting results.
I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website.
a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it.
b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc.
c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses.
All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result.
This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)
If we were designing things from scratch, I'd even want t odisaggregate two career tracks for scientists:
The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results.
The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything.
A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.
One of my favourite comments from a lecturer in my applied science degree: "In theory there's no difference between theory and practice. In practice, there is". We encountered so many times when theory said one thing but it didn't pan out when tried.
It's another way of saying "First, take a spherical cow..."
Yes, and indeed with this system a theorist can look for interesting data out there and publish with attribution, while experimenters can look for interesting failures and home methodology.
assume that, to simplify, that every research paper out of there has findings with p-value < 0.05 - that means one in 20 papers has wrong conclusions in it.
or that one in 20 teams will not be able to reproduce findings.
0.05 is just removed enough from coincidence that, given a solid theory, corroborates the findings. but running across data to find correlation and using 0.05 as threshold is a surefire way to find something that has no actual significance beyond mathematics and zero ground in reality.
Because of conditional probabilities, your analysis is wrong. This xkcd explains the issue. Someone else linked to it already, but it exactly describes the problem -- with p=0.05 >> 5% of published results are incorrect:
You're making the same mistake in reverse. The p-value is not a marker of correctness, but of confidence. If something doesn't reach the p-value cutoff, that doesn't mean that the effect doesn't exist, just that it didn't meet the confidence level. A paper could be entirely correct in its hypothesis, yet not meet the confidence level in its experimentation.
I'm thinking back to my graduate thesis project, in physics. I started with a fairly nebulous problem, and went down a lot of blind alleys before I finally got something working. And there were some unexpected results that we didn't even have a good way of describing until we saw them happen and puzzled over them. I published a few weeks later, and my advisor moved to another university a few weeks after that.
A rigid rule of pre-registering "study, methodology, and hypothesis" would have prevented me from even launching the project. At the very least, I would have had to register a formal study after my results were already established to a decent degree of confidence. It would have been wasteful and redundant, not to mention risky, since anything can happen to a grad student to prevent them from finishing a degree (illness, loss of faculty advisor, etc). At best, time is money for a grad student who wants to get out of school ASAP and start earning a salary.
My project produced a humble but solid result, and launched a new research program for my advisor, that bore fruit for another 25 years.
Now, I wasn't doing a "study," as in a life science study looking for a statistical correlation. I reported no p-value. So maybe it doesn't apply to all kinds of scientific work. And I appreciate the need to restore come kind of integrity to the medical and behavioral sciences in particular. I don't know how to help those fields, and maybe pre-registration only applies to them. But I'm concerned that declaring rigid rules for science across the board will have unintended consequences.
Edit: Another issue is a student getting "scooped," where some other lab reads the plans and rushes the work to completion. There are labs in chemistry and biology that are notorious for doing this. If a student doesn't at least have an informal guarantee of priority over their own work, they won't take the risk of launching a project in the first place.
if i were the King of All Science and were feeling despotic, I would do the opposite, I think -- put a moratorium for a few years on any publication of a p-value.
Scientists would still be free to make use of hypothesis testing to guide their research. In publications, one could use Bayesian methods and report a full posterior. One could also just give enough information about the null hypothesis and whatever distributions are involved to allow the computation of a p-value.
However, any attempt to explicitly publish the p-value itself would result in creative and embarrassing punishments.
I mean, it was a "modest proposal" type of suggestion.
There are a lot of researchers who have a limited knowledge of statistics and who basically treat the p-value as a magic number.
I don't have any sort of philosophical problem with p-values or the tools of hypothesis testing, but I think it would be good for science if the magic number were less salient, if that makes sense.
Any golden value for p is bad. If I go back to my statistics text book (and I assume all of them say something to this effect): The appropriate value of p depends on the particular topic of study.
Not on the whole discipline.
And definitely not across all studies and across all disciplines.
You cannot pick a value of p and say "this is good". Pick too low and you'll likely not be able to reproduce many legitimate results.
Fishing for findings is a problem, but also overly stringent multiple comparison correction requirements. Those who preach Bonferroni should consider a life-long proposal.
Let us propose that correction for multiple comparisons be a life long endeavor. At the start of a scientist's career as principle investigator, p<.10 is fine. But by their career's 10th comparison, they have to meet at Bonferroni p<.01, and at their career's 100th comparison, now they have to meet at Bonferroni p<.001. When they are old and gray, or doing genomics, at their 1000th comparison they have to meet at p<0.0001.
Genomics generally uses 5e-8 (p<0.00000005) as its significance threshold. That aside, your point is clever. How to handle coauthors of different ages is left as an exercise to the reader.
Young investigators tend to get smaller grants and sample size can be an issue. Relaxed threshold for the beginning of a career helps them, career wise. Older investigators frequently start heading really large, ambitious projects with large sample sizes, but they would have to face more stringent p-values.
What is funny asbout p<.05 is that while we frequently have a specific hypothesis in mind (e.g. Mean1<Mean2), convention still dictates that we should use a two-sided t-test instead of a 1-sided t-test. If I am asked to go to .001, then I will use 1-sided t-tests when I have apriori one-sided hypotheses.
66 comments
[ 8.4 ms ] story [ 112 ms ] thread1) Not really addressing the issues behind the replication crisis
2) Doing the exact opposite of what should be done (raise the significance level)
Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis.
I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
Undue emphasis on p-values is bad, but what you're suggesting is bonkers.
Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good.
The only issue is that it is a much weaker constraint on publishing than it was previously thought.
Many more results should be documented than currently are, though, including negative results.
But I agree that changing p-value thresholds is unrelated to that.
Eg. The Journal of Wacky Speculation publishes "The Moon is Made of Green Cheese". Using a highly sophisticated cheese detector, the authors measure results consistent with this hypothesis. They calculate it would happen only 4% of the time if the moon were not in fact made of green cheese, and therefore the result is significant. The cheese detector works perfectly, the calculations were accurate, but this does not mean the moon is made of green cheese! We've been there, and we're very certain that it's made of rock. Because the prior probability of the hypothesis was so low, the "significant" result was most likely a fluke. The Journal of Wacky Speculation specializes in this type of hypothesis, so they should use a much lower threshold for p.
Meanwhile, the Sensible and Boring Society publishes a study showing that round wheels roll better on ordinary roads than square wheels. We already had good reason to believe this, so p<0.05 shows it's most likely true. The Sensible and Boring Society publishes tests of hypotheses with higher prior probably, so they can use a higher p threshold.
> [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses.
Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1]
This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05.
[1]http://www.stat.columbia.edu/~gelman/research/published/asa_...
I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.
> a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses
If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish.
Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
I agree that doing something is better than doing nothing, but I disagree that this will reduce bad results in a meaningful way. Setting a more stringent threshold for significance will elimnate studies that reach the wrong conclusion due to a statistical fluke, i.e. there's no effect but due to random chance, the noise in the data makes it look like there is. But IMO these are relatively rare. It seems to me that the vast majority of bad studies reach significance by virtue of methodological issues that enable one to reach any arbitrary level of significance just by trying hard enough. If Daryl Bem continued doing what he was doing, he would have eventually reached p<0.005.
edit: I just glanced over Bem's paper and he reports several p-values under 0.005 and several other very close to it.
I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery.
Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2].
[0] http://rpsychologist.com/d3/NHST/
[1] https://github.com/jtleek/tidypvals
[2] https://twitter.com/drob/status/890260541876338690
And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg
These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows.
Edit:
Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.
This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large
Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
The overall function is a p-curve, here's a paper on it: http://journals.sagepub.com/doi/pdf/10.1177/1745691614553988
https://twitter.com/drob/status/890260541876338690
I can't even begin to imagine how one can collect that number of papers in a machine-readable format without ending up like Aaron Swartz.
Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn't exist.
Say my null hypothesis is that X=100. The alternative hypothesis is X>100.
What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result.
There are ways to fix this, but one should just be aware, though.
> What if in reality X is really 100.5?
Then the alternative hypothesis is true and the null is false. (Though normally if the tested hypothesis was X > 100, the null would be X ≤ 100.)
> Depending on the problem domain, this may well be the same as the null hypothesis.
No, X = 100.5 is not the same as X = 100, irrespective of problem domain, so long as the mathematical symbols have their usual definitions.
> But a larger sample size is much more likely to give a significant result.
Yes, if something is only just barely true, it is more likely to take a large sample size to distinguish it from the case where it is false. But since in this case, the alternative hypothesis actually is true, it's not a problem that a bigger sample is more likely to reject the null hypothesis.
Mathematically, yes. Practically, not always.
In many real world effects, we're not 100% accurate on what the null hypothesis is. We may think the existing process suggests a the temperature increase of 1C, and so we set that as the null hypothesis.
But in that problem domain, we may have difficulty being more precise. Suppose that 1C in the existing process came from the mean of measuring a sample of 100, so the current belief is that the expected value of the increase should be 1C. Suppose in reality it is 1.02C. Now we're proposing a new process and claim it is better than the existing one. This time we take a much larger sample, and see the number hovering at 1.02C - we can then claim the new process is better than the existing process and have a p value to demonstrate it.
The math will be correct, and the conclusion will be wrong.
P values measure statistic significance, not practical significance. If you are extrapolating practical significance from statistical significance, then you have a problem that is unrelated to sample size.
If you never extrapolate practical significance from statistical significance, then you have a problem. p values exist for the purpose of doing that. The problem in in research isn't the use of p values per se, but the misuse and misunderstanding of them.
An argument to do away with p-values demonstrates a basic misunderstanding of them.
If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
If you get wrong conclusions using correct math, your assumptions are wrong. In the case of your example, the incorrect assumption is treating an uncertain measurement as exact when comparing it to another uncertain measurement.
>If you do the statistical significance test correctly, you won't be able to show the new process better by taking lots of samples for it, since the variance of the old measurements dominates the uncertainty. (I think, haven't done the math to show it.)
Higher sample size means lower variance:
https://onlinecourses.science.psu.edu/stat414/node/167
That's for a single population. If you are trying to compare two populations, you have two empirical means and two variances. When you do a t-test [1] to compare your measurements, the test statistic has a variance that is approximately computed as
variance = variance_1/sample_size_1 + variance_2/sample_size_2
Adding more samples for only one of the populations means that this half of the sum vanishes, but the other half remains unchanged, thus guaranteeing that the variance never drops below a certain threshold.
[1] https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_un...
The null hypothesis is the simple negation of the affirmative hypothesis; anything else and you can't use p-values at all.
If you know what the hypothesis you are testing is (and if you don't, you have a fundamental problem), you also know what the null hypothesis is.
The proper affirmative and null hypotheses in the test you propose are “process 1 has a greater measure on the dimension of interest than process 0” and “process 1 has no better measure on the dimension of interest than process 0”. Normal statistics on parallel measures of both processes will give you results which may or may not allow you to reject the null hypothesis at a given p-values, but (except that the smallest p-values you can get goes up with sample size, so that you can't, rightly or wrongly, reject the null hypothesis to certain confidence with too small a sample size) the probability of rejecting the null hypothesis incorrectly does not go up with sample size.
Your statement didn't address anything in my comment.
"It's confounding with the meta-analysis the data comes from: Brodeur 2016 is the only one that did econ; didn't collect <= or >= from papers"
Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be "checking a box".
Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI's lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery.
At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.
It would be far easier, and more honest, to simply report the P-value obtained by performing a statistical test without adding the term "significant".
Essentially the more likely your hypothesis, the higher your P value can be, and conversely unlikely hypotheses require stronger evidence.
I don't think this solves the problem but maybe it makes that assumption about the hypothesis likelihood more specific.
I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website.
a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it.
b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc.
c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses.
All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result.
This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)
If you get null results, you have to report that.
The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results.
The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything.
A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.
It's another way of saying "First, take a spherical cow..."
assume that, to simplify, that every research paper out of there has findings with p-value < 0.05 - that means one in 20 papers has wrong conclusions in it.
or that one in 20 teams will not be able to reproduce findings.
0.05 is just removed enough from coincidence that, given a solid theory, corroborates the findings. but running across data to find correlation and using 0.05 as threshold is a surefire way to find something that has no actual significance beyond mathematics and zero ground in reality.
https://xkcd.com/882/
A rigid rule of pre-registering "study, methodology, and hypothesis" would have prevented me from even launching the project. At the very least, I would have had to register a formal study after my results were already established to a decent degree of confidence. It would have been wasteful and redundant, not to mention risky, since anything can happen to a grad student to prevent them from finishing a degree (illness, loss of faculty advisor, etc). At best, time is money for a grad student who wants to get out of school ASAP and start earning a salary.
My project produced a humble but solid result, and launched a new research program for my advisor, that bore fruit for another 25 years.
Now, I wasn't doing a "study," as in a life science study looking for a statistical correlation. I reported no p-value. So maybe it doesn't apply to all kinds of scientific work. And I appreciate the need to restore come kind of integrity to the medical and behavioral sciences in particular. I don't know how to help those fields, and maybe pre-registration only applies to them. But I'm concerned that declaring rigid rules for science across the board will have unintended consequences.
Edit: Another issue is a student getting "scooped," where some other lab reads the plans and rushes the work to completion. There are labs in chemistry and biology that are notorious for doing this. If a student doesn't at least have an informal guarantee of priority over their own work, they won't take the risk of launching a project in the first place.
>10 significant things that WILL shock you, #7 is this team
>THIS crazy trick that will make significant much harder [teams hate him!!]
>This finding prank gone significant [gone sexual!?]
Sorry, it was such a click-baity title for no reason.
Scientists would still be free to make use of hypothesis testing to guide their research. In publications, one could use Bayesian methods and report a full posterior. One could also just give enough information about the null hypothesis and whatever distributions are involved to allow the computation of a p-value.
However, any attempt to explicitly publish the p-value itself would result in creative and embarrassing punishments.
I fail to see how this is superior.
I'd never advocate banning a legitimate tool just because of abuse.
There are a lot of researchers who have a limited knowledge of statistics and who basically treat the p-value as a magic number.
I don't have any sort of philosophical problem with p-values or the tools of hypothesis testing, but I think it would be good for science if the magic number were less salient, if that makes sense.
Any golden value for p is bad. If I go back to my statistics text book (and I assume all of them say something to this effect): The appropriate value of p depends on the particular topic of study.
Not on the whole discipline.
And definitely not across all studies and across all disciplines.
You cannot pick a value of p and say "this is good". Pick too low and you'll likely not be able to reproduce many legitimate results.
Let us propose that correction for multiple comparisons be a life long endeavor. At the start of a scientist's career as principle investigator, p<.10 is fine. But by their career's 10th comparison, they have to meet at Bonferroni p<.01, and at their career's 100th comparison, now they have to meet at Bonferroni p<.001. When they are old and gray, or doing genomics, at their 1000th comparison they have to meet at p<0.0001.
Interesting dynamic.