At the very least, Romain Brette suggests to change the wording to "statistically detectable".
I 'm not sure if abandoning tests altogether is good though. What does it mean "it's detectable but not clear" for communication? How do you e.g. communicate global warming like that?
I usually don't see climate science being communicated with p-values and poor significance cutoffs. The place where you usually see this used is for communicating single study results (which often enough are things that you probably shouldn't communicate at all).
scientific papers communicate with numbers, errors, p-values next to numebrs, name of the statistical test used, and figures are usually annotated to indicate the level of significance. That's pretty careful.
Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Responsibility Fairy doesn't visit you that night and make you suddenly able to handle it when you weren't the day before, it's just that at scale, you don't have much choice but to operate that way because everything else is just too expensive.
The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words.
There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
> "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?
The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
You've mangled my sentence in your quote. The rest matters. It may not be a masterpiece of the English language, but I already said the first is not directly possible and I wrote the second as a description of p-values quite on purpose.
"Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used."
That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.
ah yes i agree with those. As you said, the selection of hypothesis is the actual problem, and statistical power doesnt solve that one either.
There are some scientists thinking over this though. This is an idea from a neuroscience lab: https://www.researchmaps.org/
The idea was to create causal directed graphs for biology from the literature , which would be used identify what experiments are missing and thus inform future science.
That may not be a bad thing. Having different solutions for different categories of experiments, or even just alternatives to select from could be useful.
In a way yes, because replication lowers the risk that the findings in a single experiment are due to random luck. Anyway there's no way around avoiding replication when you want to prove something.
Ehe. Yeah if one insists enough this can happen. It's not a panacea, and really switching to any one measure will cause it to be hacked in the end. The solution is probably multiple objectives, eventually including the reputation of the researchers as well
How about understanding and valuing a publication as a letter to the research community about something the authors think is interesting and needs further consideration. The start of a scientific debate. A "hey what do you other research groups around the world think about this?" If nobody thinks anything and it doesn't gain traction or followup debate/research/consideration, then it failed its mission.
Not as "This research has now found X and is in a reputable journal, so this is the new state of science from now on".
The problem is, academics are pushed to writing more and more papers. There's no time to discuss back and forth too much about an already published paper, there are new papers to write! The goal of the project was to publish a paper, published=won, end of story, next project on.
Major problem with "significance" is that in some areas of research (say psychology), it's possible to gather lots of data(let 1000 people fill complex questionnaire) and from that data to fish for theory that's significant (in your sample Republicans might have been dumber than Democrats). But given the size of the sample and number of theories you test, you are bound to find something significant even if that isn't true.
Tightening significance threshold just makes this fishing more difficult, Bayesian reasoning doesn't help much either, because you have to guestimate reasonable priors. What does really help against fishing like this is requirement to preregister your studies.
Many people talk about preregistration, but I'm not sure it would result in the hoped benefits.
What I predict would happen:
- either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for universities and research centers.
- or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done. This hinders research and hinders the dissemination of truly unexpected discoveries.
The human incentives are way deeper than any one solution could touch on. The whole science funding structure requires flashy and sexy results that are just not possible to produce on that scale.
> or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done
I see two things going on here.
Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing.
I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.
Finally!
If only we could really get rid of this largely meaningless 0.05 parameter.
Though I would have personally used statistically instead of strictly in the sentence:
>Strictly speaking, he says, “there’s no difference between a P value of 0.049 and a P value of 0.051.”
The problem isn't p-values, the problem is a binary distinction between p=0.049 and p=0.051. The problem would go away if everyone understood p-values, or we replaced use of the term "statistically significant" with "3% probability we're just seeing a pattern by accident". Renaming the term to something that sounds just as binary isn't any different.
Scientists have a duty to communicate what is true and what is not though. Passing that judgement duty to the general public is irresponsible and unwise (it's easy to claim anything with flawed / misleading statistics). Ideally someone would have come up with a framework that is better than statistic tests to justify levels of truth, but so far we dont have one
When you've got empirically gathered data, pretty much every type of analysis you can do with it comes under the umbrella of "statistics". I think any better truth measuring framework can only be statistical.
There are very few principles in science. Statistics tests seem to be something where scientists unanimouslu agree. Another is models, and applying occam's razor to choose the one with less parameters. Perhaps one could also justify choosing a hypothesis on the basis that it is simpler.
They certainly don’t agree. There has been 30+ years of objection to p < .05. The fact that many scientists are statistically illiterate doesn’t constitute consensus.
Scientists (should) never really "agree", they just concede temporarily preparing their next attack. statistic tests are the de facto standard that everyone concedes to, and has not been replaced.
Richard Feynman said anything you can't prove yourself must be taken on pure faith and most of the public must take everything on pure faith as somehow they've left school without the faintest idea how basic stats work.
Feynman was talking to scientists. I do think that science has to provide answers, not just data. The public interprets every datum in a myriad of ways, and does not have a mechanism to establish consensus, hence you would end up with constant crisis.
This is a a really common misconception about p values (that they can be interpreted as p(H0|x), or "probability of the null hypothesis given the data") when a p-value is in fact p(x|H0), or "probability of observing data at least this extreme given that the null hypothesis is true
Do you have a better of wording for inclusion in paper abstracts / article summaries than what I said? I'd love to hear one, but p(x|H0) is just as bad as p = 0.05.
The problem isn't that the wording is confusing, it's that p(x|H0) isn't a very useful thing to compute.
Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way.
You can actually compute p(H|x) if you use Bayesian stats.
With some caveats: the value you get for P(H|x) might be very sensitive to the priors you choose, and most people are not thoughtful enough about this.
That's definitely the hard part of switching to Bayesian methods. But the problem doesn't go away if you use p values, it just gets ignored.
Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.
You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.
You just make priors up, just like the 0.05 limit is made up. Making up a prior is basically asking the question: How likely does it seem that X has Y effect? Without having any data specific to that question.
Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.
> You can actually compute p(H|x) if you use Bayesian stats.
You can't "compute" it for any useful meaning of the word "compute". You can estimate it intuitively, or you can try to look at how many pre-registered unpublished studies or null results have been published. Otherwise there's no way to even consider getting a grasp of what P(H) would be.
> Otherwise there's no way to even consider getting a grasp of what P(H) would be.
That's called picking a prior. Usually, pick the prior that maximizes the entropy on the given interval (zero prior knowledge).
For those reading and confused. Computing:
p(H|x) = p(x|H)p(H)/p(x)
requires p(H) which the parent is suggesting is impossible to grasp.
If we want to know the probability that a coin is biased, we can assign probabilities to each hypothesis. For some people, p(bias=1/2)=1 (Dirac distribution), others might argue that p(bias=x)=1 for x in [0,1] (uniform distribution). Others might argue its some Beta function centered around 1/2. I believe what the parent is suggesting is that choosing which original belief we have in the system is a matter of philosophy, not computation.
"Our research creates a lot of confidence that the hypothesis is true. If our hypothesis happens to be false, then only 1 in 20 independent repetitions of our research (P = 0.05) will obtain the good results that we present here."
This is a more mathematically precise restatement of kerkeslager's explanation below and the obligatory xkcd [1]. The problem boils down to the fact that null results are not usually published, and that intuitive skepticism is not quantifiable.
Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied.
The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing.
The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility.
I'd argue that p(H0|x) is also (in most cases) pretty uninteresting from a scientific perspective, and so the whole "publish all the p-values" solution I think is only fighting half the battle. Gelman does a much better job of arguing this than I ever could [1], but this idea that rejecting the null is the "desired" outcome is the real problem. To put it another way, a low p-value is another way of saying "my model of the data generating process is bad", which is in most cases a pretty unsatisfactory outcome. As to the idea that priors are "impossible" to get, to quote Gelman [2] again, why "strain at the gnat that is the prior and swallow the ungainly camel that is the iid likelihood?
It's worth to point that "3% probability we're just seeing a pattern by accident" is only right when you understand it as "in the world here our hypothesis is wrong the same experiment would give such pattern in 3% cases", not as "given such result probability that we are wrong is 3%".
It seems like an education fail to me. Most people don't know the very basics of stats and we live in a world that's highly probabilistic.
It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
I never learned statistics until my second year in university, it was really mind opening, especially with showing me just how much poorly done statistics with tiny sample sizes or misrepresented data are quoted and used as facts.
I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.
They're nuanced concepts that are easily misunderstood, but in a typical US public school, statistical significance (or anything beyond basic descriptive statistics) is not typically part of the curriculum[1]. It's a hard problem to effectively educate out of existence but right now we're not even really trying.
[1] source: I never saw it as a student, and I've consulted with math teachers (well, administrators, about math) in multiple states, and I've never had it come up there either.
"3% probability we're just seeing a pattern by accident" in fact still sounds a lot better than it is if there is a positive result bias, as you'll miss the "but 99 percent of scientists who attempted this did not see a pattern"
We use hard cutoffs for a bunch of things, they aren't perfect but they are fine.
The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English.
It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter.
This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean.
Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance.
For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.
In fact, failures to reproduce "landmark" experiments have low significance (statistics word) because they find a result that is highly probable if the null hypothesis is true. But failures to reproduce have high significance (colloquial English word) because they indicate that widely-held beliefs are wrong. So in some of the most important cases, the statistics and colloquial English meanings of the words are exactly the opposite.
I don't know if there's a good solution to this, though--it's hard to change organically-emergent trends in language even if we did come up with a better term.
This is an excellent point. In my work, the interpretation error I see the most among people around be is to confuse significance with the magnitude of the predicted effect. I.e. if you use OLS to create a model, it is possible to have a statistically significant beta, with essentially no magnitude.
If you look at research on carcinogens, it is frequent to see something like "it is statistically significant that eating chemical X increases your risk of cancer" but when you read on, the magnitude of the effect is "increases risk by .0023%, for a cancer that already has a .6% incidence in the population".
My point is that people who are not statisticians (or people who use statistics), tend to be concerned with the magnitude of an effect, and frequently misconstrue the statistical significance for that magnitude.
I've been trying to not use the words "statistical significance" in my reports. It's hard to find alternatives, but "confident" usually fits the bill. Sadly, the reports need reviewed by other statisticians, and they much prefer I use the normal jargon because it's clearer to them.
But these concepts aren't clear to the people using the data. I've seen people use statistical tests done in batch without correcting for multiple testing (one of my regrettable tasks), and flat out ignore any differences that weren't statistically significant. They thought it meant "no difference."
With population-level data (e.g., Census counts or hospital records), I sometimes wonder if statistical significance has any value. It's a tool to make sure we don't accidentally claim a difference where there isn't one. But I can guarantee you that nothing worth measuring is exactly the same for men, women, whites, blacks, young, old, whatever. It'd be amazing if the difference was 0. So statistical significance ends up being an obfuscated way to state the sample size.
At the end of the day, people need to make a decision about something, and I suspect this is what is forcing the importance of this binary distinction.
My decision to select, implement, go, or no go cannot itself be a probability distribution.
Decision to select shouldn't be the subject of academic or scientific papers in the first place. There's far more context required in decision making than just how strong a correlation is.
The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs.
Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly because it is so easily misunderstood. You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood.
Worse yet, p-values are sensitive to particulars of experimental design that logically should never matter to inferences. My aunt and uncle provide a classic example. They wanted a son and a daughter. They had 6 sons then a daughter. Are they biased towards one gender? The p-value for this question is .03125 (we'd see this much evidence against with all boys, all girls, last girl, last boy, so 4/2^7). But if I instead said that they planned to have 7 children, and had 6 sons then a daughter, the p-value changes to 0.125 because the one child out could have been anywhere in succession. But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences.
(This example is not made up. When Lorna found that they had a daughter, she told Bill to get a vasectomy from the delivery table. By another crazy coincidence, their 7 children were also born on all the days of the week.)
Its not just about p-values or the likelyhood of making a particular type of error by agreeing with a thesis.
There is a fundamental misunderstanding that most people, and even many scientists make with regards to the philosophical interpretation of 'truth' using the scientific method.
By definition, scientific 'truth' is always fallible, and the vast majority of people have a very hard time dealing with this concept. A better explanation or additional data, or better data is always possible and needs to be possible for science to not be dogma. This is fundamentally different than how people interpret truth and understand what the word means. All scientific truths are replaceable.
> But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences
That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments:
1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)?
2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?
The probability of event A given that we observed B is the probability of A and B happening divided by the probability that B happened. Mighta, coulda, shoulda but didn't doesn't enter into it and can't affect the result. The fact that frequentist statistics does care and shouldn't is one of the major criticisms that Bayesians offer.
If you think you understand statistics and don't understand this fact, then you do not understand statistics as well as you think you do. But you can be pardoned. Most statistics classes are too busy cramming statistical tests into student's heads to bother them with bothersome facts about where the cracks in the foundations are.
P-values are much more problematic at the alpha=.05 threshold than the alpha=.005 or alpha=.01 thresholds.
In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are <.0001, meaning there is SO much data that even a very very small effect size can be found "significant."
The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?
Yes, thank you. Practical significance. You cannot get away with just running statistical test or deep learning models. You often need to ask whether a change in product, service or process to take advantage of some trend or difference between customers segment will actually be worth doing. That involves getting your hands dirty in the business, trying to identify all the costs and side-effects of a change, and then how much of a size effect you will actually capture once the changes are made. My views in more detail [1]
I agree with you, but I think there is an even more fundamental issue. P-values really don't tell you anything. They can refute a bad hypothesis but they cannot validate a good one. This [1] is such an awesome site. It lists a large number of spurious correlations. It's quite remarkable how strong many of them are. For instance there is a 99.79% correlation between suicides by hanging and US spending on science/space/technology.
The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship.
538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal.
Or we'd have just as many, but better at statistics.
P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful.
Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.
I have a rule of thumb: Multiply p by 10. In other words, p = 0.05 is as good as a coin toss. This is consistent with the so called replication crisis.
In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects.
I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.
I think this is part of the problem, but the more fundamental issue is that people are extremely bad about probabilistic reasoning. Even if we said "3% probability we're just seeing a pattern by accident" most people unconsciously round this down to zero.
People that work with computers often are slightly better about reasoning about probabilities because they often deal with high frequency events. A 1% error rate across 10 million events a day leads to 100,000 errors. If those events are transactions, payments, file uploads etc, a 1% chance starts to look very common. But even computer people are not immune to incorrectly reasoning about probabilities. I suspect there is some fundamental limitation at the cognitive level.
While I agree with that completely, the coinciding problem that I've run into time and time again is the temptation to accept statistical significance but ignore the size of the effect.
You might see p=0.00001 but if the size of the effect irrelevant, then the statistical significance of the relationship is still not something I care about. Great, orange juice shrinks tumors by 0.000001%. I'm still going with chemo, thanks.
If you're looking for a replacement you don't understand the problem.
The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis.
Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up heads with P=.5. If you flip the coin twice, the null hypothesis is that both flips come up heads with P=.25. The probability of all coins coming up heads is P=0.125 for 3 flips, P=0.0625 for 4 flips, and P=.03125 for 5 flips. So if we flip a coin 5 times, and get heads all 5 times, we can conclude that the coin is weighted in some way with P=0.03125.
Let's say all the major journals of coin flipping only publish results with the high significance of P<0.05. Alice flips a quarter 5 times and gets 3 heads and 2 tails, and nobody will publish her study because it has P=.3125 (see [1] for an intuitive explanation of how this was calculated). Bob flips a quarter 5 times and gets 2 heads and 3 tails--again, no journal will publish him. Catherine, David, Ellen, Frank, and Geri all perform the same experiment, most of them getting groupings 3:2 result ratios, some getting 4:1 result ratios, but nobody getting all heads or all tails, just as one would expect given the null hypothesis. And the journal editors tirelessly send out rejection letters to all their studies.
Now somewhere down the line, Robert flips a coin 5 times and gets 5 heads. This result has P=.03125, which meets the requirement of P<.05! He sends it to the American Journal of Coin Flipping Studies (AJCFS), and they are very excited to publish his results! Nature and Science magazines do front page pieces with headlines "Quarters Found Heavy-headed" and "Washington Shows His Face" respectively. A casino hires Robert as a consultant for the design of their coin-flipping games. His quarter-flipping study is cited in the abstracts of two dime-flipping studies and a half-dollar flipping study. During the trial of a murderer who placed quarters tails-side up on his victims, Robert is called as an expert witness to say that the coins were placed there, not flipped there.
A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.
Now, if you survey the AJCFS and all the other academic journals on coin flipping, you'd conclude that quarters are significantly weighted towards heads. But in fact, the null hypothesis is true: quarters are pretty evenly weighted. Robert's low-P result is exactly what you'd expect to happen eventually if you have enough people perform the 5-quarter-flip experiment--in fact, if a lot of people are studying coin flips, the P of getting a low-P result approaches P=1. But because the AJCFS has a P=.05 requirement, they've created a bias against the null hypothesis, which deceives the public into thinking that flipped quarters are more likely to come up heads than tails.
This is likely the reason why so many fields, most notably psychology[2], are having a replication crisis[3] and a similar effect can be used in P-hacking[4] to bolster results that are essentially fake.
Unlike coin-flipping, fields with replication crises like psychology and medicine have real affects on real people's lives. It's irresponsible and unethical for journals to publish with a bias against the null hypothesis, and adjusting the P-value requirements to another significance requirement, even a less arbitrary one, doesn't fix the issue.
The solution, I think, is for journals to commit to publish studies before the study has been performed, based on the methodology, previous studies on the subject, and qualifications of the researcher. This would mean that man...
This is interesting because when I learned all this stuff the P value wasn't the only measure of study design quality. There was also statistical power which here would related to number of coin flips performed. You simply can't flip a coin 5 times and say you have statistically valid results you would be laughed out of the auditorium. Maybe it requires 1000 independent trials, or 10,000, but certainly not 5. The P value was just about confidence level of rejecting the null hypothesis, but much more has to go into study design in order to make it valid.
I have not done this sort of work since the early 2000s, I am relying on memory and a couple of quick web searches.
This is actually a much better explanation of the problem than the main article.
The main article kept saying that p values were not meant to be definitive, but without explaining what was wrong with them. At least, not very clearly.
This comment is a much clearer explanation -- imho -- as to what goes wrong in the industry. I also think it is right to focus the attention away from a particular statistical measure.
It seems clear to me that the author just didn't understand the problem with P values. It's part of a larger problem of science journalism being done by journalists without scientific backgrounds. It's not their fault--even if you have natural ability and interest in both communication and discovery, it's hard to get an education in both.
I have the opposite problem: my abilities lie more in the statistics/science than the communication--I suspect the only reason that this explanation is being received so well is that I happened across an effective example by pure luck. The probability of me communicating effectively is probably P < .5. ;)
Let me check if I get it right: can this be said to be a case of incorrectly aggregating experiments? We're in a sense taking min(p) over all p-values, or any(significant) over all results, when we should use an aggregation method that takes into account the total number of studies aggregated?
For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups.
Let's calculate how unlikely a P=.006 event is if you try 47 times to find it:
A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2]
B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1).
C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits).
D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246.
So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length".
The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a P<.05 result. Doing the same math again for the probability of a P=.05 result occurring if you try 47 times, P'_47' = 1 - (1 - .05)^47 = 0.91025516807. That's over a 90% chance! In other words, by doing the test on 47 food groups, they nearly guaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place.
This study has not been replicated as far as I know, but is often cited in pop science[3][4][5].
[2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51).
This is correct, and also illustrates why merely replacing frequentist methods with Bayesian methods is not sufficient to fix the scientific reproducibility crisis.
Is there a compendium somewhere of null hypotheses as well as observed p values (or whatever statistic) for experiments with both "significant" and "insignificant" results? Depending on the phenomenon and the hypothesis, an event could have a different probability of occurring and require a different threshold.
It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.
This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
The current approach also ties the experimental method to the result, requiring potentially fallible oversight of the experimental design. The hypothesis-testing aggregator reduces this reliance, which is perhaps more honest about our ability to design good experiments, would allow for a greater diversity of approaches, and could also be more efficient when evaluating results.
The most interesting aspect of this is if you could somehow create relationships between standardized hypotheses found on the hypothesis menu such that you could infer other hypotheses in a more systematic way.
> It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.
You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science.
> This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
"Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue.
A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).
> we can conclude that the coin is weighted in some way with P=0.03125
No we can't. There are 2 (related) reasons for this. The first is that to say "after experiment X, we can conclude thing Y is happening with probability ..." you need to know something about how often thing Y happens on its own. This is also known as a prior. The other reason is that in this specific case, the prior for Y (that the coin is weighted) can be concisely summarized as P(Y)=0, because it's physically impossible to construct a coin that's biased to one side or the other (independent of what side it starts on). Some people think the coin flipping examples are bad because biased coins are impossible, but they're actually good for exactly this reason. We have a really good prior (from math/physics/geometry) that biased coins can't exist, so you're going to have to not only flip a lot of them to convince me otherwise, but also give me an explanation of how that's possible (i.e justify your prior)
This is an excellent write-up! Some friends of mine worked at one of the psychology departments as programmers/IT-guys while studying CS, lesson learned: Real knowledge of statistical methods is rare among psychology researchers; one (of my friends) even got fired because he pointed out that they asked him to fake results (the experiment basically assumed the Halting Problem to be solvable).
> A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.
Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of tails is at most 15%. Her p-value would be 2.66% (prob of seeing <= 2 heads) and if the world was fair, AJCFS would publish her study. But the world isn't and if Robert is a renowned scientist, significant+ is enough for him but failed replications of his work requires significant+++.
I had to think about this for a while to come up with a reasonable response. What you're saying seems a bit problematic to me.
The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation.
The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies.
Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.
The problem is not with using statistical significance as a mean to accept or reject a hypothesis (ie. the scientific method). The problem is P-hacking (effectively, choosing your results then customizing analysis to obtain them).
It's not the statistical analysis that's the problem, it's the bad "science" and irresponsible journalism.
Most comments here point to cherry picking and "p hacking" as being the primary problems with p values. Certainly those are major issues, but I think they miss the real point of the article, which is that null hypothesis testing is fundamentally broken, or at the very least doesn't do what most people think.
A simple example of this can be shown with the following pair of tests:
Testing for a fair coin:
- Null hypothesis is you have a fair coin
- You observe 100 heads in a row
- Given a fair coin, it's extremely unlikely to observe 100 heads
- Therefore it's not a fair coin
Okay, that makes sense, but this is logically the same as:
Testing whether a person "Bill" is an American:
- Null hypothesis is Bill is an American
- You observe *Bill is a US congressman*
- Given Bill is an American, it's extremely unlikely to be a congressman
- Therefore he's not an American
Obviously that's some broken logic, but it's a perfectly valid way to get p < .05
I have a degree in statistics and I've never understood p-values. Even if it's unlikely that you'll get the result you expect with 5% likelihood, there are enough people doing enough tests that you're going to have 5% wrong answers. And that philosophical problem doesn't go away by choosing a different percentage.
Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Where is it written that our null and alternative hypothesis should conform to what is true, when our best ability to predict the future (as people) is based on our past experience? The world doesn't always behave predictably, and yet that seems to be baked into the assumption of every regression test ever run. Completely bonkers if you ask me.
Ultimately, I think what we are trying to do is find out if a result is "surprising", as in it doesn't do what we would expect or it would do what we would expect, which is based on an emotional point of view from the perspective of the tester. Much like in art, in the sciences there is much left up to the muse, although in the sciences the dedication to rigor leads more often than not to a disregard to emotion or just assume that all utterances are by definition rational.
I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change.
But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out.
EDIT:
To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use.
I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument.
But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question?
I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.
p-values are two way decision procedures. For historical reasons we report the critical value of the type I error, because that was the piece people figured out before we got decision theory and the Neyman-Pearson lemma. All the null hypothesis vs other hypothesis stuff is distractions from the basic fact of choosing between two decisions based on data.
> Likewise, we're supposed to assume that there is something magical about our prior assumptions?
No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.
Right, because you can't answer that without prior information. Log-odds is a step towards giving a function of prior information (but it is limited to the actual class of hypotheses investigated).
There's no way to guard against all false positives. And while changing the P value cutoff would reduce them, it would also increase false negatives.
The answer doesn't lie in hard-line stances for or against P values or with an alternative that will have its own set of problems. It lies with greater education of those who run experiments & those who consumer the literature about proper interpretation and other methods of analysis that should be used along side it.
I think duplication and access to raw data would be two huge steps forward. It's actually a bit mystifying that researchers are able to present their own final conclusions & interpretations without making the actual data available.
Start funding psychology etc like we fund physics and the problems would be much smaller. Most studies are underpowered. It’s not because scientists are lazy, it’s because they’re busy.
This is a very old argument. I got my bachelor's degree in psychology at Harvard in 1993, and was told repeatedly that p-tests are abused, overused, and not terribly useful.
To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful.
We were taught that "effect size" measures were crucial to understanding and interpreting results. If you have p<.05 but a tiny effect size, you're likely not seeing a meaningful difference.
Odd that in some ways psychology seems to be more informed, as a field, of these nuances and pitfalls and yet they have failed as a field to adequately police themselves. The problem isn’t on the statistics end of things, it’s the labor market and gatekeeping end of things. Another aspect of the inability to scale higher ed in the stupid way that many think is optimal.
> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful.
Just to clarify. Any study with a sufficiently large sample will find a real effect even if the effect isn't clinically significant.
> To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful.
I don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.
IMO the expression ‘statistical significance’ is a big part of the problem. Popular reporting of research translates a tiny but discriminable effect size into a SIGNIFICANT effect.
Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.
> Changing the nomenclature to ‘statistically discriminable’ would go a long way to improving popular understanding.
I think you overestimate the intelligence of people prone to misunderstanding. I don't think they're likely to understand big words like "statistically" or "discriminable" when they already don't understand big words like "statistical" or "significance".
when they already don't understand big words like "statistical" or "significance".
They do understand what "significant" means, they use the world every day. The problem is that it means something different than the intuitive every day meaning when used in context of p-values. Using a word like "discriminable" might help clear things up since it's a word that doesn't have have so much meaning packed into it already.
It's like when a mathematician says that something is "almost always" true, they mean something very different than when a non-mathematician says something is almost always true.
> The problem is that it means something different than the intuitive every day meaning
Yes that's my point. Using words that are hard to understand or only understandable in the context of P-values won't solve the problem; the problem being that the general public won't understand the subtle differences in meaning
I'm not sure. I think much of the confusion comes from people instinctively applying their every day understanding of the word "significance" and falsely believing they understand what it means in the statistical context. Had it been called statistical confliburance or some other made up term then they wouldn't think they knew what it meant and might find out what it actually meant.
The paper "Frequentist and Bayesian inference: A conceptual primer" has some great things to say about the p-value and why a bayesian approach might solve some of the conceptual problems :
https://www.researchgate.net/publication/326112369_Frequenti...
(Premise) If Tracy is an American then it is very unlikely that she is a US congresswoman;
(Premise) Tracy is a US congresswoman;
(Conclusion) It is very likely that Tracy is not an American
(Premise) If the H0 is true, then is it very unlikely that I will observe result X
(Premise) I observe result X
(Conclusion) Therefore, it is very likely that the H0 is not true
P-values are just a symptom of a much larger problem--the incentives of being an honest researcher don't exist, and the incentives for appearing to generate results are massive. Picking on p-values is like attacking Toyota for contributing to pollution. The problem goes beyond one company and one country.
I am pretty surprised to see no discussion of statistical power in the article and very little mention in the comments here. To me, having more statistical power solves many of the issues mentioned in the article. Many of the rest can be handled with use of Bayesian priors, context-specific p-values thresholds (.01, .1, etc.), and replication.
There's decent working guidelines for statistical power, but a lot of the issues (especially around replication) I see are mainly due to sample sizes being too small to adequately detect effects, especially if those effects are small.
I think the best term would have been statistically surprising, because it strongly hint at the fact that the result would be surprising under the null hypothesis, witch really is all that "statistically significant" really means. Sometimes surprising results happen, but all other things being equal they might hint at the null hypothesis being false. I could also live with "statistically interesting". "Detectable", suggested in another comment, seems to have some of the same issues as significant, it is too strong and seems to imply that now we know something is really there.
By the reasoning behind significance, "surprising" would be a great drop-in. However, in most studies, it would be more surprising if the null hypothesis were true. Statistically significant results are pretty much a given.
Although well written, this article misses the main point why statistical significance leads us astray and needs to be deemphasized. It is because "reproducibility" needs to be the new gold standard that displaces "significance testing". There are too many highly significant results that are totally unreproducible.
"More than 800 statisticians and scientists are calling for an end to judging studies by statistical significance in a March 20 comment published in Nature."
While their sources are supporting this statement, I'm getting mixed signals.
This is their primary source, of the statisticians calling to retire statistical significance. However, their primary reasoning is because statistics is misused to make erroneous conclusions. It seems like there is a lack of understanding about the philosophy and mathematics behind statistics that's the problem by its practitioners, not statistics itself.
Can someone help me understand the concern about p-value hacking?
One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacking.
But in that comic, the only difference I notice between the original hypothesis (jelly beans cause acne) and the p-hacked hypothesis (green jelly beans cause acne) is whether or not the hypothesis occurred to the researcher at the beginning of the study. And I don't understand why that would bear on the importance of each hypothesis.
We already expect data to sometimes indicate significant evidence against the null hypothesis when the null hypothesis is actually correct. That's just the nature of the way we structure our statistics practices. But for each additional set of variables you have and could test for relationships, you increase your chances of finding a strong relationship that appears to be significant due to random chance, but was in fact not. (You may notice that this is just a different flavor of the issues that arise from the way researchers use significance tests).
The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.
Thanks, although I must admit I'm having some trouble following your logic.
IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do.
Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem.
Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne.
But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data?
It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.
141 comments
[ 2.7 ms ] story [ 173 ms ] threadI 'm not sure if abandoning tests altogether is good though. What does it mean "it's detectable but not clear" for communication? How do you e.g. communicate global warming like that?
To get an idea how climate science is trying to communicate look at the summary for policymakers of the SR15 report: https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf
They have different confidence levels they indicate instead of a single cutoff.
The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words.
There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
"Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used."
That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.
There are some scientists thinking over this though. This is an idea from a neuroscience lab: https://www.researchmaps.org/
The idea was to create causal directed graphs for biology from the literature , which would be used identify what experiments are missing and thus inform future science.
> Is there a better way to judge if a study is solid?
> Unfortunately, there is no single alternative that everyone agrees would be better for all experiments.
nobody has solved the trust issue
Not as "This research has now found X and is in a reputable journal, so this is the new state of science from now on".
The problem is, academics are pushed to writing more and more papers. There's no time to discuss back and forth too much about an already published paper, there are new papers to write! The goal of the project was to publish a paper, published=won, end of story, next project on.
Tightening significance threshold just makes this fishing more difficult, Bayesian reasoning doesn't help much either, because you have to guestimate reasonable priors. What does really help against fishing like this is requirement to preregister your studies.
What I predict would happen: - either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for universities and research centers.
- or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done. This hinders research and hinders the dissemination of truly unexpected discoveries.
The human incentives are way deeper than any one solution could touch on. The whole science funding structure requires flashy and sexy results that are just not possible to produce on that scale.
I see two things going on here.
Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing.
I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.
Though I would have personally used statistically instead of strictly in the sentence: >Strictly speaking, he says, “there’s no difference between a P value of 0.049 and a P value of 0.051.”
Everybody wants to compute p(H|x), the probility of a scientific hypothesis given the data. People want to do this so badly that they can't help interpreting the p-value that way.
You can actually compute p(H|x) if you use Bayesian stats.
Typically people doing research have prior information about what they are researching. E.g. previous studies have found effect sizes to be in some interval.
You can fallback on an uninformative prior or other tricks if you really want to model the idea that you know literally nothing about what you're researching. But that should be very rare.
Our 0.05 p-value limit is effectively just answering that question with a fixed 5%, no matter how ridiculous the proposition is.
You can't "compute" it for any useful meaning of the word "compute". You can estimate it intuitively, or you can try to look at how many pre-registered unpublished studies or null results have been published. Otherwise there's no way to even consider getting a grasp of what P(H) would be.
That's called picking a prior. Usually, pick the prior that maximizes the entropy on the given interval (zero prior knowledge).
For those reading and confused. Computing: p(H|x) = p(x|H)p(H)/p(x)
requires p(H) which the parent is suggesting is impossible to grasp.
If we want to know the probability that a coin is biased, we can assign probabilities to each hypothesis. For some people, p(bias=1/2)=1 (Dirac distribution), others might argue that p(bias=x)=1 for x in [0,1] (uniform distribution). Others might argue its some Beta function centered around 1/2. I believe what the parent is suggesting is that choosing which original belief we have in the system is a matter of philosophy, not computation.
Essentially, we want to get p(H0|x), but we need Bayes Law to get this from p(x|H0). But we need some notion of what priors to use. This is of course impossible to actually get, but if we published null studies then it would allow us to estimate it with better confidence. Alternatively we can ballpark it by saying how unexpected the result is, and how long the effect being explained has been studied.
The "common-sense" version of this is intuitive, that "extraordinary claims require extraordinary evidence". The more unexpected a result is, the lower the p-value has to be to be convincing.
The epidemiological version of this shows up in things like the Bradford Hill criteria [2], which include significance of association but also attempt to bring in plausibility.
[1] https://xkcd.com/882/
[2] https://en.wikipedia.org/wiki/Bradford_Hill_criteria
[1] https://statmodeling.stat.columbia.edu/2015/03/02/what-hypot... [2] https://statmodeling.stat.columbia.edu/2015/07/03/why-should...
It seems like something that should be taught alongside math from elementary school, not something you can maybe get an elective in in high school or college.
I actually found statistics pretty fun and eventually we learned to use R for it. Though I ended up missing 1 lecture and it almost screwed me over for the rest of the class. I found a Kahn academy video though that covered it and was able to catch up. There really is a ton of information to learn. It would have been nice to have at least touched on it a little bit in highschool beyond learning about means and the tiniest bit about standard deviations.
I know a lot of people have trouble thinking clearly and correctly about probability and statistical inference.
But do we know if, practically speaking, that can be addressed by a modified educational curriculum?
Or are these concepts that would take an extraordinary amount of effort for many persons to understand well?
[1] source: I never saw it as a student, and I've consulted with math teachers (well, administrators, about math) in multiple states, and I've never had it come up there either.
The problem is that we are imbuing the words "statistical significance" with a whole bunch of math. This would be fine, except for the inconvenient fact that people also want to use the word "significance" as it is defined in English.
It is not only possible, but likely that people will be producing results that are insignificant but statistically significant. I mean seriously, if I tell my boss that the results are statistically significant, how do I expect him to understand that the results might reasonably be insignificant? How do I expect anyone non-technical to take that sentence seriously? You lose all credibility pretty quickly when people start saying that the result is significant except it doesn't matter.
This is even more stupid than calling complex numbers "imaginary" and "complex". Those names are just arbitrary. Statistical significance is overloading words that are likely to come up with meanings other than what they are expected to mean.
Anyway, this is the problem that things like this article are going to. People are treating significance values as though they are significant. They aren't significant, they are statistically significant. It means something different to significance.
For example; if I run an experiment 20 times and get a p value consistent with a 5% chance in one experiment, it is statistically significant but the result is obviously not significant in the English-language meaning of the word.
In fact, failures to reproduce "landmark" experiments have low significance (statistics word) because they find a result that is highly probable if the null hypothesis is true. But failures to reproduce have high significance (colloquial English word) because they indicate that widely-held beliefs are wrong. So in some of the most important cases, the statistics and colloquial English meanings of the words are exactly the opposite.
I don't know if there's a good solution to this, though--it's hard to change organically-emergent trends in language even if we did come up with a better term.
If you look at research on carcinogens, it is frequent to see something like "it is statistically significant that eating chemical X increases your risk of cancer" but when you read on, the magnitude of the effect is "increases risk by .0023%, for a cancer that already has a .6% incidence in the population".
My point is that people who are not statisticians (or people who use statistics), tend to be concerned with the magnitude of an effect, and frequently misconstrue the statistical significance for that magnitude.
But these concepts aren't clear to the people using the data. I've seen people use statistical tests done in batch without correcting for multiple testing (one of my regrettable tasks), and flat out ignore any differences that weren't statistically significant. They thought it meant "no difference."
With population-level data (e.g., Census counts or hospital records), I sometimes wonder if statistical significance has any value. It's a tool to make sure we don't accidentally claim a difference where there isn't one. But I can guarantee you that nothing worth measuring is exactly the same for men, women, whites, blacks, young, old, whatever. It'd be amazing if the difference was 0. So statistical significance ends up being an obfuscated way to state the sample size.
My decision to select, implement, go, or no go cannot itself be a probability distribution.
The problem is that people want an answer to the question, "Here is a pile of data, what should I believe?" But mathematically, the proper role of data is to modify existing beliefs, and not to dictate beliefs.
Statisticians can spend forever explaining this. But instead have gone with the cop-out of asking a question that is confusingly similar to the one that people want to ask. It is popular exactly because it is so easily misunderstood. You can give any number of lectures on what it actually means - I guarantee that it will be misunderstood.
Worse yet, p-values are sensitive to particulars of experimental design that logically should never matter to inferences. My aunt and uncle provide a classic example. They wanted a son and a daughter. They had 6 sons then a daughter. Are they biased towards one gender? The p-value for this question is .03125 (we'd see this much evidence against with all boys, all girls, last girl, last boy, so 4/2^7). But if I instead said that they planned to have 7 children, and had 6 sons then a daughter, the p-value changes to 0.125 because the one child out could have been anywhere in succession. But in Bayes' formula, their plans if something else had happened cannot ever affect how we adjust our inferences.
(This example is not made up. When Lorna found that they had a daughter, she told Bill to get a vasectomy from the delivery table. By another crazy coincidence, their 7 children were also born on all the days of the week.)
Its not just about p-values or the likelyhood of making a particular type of error by agreeing with a thesis.
There is a fundamental misunderstanding that most people, and even many scientists make with regards to the philosophical interpretation of 'truth' using the scientific method.
By definition, scientific 'truth' is always fallible, and the vast majority of people have a very hard time dealing with this concept. A better explanation or additional data, or better data is always possible and needs to be possible for science to not be dogma. This is fundamentally different than how people interpret truth and understand what the word means. All scientific truths are replaceable.
That's not how statistics works. Frequentist p-values and Bayesian inferences are both entirely dependent on the mathematical model in question. "Their plans if something else had happened" are a key part of the model. What you're describing at 2 entirely different experiments:
1) A couple has children until they have 1 of each gender. they have 7 children. How likely is a result this extreme (or greater)?
2) A couple has 7 children. They have a gender ratio of 6:1. How likely is a result this extreme (or greater)?
The probability of event A given that we observed B is the probability of A and B happening divided by the probability that B happened. Mighta, coulda, shoulda but didn't doesn't enter into it and can't affect the result. The fact that frequentist statistics does care and shouldn't is one of the major criticisms that Bayesians offer.
If you think you understand statistics and don't understand this fact, then you do not understand statistics as well as you think you do. But you can be pardoned. Most statistics classes are too busy cramming statistical tests into student's heads to bother them with bothersome facts about where the cracks in the foundations are.
So true. That said, here's a great interactive explaining the p-value: https://www.jwilber.me/permutationtest/
In my work as a data scientist, 99% of the time this is not even relevant. We have huge sample sizes and most of the p-values we see are <.0001, meaning there is SO much data that even a very very small effect size can be found "significant."
The question then: does a mean of 20.02 being significantly different from a mean of 20.03 over 10 million patients, is that actually important to us?
[1] http://computationalimagination.com/article_practical_signif...
The big problem is that since we've already long since entered the world of big data you can now find spurious correlations, similarly strong, but constrained to variables that are plausibly connected. For instance what if this correlation was between spending on science/space/technology and an increase in the number of people pursuing STEM fields? People would immediately just accept it without question, even though it should in theory be held to the same scientific rigor and critique as one that challenges our biases. Science should not be a glorified exercise in confirmation bias. Spurious correlations don't mean you have a biased sample or that such correlations don't exist - it simply means that correlations have no direct relationship.
538 has a great little interactive p-hacking game you can play, with real data. [2] Ultimately any single datum that tries to legitimize a piece of research is going to be gamed. Science should be judged as it was in its 'glory days' - by the logic, predictability, and falsifiability of the ideas presented. Anything that steps outside these bounds should be treated with the most extreme of prejudice. It may be legitimate but the great burden of proof is on the presenter, and that doesn't come down to p-hacking up a 0.00001 correlation and calling it causal.
[1] - http://www.tylervigen.com/spurious-correlations
[2] - https://projects.fivethirtyeight.com/p-hacking/
If we required scientists to be good statisticians there’d be far fewer scientists.
P hacking isn't always done on purpose. It's misunderstanding. That's probably why it still passes peer review. There's also tons of incentives that encourage this behavior. These are the problems. An arbitrary mark to meet makes people weak at statistics not understand their data as well. I'd rather more accurate data than more scientists. More workers doing poor work isn't useful.
Also, you can Bayes hack. Bayes helps, but it doesn't address the underlying issues.
In my view, understanding p-values won't help. Feeding data into a formula that produces a p-value doesn't account for things that can go wrong with experiments: Uncontrolled experimental conditions, biased sampling, hidden correlations, and simply non-ergodic systems. It is possible that some systems can't be controlled to the point where it's reasonable to start looking for real effects.
I suspect that if the results were any good, the precise interpretation of the p-value wouldn't matter. Physics has lived without an agreed-upon interpretation of quantum theory for a century.
People that work with computers often are slightly better about reasoning about probabilities because they often deal with high frequency events. A 1% error rate across 10 million events a day leads to 100,000 errors. If those events are transactions, payments, file uploads etc, a 1% chance starts to look very common. But even computer people are not immune to incorrectly reasoning about probabilities. I suspect there is some fundamental limitation at the cognitive level.
You might see p=0.00001 but if the size of the effect irrelevant, then the statistical significance of the relationship is still not something I care about. Great, orange juice shrinks tumors by 0.000001%. I'm still going with chemo, thanks.
The problem isn't that P=.05 is an arbitrary measure of significance. The problem is that only publishing significant results is a bias against the null hypothesis.
Let's say you're doing a study of flipping coins. The null hypothesis is that the coin is evenly weighted. If the null hypothesis is true, when you flip a coin once, it will come up heads with P=.5. If you flip the coin twice, the null hypothesis is that both flips come up heads with P=.25. The probability of all coins coming up heads is P=0.125 for 3 flips, P=0.0625 for 4 flips, and P=.03125 for 5 flips. So if we flip a coin 5 times, and get heads all 5 times, we can conclude that the coin is weighted in some way with P=0.03125.
Let's say all the major journals of coin flipping only publish results with the high significance of P<0.05. Alice flips a quarter 5 times and gets 3 heads and 2 tails, and nobody will publish her study because it has P=.3125 (see [1] for an intuitive explanation of how this was calculated). Bob flips a quarter 5 times and gets 2 heads and 3 tails--again, no journal will publish him. Catherine, David, Ellen, Frank, and Geri all perform the same experiment, most of them getting groupings 3:2 result ratios, some getting 4:1 result ratios, but nobody getting all heads or all tails, just as one would expect given the null hypothesis. And the journal editors tirelessly send out rejection letters to all their studies.
Now somewhere down the line, Robert flips a coin 5 times and gets 5 heads. This result has P=.03125, which meets the requirement of P<.05! He sends it to the American Journal of Coin Flipping Studies (AJCFS), and they are very excited to publish his results! Nature and Science magazines do front page pieces with headlines "Quarters Found Heavy-headed" and "Washington Shows His Face" respectively. A casino hires Robert as a consultant for the design of their coin-flipping games. His quarter-flipping study is cited in the abstracts of two dime-flipping studies and a half-dollar flipping study. During the trial of a murderer who placed quarters tails-side up on his victims, Robert is called as an expert witness to say that the coins were placed there, not flipped there.
A week later Sally flips a quarter 5 times and gets 2 heads and 3 tails. She sends the results of her study to the AJCFS noting a failure to reproduce Robert's result, but her study is rejected because it has P=.3125.
Now, if you survey the AJCFS and all the other academic journals on coin flipping, you'd conclude that quarters are significantly weighted towards heads. But in fact, the null hypothesis is true: quarters are pretty evenly weighted. Robert's low-P result is exactly what you'd expect to happen eventually if you have enough people perform the 5-quarter-flip experiment--in fact, if a lot of people are studying coin flips, the P of getting a low-P result approaches P=1. But because the AJCFS has a P=.05 requirement, they've created a bias against the null hypothesis, which deceives the public into thinking that flipped quarters are more likely to come up heads than tails.
This is likely the reason why so many fields, most notably psychology[2], are having a replication crisis[3] and a similar effect can be used in P-hacking[4] to bolster results that are essentially fake.
Unlike coin-flipping, fields with replication crises like psychology and medicine have real affects on real people's lives. It's irresponsible and unethical for journals to publish with a bias against the null hypothesis, and adjusting the P-value requirements to another significance requirement, even a less arbitrary one, doesn't fix the issue.
The solution, I think, is for journals to commit to publish studies before the study has been performed, based on the methodology, previous studies on the subject, and qualifications of the researcher. This would mean that man...
I have not done this sort of work since the early 2000s, I am relying on memory and a couple of quick web searches.
The main article kept saying that p values were not meant to be definitive, but without explaining what was wrong with them. At least, not very clearly.
This comment is a much clearer explanation -- imho -- as to what goes wrong in the industry. I also think it is right to focus the attention away from a particular statistical measure.
I have the opposite problem: my abilities lie more in the statistics/science than the communication--I suspect the only reason that this explanation is being received so well is that I happened across an effective example by pure luck. The probability of me communicating effectively is probably P < .5. ;)
For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups.
Let's calculate how unlikely a P=.006 event is if you try 47 times to find it:
A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2]
B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1).
C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits).
D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246.
So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length".
The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a P<.05 result. Doing the same math again for the probability of a P=.05 result occurring if you try 47 times, P'_47' = 1 - (1 - .05)^47 = 0.91025516807. That's over a 90% chance! In other words, by doing the test on 47 food groups, they nearly guaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place.
This study has not been replicated as far as I know, but is often cited in pop science[3][4][5].
[1] https://academic.oup.com/ajcn/article/88/5/1405/4649028
[2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51).
[3] https://www.livestrong.com/article/506649-foods-that-boost-t...
[4] https://resources.teloyears.com/diet-and-telomeres/processed...
[5] https://siimland.com/how-to-increase-telomere-length/
medicine is so ancient and people havent realised it yet. I cant wait for it to change
It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis.
This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
The current approach also ties the experimental method to the result, requiring potentially fallible oversight of the experimental design. The hypothesis-testing aggregator reduces this reliance, which is perhaps more honest about our ability to design good experiments, would allow for a greater diversity of approaches, and could also be more efficient when evaluating results.
The most interesting aspect of this is if you could somehow create relationships between standardized hypotheses found on the hypothesis menu such that you could infer other hypotheses in a more systematic way.
You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science.
> This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result.
"Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue.
A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).
No we can't. There are 2 (related) reasons for this. The first is that to say "after experiment X, we can conclude thing Y is happening with probability ..." you need to know something about how often thing Y happens on its own. This is also known as a prior. The other reason is that in this specific case, the prior for Y (that the coin is weighted) can be concisely summarized as P(Y)=0, because it's physically impossible to construct a coin that's biased to one side or the other (independent of what side it starts on). Some people think the coin flipping examples are bad because biased coins are impossible, but they're actually good for exactly this reason. We have a really good prior (from math/physics/geometry) that biased coins can't exist, so you're going to have to not only flip a lot of them to convince me otherwise, but also give me an explanation of how that's possible (i.e justify your prior)
Sally must be working using the wrong null hypothesis. If she attempts to reproduce Robert's results her null hypothesis should be whatever is the one he formulated. Perhaps that the probability of tails is at most 15%. Her p-value would be 2.66% (prob of seeing <= 2 heads) and if the world was fair, AJCFS would publish her study. But the world isn't and if Robert is a renowned scientist, significant+ is enough for him but failed replications of his work requires significant+++.
The most obvious criticism that stands out to me is that when a control group is possible, that's where your null hypothesis should come from. But in the contrived coin-flipping example, there isn't a natural control, and in practice, many times control groups aren't available. So I don't know if that criticism actually applies to this situation.
The next thought I have is that if your goal is to determine significance (which should be the goal), rather than to find a significant result (which should not be the goal), using the same alternative and null hypotheses makes the math for aggregating the results easier. The power of replication is that it's multiplicative: you can get an aggregate P by multiplying the P values of aggregate studies. This means a P=.2 study can become significant if it's replicated (P=.2.2=.04), while a P=.03125 study can become lose significance with just one failure to reproduce (aggregating Robert and Sally's results, P=.03125.3125^-1=.1). I guess swapping the null and alternative hypotheses is a viable strategy for bringing attention to failures to replicate and it's not mathematically invalid, but that seems like changing your hypothesis to fix a problem with the academic publisher, and it complicates aggregating the P-values of replication studies.
Finally, it feels weird to me to use Robert's hypothesis as your null hypothesis. When Robert performed his experiment, his hypothesis was a stab in the dark due to a lack of pre-existing data (since he ostensibly didn't know about the other coin flipping experiments). But when attempting to reproduce a result, you have data, so it makes more sense to base your null hypothesis off the results of the previous experiment, not off the hypothesis of the previous experiment. As such, since Robert had 5 heads, the hypothesis would be that the probability of getting tails on a single coin flip is P=0, which gives Sally a VERY significant result. The numbers are weird because the experiment is contrived, but I think the general concept holds.
It's not the statistical analysis that's the problem, it's the bad "science" and irresponsible journalism.
A simple example of this can be shown with the following pair of tests:
Testing for a fair coin:
Okay, that makes sense, but this is logically the same as:Testing whether a person "Bill" is an American:
Obviously that's some broken logic, but it's a perfectly valid way to get p < .05The purpose of a paper is supposed to be to contribute something new to a field, not just perform a statistical study.
But you or I cant say that now as so many organisations use "studys" as the basis for their field while maintaining that they are scientific.
For me, the fact this is even being discussed really is good news.
Likewise, we're supposed to assume that there is something magical about our prior assumptions? Why? Where is it written that our null and alternative hypothesis should conform to what is true, when our best ability to predict the future (as people) is based on our past experience? The world doesn't always behave predictably, and yet that seems to be baked into the assumption of every regression test ever run. Completely bonkers if you ask me.
I would probably keep the p-value, or something like it, for the rigorous analytical side, but I would make the assumptions to be tested much more rigorous. I would want to know not only why the test should be surprising, but I would want to know if our level of surprise is going up and down over time. It may be, for example, that as we accumulate more knowledge over time we are less and less likely to be surprised (or maybe more!) and therefore we should expect the p-values that elicit a surprise or not to change.
But overall, I think there is not enough emotional soul searching over what it means for a mathematician to be surprised. And since that is the primary axiom over which everything else follows, and a lot of statistical papers don't really think this through well, there is a ton of faulty analysis out there. Garbage in garbage out.
EDIT:
To give another clearer example - Suppose we want to test if a miracle drug makes people immortal, to give an obviously ridiculous premise. Then in this case maybe a p-value of .001 is most appropriate because it would overturn huge amounts of medical knowledge. In this case though we need a judgement call backed up by evidence, in the sense of evidence to convince another such as in a law trial, to make a convincing case that this is the "correct" p-value to use.
I think there is a desire in science to simply say, "Through sheer logic I have found the answer and therefore charisma/politics/judgement have no bearing on my analysis. And therefore I cannot be disputed on such things". This is an emotional viewpoint that seems in contradiction to the evidence, and it seems a common enough viewpoint that I don't believe it's purely a straw-man argument.
But then if the politics do exist how do we get around them? After all, if we start making broad based judgement calls on what p-values should exist then how do we make sure papers are not disputed for begging the question?
I would say that the best solution would be to force statisticians to get a third party to give them an appropriate p-value. Then the science has quickly, as many things, become a social problem - how do we organize such a system of statisticians giving each other appropriate p-values such that it is accurate but not corrupt? Unfortunately, this sort of thing is generally considered a "hard" problem.
I wrote an explanation for clinicians a while back that I people have had good luck with: http://madhadron.com/posts/2016-01-25-p_values_for_clinician...
> Likewise, we're supposed to assume that there is something magical about our prior assumptions?
No, we construct trials that a reasonable practitioner thinks satisfies the assumptions of the analysis used. And sometimes we learn that we had overlooked something and all those trials we did before we accounted for it were flawed.
Because that's how the math works. It isn't possible to compute a posterior without first having a prior, so you have to decide on a prior somehow.
You can dress it up and try to hide it, but you can't avoid Bayes' theorem forever.
The answer doesn't lie in hard-line stances for or against P values or with an alternative that will have its own set of problems. It lies with greater education of those who run experiments & those who consumer the literature about proper interpretation and other methods of analysis that should be used along side it.
To my mind, the most hackable flaw is that the number of subjects in the study is a term in the denominator of the p-value calculation. Any study with a sufficiently large sample will find "significance" with p<.05, but it won't be meaningful.
We were taught that "effect size" measures were crucial to understanding and interpreting results. If you have p<.05 but a tiny effect size, you're likely not seeing a meaningful difference.
Just to clarify. Any study with a sufficiently large sample will find a real effect even if the effect isn't clinically significant.
I don't think that is right. Think about tossing unbiased coins. The probability that you will get an improbable (p < 5%) result is exactly the same no matter how many or few coins you toss.
I think you overestimate the intelligence of people prone to misunderstanding. I don't think they're likely to understand big words like "statistically" or "discriminable" when they already don't understand big words like "statistical" or "significance".
They do understand what "significant" means, they use the world every day. The problem is that it means something different than the intuitive every day meaning when used in context of p-values. Using a word like "discriminable" might help clear things up since it's a word that doesn't have have so much meaning packed into it already.
It's like when a mathematician says that something is "almost always" true, they mean something very different than when a non-mathematician says something is almost always true.
Yes that's my point. Using words that are hard to understand or only understandable in the context of P-values won't solve the problem; the problem being that the general public won't understand the subtle differences in meaning
(Premise) If Tracy is an American then it is very unlikely that she is a US congresswoman; (Premise) Tracy is a US congresswoman; (Conclusion) It is very likely that Tracy is not an American
(Premise) If the H0 is true, then is it very unlikely that I will observe result X (Premise) I observe result X (Conclusion) Therefore, it is very likely that the H0 is not true
There's decent working guidelines for statistical power, but a lot of the issues (especially around replication) I see are mainly due to sample sizes being too small to adequately detect effects, especially if those effects are small.
The basic idea is: Get more data, especially if you're measuring something subtle.
I am very happy to see it in Nature.
While their sources are supporting this statement, I'm getting mixed signals.
https://www.nature.com/magazine-assets/d41586-019-00857-9/da...
This is their primary source, of the statisticians calling to retire statistical significance. However, their primary reasoning is because statistics is misused to make erroneous conclusions. It seems like there is a lack of understanding about the philosophy and mathematics behind statistics that's the problem by its practitioners, not statistics itself.
One of the comments below references this XKCD comic [0], which IIUC is an example of p-hacking.
But in that comic, the only difference I notice between the original hypothesis (jelly beans cause acne) and the p-hacked hypothesis (green jelly beans cause acne) is whether or not the hypothesis occurred to the researcher at the beginning of the study. And I don't understand why that would bear on the importance of each hypothesis.
[0] https://xkcd.com/882/
The reason you need to pick your hypothesis ahead of time is because otherwise you may use intuition or just your eyeballs to find relationships in the data that are only there due to random chance. This will supposedly increase the probability of detecting spurious relationships and erroneously rejecting the null.
IIUC, you're pointing out that when examining the empirical data gathered during an experiment, it's often possible to find some identifiable subset of the data that are consistent with a refined version of the original hypothesis. E.g., maybe jellybeans in general don't correlate with acne, but green ones do.
Assuming that the experiment is part of a larger effort to build or refine some model, I don't see the problem.
Suppose that someone refrained from p-hacking during the first run of that experiment. IIUC, they'd look at the experimental results, and wonder if they had missed some "X" factor. So they might conject that jellybean color was relevant, and rerun the experiment with the hypothesis, "consumption of jellybeans, but only of a particular color, correlates with acne." And (assuming sample sizes were big enough), the data gathered during that second experiment would likely confirm that green jellybeans correlate with acne.
But what's the point of having run that second experiment, when they could have just reached the same conclusion by testing additional hypotheses from the first experiment's data?
It seems like regardless of whether you just ran one experiment and did "p-hacking", or instead ran a follow-on experiment, you end up with the same refinements to the model you're working on.