They should change the nomenclature "significant", which in normal language tends to imply a large size to statistically "detectable" or "discernible".
P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise".
Then researchers should always put more emphasis on confidence intervals. Readers of papers can then see if the size is enough to make the results relevant and not likely caused by experimental accident. Plus given the perverse incentives researchers are subject to, maybe assume that the real effect size is probably closer to the lower bound of the interval.
Null hypotheses have limited usefulness even at p=0.005. A tiny systematic bias in the experiment can make it cross that threshold and there are _always_ at least small biases. These can be caused by not exactly calibrated instruments or small differences in the way tests are performed by different researchers on the team etc. Inherent in null hypothesis testing is this ridiculous assumption that there are no systematic biases. This is its fatal flaw.
The null hypothesis test never tells you that the effect is zero or that it is any other value, it only gives you an hint about whether or not the data is too noisy to say anything at all.
>P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise".
No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.
When I say that "There is a 95% chance that this data isn't pure noise" I'm being slightly indirect. More directly it would be something like "There is 5% chance that pure noise generator you would give you this result" but since I want to reason about the actual world that the experiment is trying to study, I think it is legitimate to do the Bayesian flip and infer the more useful first statement that the world is likely too noisy to detect an effect using this experiment.
I think I understand what you're trying to get at, but I don't think describing a p-value in terms of "pure noise" is quite right. The null hypothesis may well carry with it all sorts of specific assumptions: independence, finite variance, maybe even specific probability distributions (and not necessarily the normal distribution). Typically somewhere in there we define a θ parameter and then include something like θ = 0 that provides grounds for possible rejection. We think of "θ = 0" as the null hypothesis, but all the other assumptions are part of the null hypothesis as well; the statement "θ = 0" does not make sense without them.
If you see clouds how likely is rain? If you see rain how likely is it there are clouds? Are the answers to these at all the same? No.
There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip".
Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition. Experiments are useless otherwise. There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language.
Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
>"If you want your science to be about the real world and not about some hypothetical scenarios that don't exist you have no choice but to do the transposition."
How can saying false things make your science "about the the real world"? The only thing I can think of is that by "real world" you are referring to using the publication of massive amounts of incorrect/questionable information as a metric for success. Yes, that is indeed the sham going on.
>"There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language."
I have no problem talking about data without transposing the conditional, if you do that is some issue with your training.
>"Sure there are debates about priors."
The error you advocate (of transposing the conditional) has nothing to do with priors.
There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying, if it's not for building a predictive model of the subjects. This is the entire reason for research.
Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior.
>"There is no point in talking about the data if it doesn't tell you useful things about the population or type of thing you are studying"
Ok, but I fail to see the relevance of this to anything.
>"Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior."
Here, A = Hypothesis and B = data (or vice versa). You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"?
Based on what? What principle leads you to think these two probabilities should be near each other? BTW, if you actually have one this is like the holy grail of statistics.
> You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"?
Shouldn't that be "the probability of seeing the data, _given_ that the hypothesis is correct" and "the probability of the hypothesis being correct, _given_ that the data has been observed"?
I agree there are debates surrounding priors, especially when you get into higher order, meta or purely hypothetical territory (there is no actual noise generator here) like this, but I think these are unavoidable.
For this to be true, P(observations) ~ P(pure noise generator) as you said earlier. Anyway, you can look at it from either perspective, doesn't really matter.
On what basis do you claim this is reasonable to expect? I suspect you have none whatsoever... in fact simple thought experiments will lead us to the opposite conclusion, and there is plenty of data telling us the same.
> P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise".
This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.
"there is a 5% chance of the data being noise" is a statement on the probability of whether the phenomenon is not real given the data: p(it's just noise | data). A p value is a statement on the probability we'd see this data given that the phenomenon isn't real: p(data | it's just noise).
Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
Right at the end is a key thing: reward failure better.
Publish negative results.
I'm kinda embarrassed by my negative results, such as they are, but I'm clear in this latest case that I wanted the method reviewed most, since others then may be able to use it.
(Warning: not a real scientist, but being validated by real scientists in the case in question.)
I'm not a scientist, but this is very surprising to me. I would have imagined that our scientific culture would be one where any result is recognized as something from which valuable knowledge can be gained. The idea that unanticipated results reflects some failure on the part of the researcher seems so strange and unscientific. But again, I'm not a scientist.
In an ideal world, scientists would be judged solely on their study design and execution without considering the outcome. But that takes a lot of effort. It's much easier to judge based on the results that the scientist achieved ("I discovered the priming effect"). We remember Watson and Crick, and forget many other scientists who worked just as hard.
Science doesn't really have a place for participation trophies. That and it's pretty natural to judge based on results. When handing out recognition and fame, the results are probably the best criteria - though that should include those who helped and that each stood on the shoulders of giants. That information should be included in the papers for those who are curious.
Negative results are real science, and science is collaborative. When we don't incentivize negative results, scientists don't take the time to analyze and share those findings, which results in a shallower understanding and possibly wasted effort by other scientists. It also biases scientists towards claiming results where none exist (e.g. p-hacking).
The solution isn't publishing negative results. The solution is publishing better results. The solution is holding people to a higher standard and insisting on reproducibility before accepting the findings as factual.
There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in.
Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you.
But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results.
If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable.
Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done.
The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it.
Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers.
You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable.
Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality.
I will add this...
Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available.
So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/ins...
There was no relationship between variables is not participation trophy. It is a result and should count as one. As long as science is concerned with finding truth.
If you want to see the negative results, contact the researcher. There is no room, publishing mechanism, or benefit to most negative results. If the negative results are novel or reveal a new method, they will get published.
Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing.
Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful.
If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value.
It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems.
The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.
What if there were an open platform where these results could be published? Not only published, but the data related to variables could be fed into a database. So if you wanted to check which variables have been found to not be correlated, you could query the database. It seems like that could be useful for scientists who want to avoid wasting their time.
Most ideas people have suck. That's sort of the problem of new ideas in particular. When you're working, their failures aren't recognized as some kind of thing you need to tell the world about, you just tried something speculative and it didn't work out. Writing papers sucks (though it can be ego satisfying if you did something really cool), and writing papers about negative results that no one will publish sucks even more (or give you money for). Also, if you do something really dumb, now everyone will know about it.
Yet people still do studies on alternative "medicines" and other stupidity like "wind turbine sickness." That is literally doing something dumb that everyone will hear about.
Doing studies on those things you think are stupid is the right thing to do. Counter-intuitive stuff is found all the time in science. Occasionally those alternative medicines get tested and turn out to work, in which case they become regular medicine.
By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.
Which is funny, because in science we narrow down answers by crossing ideas off a list. Not so much proving an answer as disproving others. But you don't get funding, especially in the softer fields, when you don't produce positive results. Unfortunately this also sows distrust in other fields.
Underpowered negative results are a class of potentially genuinely uninteresting results.
Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives.
So, underpowered results, whether positive or negative, are less interesting.
Pre-registration of studies is an interesting concept in that regard (also briefly mentioned in the article), where researchers can submit their study design and goals to a journal before conducting the study. It will then be accepted or rejected for publication based on the study design etc., not on the eventual outcome (as long as the study was actually conducted as promised and with care, of course).
The idea is probably only applicable to a small subset of scientific work (not good for exploratory and very novel stuff, for instance), but in certain specific areas it might make publication of negative results a more viable path for scientists because they don't have to fear that their idea was bad or they seem incompetent because of a negative result. (Since their design was accepted, at least some other people thought it was a good idea too.)
The problem is the incentives don't change with a system like that, all that will happen is you have people rigging the pre-registers after they have already completed the study then sitting on results until the correct amount of time has passed.
That's a very overt form of fraud, and it probably requires very different countermeasures than the kind of unintentional biases under discussion here.
Many have discussed this, but like writing bad/buggy code, there are just too many (combinatorial) ways to get a negative result, and most of them are frankly boring, distracting, or worse than useless. You would need a review paper of all the most common failures, screwups, and anti-patterns to be particularly useful... preferably with a good write-up of proper procedure. Some labs publish good protocols, but many (academics!) consider them to be secrets to used for advantage over other labs.
Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.
This sounds like a great case for publishing negative results: it will result in meta-analysis about what practices routinely cause bad results and then those papers will be used to reduce the number of bad results. Labs keeping secrets in how to do research seems like a bug not a feature.
That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.
I agree, there are always more way to make mistakes, 2nd law and all, but the process of assessing good study design, good protocol, good statistical methods, etc. are all the same as for manuscripts with non-null results. Many of these issues will, in theory, have been sorted at the funding level.
What holds us back, it seems to me, isn't process, it's culture and inertia.
The code analogy is ok, but remember that the natural sciences are not engineering - a problem to be overcome.
There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.
Er, the type of the result(positive/negative) should be orthogonal to the methodology, from the point of the reviewer. Reviewers should be checking for all these many combinatorial ways to get an incorrect result, whether it's positive or negative.
It may be true that positive results are more useful than negative ones, but that doesn't mean we should put zero value in negative results. I think that's what parent was trying to point out.
Agreed, a negative result should not be confused with shoddily executed science. A solid methodology that produces a negative result could save other researchers from going down the same rabbit hole to come to the same conclusion. We should record these conclusions. Perhaps maybe part of the problem would be the increase in number of publishable results overloading editors and reviewers?
I fully agree. Also, I think (as embarrassing as it is) we need raw data and code. Would make peer review much easier as you could perform the same analysis on the same data. Makes it easy to correlate results and validate premises of data sampling methods.
You don't need a degree to be a real scientist. You just need to follow the process. If anything, I think the discouragement of the citizen scientist is a bad thing. I think more people should try.
FWIW I have a couple of degrees, but the topic I'm outlining is most definitely not rocket science. Indeed, it's rabout educing something people assume is hard and expensive to some meter readings and a bit of simple Excel jockeying or similar. That method is published and being validated (and we should get a shiny cert from a prestigious lab soon), along with the code and inconclusive results (not enough data, so uncertainty too high).
Absolutely go for the gusto. Get your name attached to the paper, if you indeed earned it. It's even more awesome if you can get your name listed as the primary.
It's rare, but sometimes people without any degrees get published in reputable journals.
If I were in your shoes, I'd do what I can to get published in something reputable and buy a bunch of printed copies. I'd then autograph them and hand them out like a rock star.
When I was first published in a journal, I brought a copy of the journal and a couple of 'official' single article prints for my mother. One of them she hung on the fridge and even stuck a gold star sticker on it.
Of course, buying a copy of the whole journal and single article reprints was a lot less expensive then. I'd still get physical copies, just to show them off.
Best of luck and feel free to yell, both when you've got it published and if you have any questions about formatting or want a proofreader. If it encourages you to publish, and encourages people like you, I'll do it. I really do wish there were more citizen scientists, or at least people who lack the standard credentials, submitting their ideas, being interested, and being willing to share their insights and research.
Either way, best of luck! You've made my night. I love stories like that.
uninvolved@outlook.com (generic public email address)
I think that you may be confusing me with someone clever and industrious!
I do have my name on a pukka scientific paper from many moons ago, as the summer help for a PhD, where the real authors slung me in the author list out of kindness.
But I do like the idea of a gold star. Then I can be 'award winning'! B^>
That was molecular biology, this topic is home heating efficiency. That one is online for all to see, this one is too but is a bunch of very boring docs and part of a GitHub repo!
Only badly designed experiments overrely on p-values.
A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test.
Obviously, these tests are important, but people focus too much on the p-value rather than designing an experiment which will produce a meaningful result (regardless of whether the result is positive or negative).
Proper experimental design is like a cake, and p-values are the frosting. Everybody wants cake, most often with frosting. Nobody wants frosting without the cake, or at least those that do have questionable taste.
Not all phenomena are strong enough to allow such experiments. Some make only a small difference to an inherently noisy process. These can only be discovered with statistical tests on substantial samples.
Certainly, but a well-designed experiment can identify the wedges that differentiate hypothesis A from hypothesis B without requiring a statistical analysis of noisy data.
And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter.
In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
Are there many real examples of studies where statistical tests are needed to see the effect, but a different experiment could demonstrate the same effect without any statistics? I haven't been able to think of one in the fields I know well.
In my field it often comes down to 'mechanism'. If you see an effect of part X on part Y, via statistics, you could write a paper, and do good science.
However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment.
The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. However the scientific results from that experiment really don't hinge on statistical signifigances. The results are either binary in result, or better yet, contradictory - elucidating hitherto unknown variables (something a purely statistical result won't help with). And if mechanism is shown in multiple systems, with multiple techniques, the results can very quickly become near-definitive.
> However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment.
Yeah, in molecular biology this always seemed to be the most convincing evidence. Deleting a gene and then adding it back. Sometimes going a step further and overexpressing the gene. You can generally publish that data without any statistical analysis.
> The later experiment is often harder, and might not even be proposable until you've seen some statistical effect.
I think the better journals/reviewers can push back at this step, though: If you don't have a demonstrable mechanism, you can't publish. Don't let people stop at "...this gene is important (p < 0.005)." (I know not all things are as simple as gene deletion and complementation)
I think examples would be when identifying necessary and sufficient causes for a given effect.
A trivial case would be figuring out how fire works. You have have heat, fuel, and oxygen. Take any of those away, and there is no fire.
This can be demonstrated by direct experiment. But it’s unfortunately a pretty blunt tool. Because the real (and non-trivial) case is that this isn’t strictly true.
What you need is not specifically oxygen, but rather anything that can function as an oxidizing agent.
Still demonstrable by direct experiment, but more difficult. And how you want to understand fire is directly related to what you want to do with it: create it? Or make it go away?
If you want to make fire, it might be enough to understand the basic three element. You stumble onto the fact that magnesium will burn, and that’s good enough that you don’t look any further.
If you’re interested in putting fires out, it’s also “easy” to show that sumberging a burning object in water puts out the fire by direct experiment. Until you try to put out a magnesium fire like that. Similarly a phone battery that’s burning up will get worse if you dunk it in water.
Then on top of that, there are things that act like fire but are fundamentally different. Nuclear reaction as opposed to chemical ones.
Now, of course, we have a pretty good idea of how all of this works at a molecular or atomic level, but direct experimentation gets us a certain way down the path of investigating what’s really going on, even though it’s pretty imperfect.
But pretend that you have no knowledge of chemistry or physics, and you goal is to understand what causes fire. What you do know is some math and some statistics. How do you design a statistical approach to figuring out what makes fire? Or what makes fire go away?
You know that fire is related to heat because it’s hot. But you don’t know how. We also know that fire is related to wood because that our most frequent experience with fire. We think fire causes heat because everything that’s on fire is hot, and only some things not on fire are hot.
So there’s our null hypothesis: fire causes heat. The obvious alternative hypothesis is that heat causes fire.
And because we’re somewhat primitive in this example, we are really focusing on wood.
So we sample as many different types of burning wood as we can find. We go through the effort of finding samples of 1800 different types of tree on fire. (Doesn’t matter how the fire started, it just happened).
We find in every case that the burning wood was associated with heat of a certain temperature. So we accept the null hypothesis and reject the alternative.
Obviously, there’s a shit-ton of things that are obviously wrong with that statistical approach. And it probably seems comically bad to everyone. But the fact is that once you wade through the jargon specific to a given field, that’s really what you’re dealing with.
Null and alternative hypotheses that are causally confused, accepting rather than failing to reject . . . The whole thing. It happens mostly in fields where you cannot discover causal mechanisms through greater understanding of the platform. Social sciences, Econ, neuroscience, Climatology, etc.
And in the bad case above, note that the p-value would make no difference. The design itself and the misinterpretation is to blame. Not the p-value itself.
Yes, I think p-values and hacking them is a huge problem. But not because people have a habit of creatively interpreting data to get published. It’s because a focus on p-values encourages bad bad experimental designs, and reviewers don’t pay enough attention to that.
> Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test.
Part of the problem, though, is that scientific results are often counterintuitive, or (particularly in the social sciences) are easy to view as obvious after we know what the answer is.
One famous example: the sociologist Paul Lazarsfeld once gave a talk where he asked whether urban or rural men had adjusted more easily to the day-to-day routines of military service in World War II. There are equally obvious reasons why rural men would adjust more easily (e.g. they have more experience with physical labor, have experience with guns) or urban men would (e.g. they're used to crowded conditions, have more experience with a hierarchical division of labor).
I'm not sure that the results should be intuitive. I have a moderate fascination with quantum mechanics and I don't find much of that to really be intuitive. Some of it is pretty hard for me to fully grasp.
>I'm not sure that the results should be intuitive.
This is an appeal to complexity fallacy.
A result may initially be counterintuitive, but it is the the duty of the experimenter to alter their model to fit the new data. To explain the results in terms of known phenomenon and develop a new intuition which includes the unexpected results in the model.
If you are having trouble fitting a result into your current intuition, it is because your intuition is an using outdated or improper model of the phenomenon.
Nobody is born with intuition. It is learned, via bayesian style updates of information to model reality. The more effort we put into quality updates of our intuitive model, the better our intuition of reality becomes.
I personally find NHST suspicious, even if the p-value is less than 0.005. It means that, ASSUMING that the null hypothesis is correct, the probability of observing the data is less than 0.5%. That's still not zero, though. For example, if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens. Conversely, the p-value being high isn't really meaningful either. For example, if I try to decide if a white person is from South Africa, and the null hypothesis is that they are, and since ~9% of the population in South Africa is white, I would fail to reject the null hypothesis, even if the person's actually from Europe or literally any other country in the world. p-values are dependent on the sample size, too, and become more and more sensitive to smaller differences in the ASSUMED population distribution(s), when the effect size can in fact be small. NSHT seems highly dependent on quality null hypothesis that's correct, which you can't really establish since that's what you're trying to find out. So the approach doesn't really conclude anything either way, and just really seems weak as an approach to scientific testing overall.
I don't really follow. Your examples seem like weird or poorly thought out experiments (well, surveys, not experiments), not anything to do with significance testing.
If I wanted to see if native Hawaiians are citizens and didn't have access to law, real data, anything like that, I'd sample from the population of Hawaiians, not citizens as a whole, and then regardless of the null hypothesis being "are citizens" or "aren't citizens" I think the results would be overwhelmingly one-sided enough to work out. And the South Africa example is asking about 1 single person? That not a question of science, then, just a question?
Maybe my example was contrived, but my main argument was that p-values represent P(D|H), or probability seeing your data conditioned on assuming that the hypothesis is correct, and that approach seems inherently flawed to me. If you're doing an experiment, p-values don't say anything about if the hypothesis is true or false, in fact the approach can't theoretically give that information. If you're doing a basic t-test, it goes something like this:
1. You have a null hypothesis you assume to be true
2. Based on this null hypothesis and the central limit theorem, if you theoretically conduct many experiments you expect some summary statistic to have a specific distribution around the 'true' population summary statistic and 'true' population variance, which you don't have so you assume it to be the null hypothesis
3. You compare your experiment's summary statistic to the hypothesized distribution. If the probability of seeing your summary statistic is below some threshold, you say it's unlikely to see this summary statistic again.
But the probability of seeing that summary statistic again actually depends on P(H), which NHST doesn't provide any information on. My examples were meant to highlight the nature of conditional probabilities, rather than how real life experiments are conducted.
Your examples weren't so much contrived as incoherent, to the point where I begin to suspect you don't really understand NHST.
"if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens."
So what is the observation in this case, and what would the corresponding prediction from the null hypothesis be? The observation that "0.2% of the US population is native Hawaiian" has no relation to your claimed null hypothesis at all.
The rest of your objection seems like one of those confused arguments trying to rule out basic reductio ad absurdam ("but if X really isn't true, then your arguments about seeing or not seeing the consequences of X have no basis!").
(And the central limit theorem has nothing to do with null hypothesis testing: you can do NHST with completely non-Gaussian statistics.)
Yes, you can conduct NHST without the central limit theorem. However, it's used very widely in NHST. Was there anything wrong with what I said about what a typical t-test usually looked like? My lab would use that approach to do molecular biology.
You don't seem to understand my argument, so let me rephrase: The example about Native Hawaiians was meant to highlight the nature of conditional probabilities, and p-values are conditional probabilities. Just because p-values are below some threshold doesn't necessarily mean that the null hypothesis is incorrect and therefore should be rejected. Just because p-values values are high doesn't mean that the null hypothesis should fail to be rejected. P-values do not theoretically give that information. It doesn't even represent the probability of observing that value, since it's a conditional probability - as in, the probability of observing that value given that the null hypothesis is true, not the probability of observing that value. If the p-value is below 0.005, can you scientifically, theoretically conclude that the null hypothesis should be rejected? The probability of seeing a Native Hawaiian person given that the person is a US citizen is below the threshold of 0.005, but does that mean the conditioned part (the US citizen thing) should be rejected? Granted, it's hard to relate that example to actual experiments, but my argument is that p-values don't theoretically give any conclusions either way, and trying to make it "scientific" to draw conclusions by introducing thresholds to a conditional probability, no matter how strict, seems inherently flawed. Using it as a single metric among many, to use it as a tool for exploration makes sense to me. Even to make strong suggestions, sure, especially with all the controls RCTs put in. But to make hard conclusions, as in NHST? The approach itself doesn't have the theoretical power to do so.
Pretty good take on this for a general-purpose audience. I was ready to complain about this line for implying P(H|D) (rather than P(D|H_0), as actually occurs):
> The researcher basically asks: How ridiculous would it be to believe the null hypothesis is the true answer, given the results we’re seeing?
but I think they (indirectly) correct it pretty well in the next section.
It isn't published yet, but Gelman and co's recent piece on this seems like an important contribution to the debate [1]. Short version: take the p-value down from it's pedestal, abandon the dichotomous view of is/isn't significant, and consider it with all the other evidence and data.
P-value is such a stupid metrics. Never understood why people still use it and conjure 0.05 magic number as the one that "proves" something. It should be considered a joke, but well, there is economics and a bunch of other "sciences" that are considered "reputable", and people there are daily making living off dubious theories, so nothing will change.
I co-authored a paper into IMC discussing stats and network questions with a statistician. We got bounced for being too polemical and teaching-mode instead of classical science. the whole point of the paper, was to try and argue for a more mature outlook into statistical methods applied to network operations!
My motivation to co-write was strongly influenced by being called out for p-jacking. Its remarkably easy to fall from grace. I did. walking back is very hard, when peer review doesn't want to have a polemical discussion.
> Ideally, Lakens says, the level of statistical significance needed to prove a hypothesis depends on how outlandish the hypothesis is.
> Yes, you’d want a very low p-value in a study that claims mental telepathy is possible. But do you need such stringent criteria for a well-worn idea? The high standards could impede young PhDs with low budgets.
Watching someone try to be Bayesian without using Bayesian techniques is painful.
> There are also new, advanced statistical techniques — like Bayesian analysis — that, in some ways, more directly evaluate a study’s outcome.
Almost as painful as watching Bayesian analysis described as "new."
Either general AI or reducing chat into a theorem and then running a prover - but that's for logic not facts. And liberals would have that banned within hours.
Give them a break. It's a pop science article. Such articles can describe 1.5 concepts at most, so having used that budget on p-values, it's not available for Bayes, so explaining the concept from a frequentist perspective is reasonable.
Also Bayesian analysis (as a mainstream stats technique) is relatively new as it has only become mainstream with the advent of cheap computing power.
When I looked at the authors list I got the impression that it was mostly sociologists, public health researchers etc, but not very many well known statisticians, or people from the traditional physical and biological sciences. It seems like it would have been better to have more authors from the latter groups to make such a broad suggestion; their absence suggests that the proposal doesn't have much support among those fields.
Why not just be more clear and instead of writing the "p-value < 0.05", write: The probability that the difference between these two groups is by chance is less than 5%. Then everybody knows that if you do this a bit more than 20 times this results is achieved by chance alone. Then, just show the damn plots and let me see how much distributions overlap. I have seen insanely small P-values with completely overlapping distributions just because of large amounts of data. I find the P-value near useless.
Personally I much prefer ROC curves, they are much better at showing the difference between distributions. Still, nothing beats the raw data and healthy skepticism.
> The probability that the difference between these two groups is by chance is less than 5%.
That's a common misunderstanding of the p-value, but that's not what it means. (It means: assuming there is no difference, the probability that we'd nevertheless see a difference as big as we are seeing or even bigger is less than 5%).
92 comments
[ 2.8 ms ] story [ 163 ms ] thread> The case against p<.005
> The real problem isn’t with statistical significance; it’s with the culture of science
Right. The substance happens in the last 10% of the article.
...and there isn't any solution or recommendation.
Unfortunately, I think this article is really long way of saying, "There exists p-hacking". I was hoping for more :(
P=0.05 is best interpreted as "There is a 95% chance that this data isn't pure noise".
Then researchers should always put more emphasis on confidence intervals. Readers of papers can then see if the size is enough to make the results relevant and not likely caused by experimental accident. Plus given the perverse incentives researchers are subject to, maybe assume that the real effect size is probably closer to the lower bound of the interval.
Null hypotheses have limited usefulness even at p=0.005. A tiny systematic bias in the experiment can make it cross that threshold and there are _always_ at least small biases. These can be caused by not exactly calibrated instruments or small differences in the way tests are performed by different researchers on the team etc. Inherent in null hypothesis testing is this ridiculous assumption that there are no systematic biases. This is its fatal flaw.
The null hypothesis test never tells you that the effect is zero or that it is any other value, it only gives you an hint about whether or not the data is too noisy to say anything at all.
It's much closer to, this result rises above the noise floor. Suggesting but not implying a possible link between X and Y, fell free to investigate.
No, that's not what P=0.05 means. This misconception is exactly why p values are so problematic. The question we want to answer is unanswerable, so instead we subtly substitute a different question in its place and don't even realize we've swapped concepts.
There is absolutely nothing valid about your "flip" (called transposing the conditional) and the damage done to the human species by people doing the "flip" is too huge for most to comprehend. Sorry, but I really cannot overstate the problems with this "flip".
Perhaps people who get paid by grants should get three chances regarding this flip. If they strike out they are banned from ever being funded by NIH, etc for life. The result (either 99+% of current researchers would be culled, or p-values would become appropriately rare) would do more to advance science than any other idea I have seen.
Sure there are debates about priors. These are unavoidable. The hidden assumption in my example is a 50/50 prior on too noisy, not too noisy hypotheses. IMO a reasonable uniform (maximum entropy) prior to start a discussion from. Of course, one should be more precise when doing this type of analysis in a paper. The reason a good prior is hard to define here is not because you shouldn't do the flip, it is because the question that the t-test is trying to answer doesn't make much sense.
How can saying false things make your science "about the the real world"? The only thing I can think of is that by "real world" you are referring to using the publication of massive amounts of incorrect/questionable information as a metric for success. Yes, that is indeed the sham going on.
>"There is a reason humans naturally and instinctively do it when they talk about experiments. This is how you describe things in real world human language."
I have no problem talking about data without transposing the conditional, if you do that is some issue with your training.
>"Sure there are debates about priors."
The error you advocate (of transposing the conditional) has nothing to do with priors.
Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior.
Ok, but I fail to see the relevance of this to anything.
>"Also P(A|B) = P(B|A) if the prior probabilities are equal, often a reasonable prior."
Here, A = Hypothesis and B = data (or vice versa). You are claiming that the "probability of seeing your data whether or not the hypothesis is correct", is often near the "probability the hypothesis is correct independent of your data"?
Based on what? What principle leads you to think these two probabilities should be near each other? BTW, if you actually have one this is like the holy grail of statistics.
Shouldn't that be "the probability of seeing the data, _given_ that the hypothesis is correct" and "the probability of the hypothesis being correct, _given_ that the data has been observed"?
The claim is that it is reasonable to assume P(A) ~ P(B), so that P(A|B) ~ P(B|A). I am referring to P(A) and P(B) in the equation.
I agree there are debates surrounding priors, especially when you get into higher order, meta or purely hypothetical territory (there is no actual noise generator here) like this, but I think these are unavoidable.
For this to be true, P(observations) ~ P(pure noise generator) as you said earlier. Anyway, you can look at it from either perspective, doesn't really matter.
On what basis do you claim this is reasonable to expect? I suspect you have none whatsoever... in fact simple thought experiments will lead us to the opposite conclusion, and there is plenty of data telling us the same.
This is incorrect. A p-value is the probability of observing a value of the test statistic that is at least as extreme as the reported value, assuming the null hypothesis is true.
Now, the exact values of a given data set are vanishingly unlikely, so we actually ask about the probability we'd say this data or even more extreme data given that we're just looking at noise. That's a typical p value.
The whole terminology with values being "extreme" is just unintuitive and unwieldy, IMO.
Publish negative results.
I'm kinda embarrassed by my negative results, such as they are, but I'm clear in this latest case that I wanted the method reviewed most, since others then may be able to use it.
(Warning: not a real scientist, but being validated by real scientists in the case in question.)
There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in.
Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you.
But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results.
If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable.
Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done.
The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it.
Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers.
You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable.
Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality.
I will add this...
Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available.
So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/ins...
Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing.
Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful.
If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value.
It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems.
The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.
By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.
Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives.
So, underpowered results, whether positive or negative, are less interesting.
The idea is probably only applicable to a small subset of scientific work (not good for exploratory and very novel stuff, for instance), but in certain specific areas it might make publication of negative results a more viable path for scientists because they don't have to fear that their idea was bad or they seem incompetent because of a negative result. (Since their design was accepted, at least some other people thought it was a good idea too.)
Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.
That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.
What holds us back, it seems to me, isn't process, it's culture and inertia.
There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.
It may be true that positive results are more useful than negative ones, but that doesn't mean we should put zero value in negative results. I think that's what parent was trying to point out.
(Warning: In Scientist-Engineer limbo)
Or just don't publish the results at all, only publish the methodology and make the results available elsewhere along with the data and source code.
It's rare, but sometimes people without any degrees get published in reputable journals.
If I were in your shoes, I'd do what I can to get published in something reputable and buy a bunch of printed copies. I'd then autograph them and hand them out like a rock star.
When I was first published in a journal, I brought a copy of the journal and a couple of 'official' single article prints for my mother. One of them she hung on the fridge and even stuck a gold star sticker on it.
Of course, buying a copy of the whole journal and single article reprints was a lot less expensive then. I'd still get physical copies, just to show them off.
Best of luck and feel free to yell, both when you've got it published and if you have any questions about formatting or want a proofreader. If it encourages you to publish, and encourages people like you, I'll do it. I really do wish there were more citizen scientists, or at least people who lack the standard credentials, submitting their ideas, being interested, and being willing to share their insights and research.
Either way, best of luck! You've made my night. I love stories like that.
uninvolved@outlook.com (generic public email address)
I do have my name on a pukka scientific paper from many moons ago, as the summer help for a PhD, where the real authors slung me in the author list out of kindness.
But I do like the idea of a gold star. Then I can be 'award winning'! B^>
That was molecular biology, this topic is home heating efficiency. That one is online for all to see, this one is too but is a bunch of very boring docs and part of a GitHub repo!
A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test.
Obviously, these tests are important, but people focus too much on the p-value rather than designing an experiment which will produce a meaningful result (regardless of whether the result is positive or negative).
Proper experimental design is like a cake, and p-values are the frosting. Everybody wants cake, most often with frosting. Nobody wants frosting without the cake, or at least those that do have questionable taste.
And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter.
In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment.
The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. However the scientific results from that experiment really don't hinge on statistical signifigances. The results are either binary in result, or better yet, contradictory - elucidating hitherto unknown variables (something a purely statistical result won't help with). And if mechanism is shown in multiple systems, with multiple techniques, the results can very quickly become near-definitive.
Yeah, in molecular biology this always seemed to be the most convincing evidence. Deleting a gene and then adding it back. Sometimes going a step further and overexpressing the gene. You can generally publish that data without any statistical analysis.
> The later experiment is often harder, and might not even be proposable until you've seen some statistical effect.
I think the better journals/reviewers can push back at this step, though: If you don't have a demonstrable mechanism, you can't publish. Don't let people stop at "...this gene is important (p < 0.005)." (I know not all things are as simple as gene deletion and complementation)
A trivial case would be figuring out how fire works. You have have heat, fuel, and oxygen. Take any of those away, and there is no fire.
This can be demonstrated by direct experiment. But it’s unfortunately a pretty blunt tool. Because the real (and non-trivial) case is that this isn’t strictly true.
What you need is not specifically oxygen, but rather anything that can function as an oxidizing agent.
Still demonstrable by direct experiment, but more difficult. And how you want to understand fire is directly related to what you want to do with it: create it? Or make it go away?
If you want to make fire, it might be enough to understand the basic three element. You stumble onto the fact that magnesium will burn, and that’s good enough that you don’t look any further.
If you’re interested in putting fires out, it’s also “easy” to show that sumberging a burning object in water puts out the fire by direct experiment. Until you try to put out a magnesium fire like that. Similarly a phone battery that’s burning up will get worse if you dunk it in water.
Then on top of that, there are things that act like fire but are fundamentally different. Nuclear reaction as opposed to chemical ones.
Now, of course, we have a pretty good idea of how all of this works at a molecular or atomic level, but direct experimentation gets us a certain way down the path of investigating what’s really going on, even though it’s pretty imperfect.
But pretend that you have no knowledge of chemistry or physics, and you goal is to understand what causes fire. What you do know is some math and some statistics. How do you design a statistical approach to figuring out what makes fire? Or what makes fire go away?
You know that fire is related to heat because it’s hot. But you don’t know how. We also know that fire is related to wood because that our most frequent experience with fire. We think fire causes heat because everything that’s on fire is hot, and only some things not on fire are hot.
So there’s our null hypothesis: fire causes heat. The obvious alternative hypothesis is that heat causes fire.
And because we’re somewhat primitive in this example, we are really focusing on wood.
So we sample as many different types of burning wood as we can find. We go through the effort of finding samples of 1800 different types of tree on fire. (Doesn’t matter how the fire started, it just happened).
We find in every case that the burning wood was associated with heat of a certain temperature. So we accept the null hypothesis and reject the alternative.
Obviously, there’s a shit-ton of things that are obviously wrong with that statistical approach. And it probably seems comically bad to everyone. But the fact is that once you wade through the jargon specific to a given field, that’s really what you’re dealing with.
Null and alternative hypotheses that are causally confused, accepting rather than failing to reject . . . The whole thing. It happens mostly in fields where you cannot discover causal mechanisms through greater understanding of the platform. Social sciences, Econ, neuroscience, Climatology, etc.
And in the bad case above, note that the p-value would make no difference. The design itself and the misinterpretation is to blame. Not the p-value itself.
Yes, I think p-values and hacking them is a huge problem. But not because people have a habit of creatively interpreting data to get published. It’s because a focus on p-values encourages bad bad experimental designs, and reviewers don’t pay enough attention to that.
In any case, while I can't say I agree (and you gave the reason why), it sure is a more _exiting_ experiment if the result is immediately apparent!
(Gaither's Dictionary of Scientific Quotations gives the attribution "In N.T. Bailey The Mathematical Approach to Biology and Medicine")
I need this tattoo'd on my forehead.
Part of the problem, though, is that scientific results are often counterintuitive, or (particularly in the social sciences) are easy to view as obvious after we know what the answer is.
One famous example: the sociologist Paul Lazarsfeld once gave a talk where he asked whether urban or rural men had adjusted more easily to the day-to-day routines of military service in World War II. There are equally obvious reasons why rural men would adjust more easily (e.g. they have more experience with physical labor, have experience with guns) or urban men would (e.g. they're used to crowded conditions, have more experience with a hierarchical division of labor).
The only reason they are counterintuitive is because the current model of intuition is flawed.
A good experiment creates new intuition by creating a new model by which once counterintuitive results become intuitive.
This is an appeal to complexity fallacy.
A result may initially be counterintuitive, but it is the the duty of the experimenter to alter their model to fit the new data. To explain the results in terms of known phenomenon and develop a new intuition which includes the unexpected results in the model.
If you are having trouble fitting a result into your current intuition, it is because your intuition is an using outdated or improper model of the phenomenon.
Nobody is born with intuition. It is learned, via bayesian style updates of information to model reality. The more effort we put into quality updates of our intuitive model, the better our intuition of reality becomes.
If I wanted to see if native Hawaiians are citizens and didn't have access to law, real data, anything like that, I'd sample from the population of Hawaiians, not citizens as a whole, and then regardless of the null hypothesis being "are citizens" or "aren't citizens" I think the results would be overwhelmingly one-sided enough to work out. And the South Africa example is asking about 1 single person? That not a question of science, then, just a question?
1. You have a null hypothesis you assume to be true
2. Based on this null hypothesis and the central limit theorem, if you theoretically conduct many experiments you expect some summary statistic to have a specific distribution around the 'true' population summary statistic and 'true' population variance, which you don't have so you assume it to be the null hypothesis
3. You compare your experiment's summary statistic to the hypothesized distribution. If the probability of seeing your summary statistic is below some threshold, you say it's unlikely to see this summary statistic again.
But the probability of seeing that summary statistic again actually depends on P(H), which NHST doesn't provide any information on. My examples were meant to highlight the nature of conditional probabilities, rather than how real life experiments are conducted.
"if I try to decide that native Hawaiians are US citizens, and the null hypothesis is that they are, but since only ~0.2% of total US population is native Hawaiian, NHST would conclude that native Hawaiians aren't US citizens."
So what is the observation in this case, and what would the corresponding prediction from the null hypothesis be? The observation that "0.2% of the US population is native Hawaiian" has no relation to your claimed null hypothesis at all.
The rest of your objection seems like one of those confused arguments trying to rule out basic reductio ad absurdam ("but if X really isn't true, then your arguments about seeing or not seeing the consequences of X have no basis!").
(And the central limit theorem has nothing to do with null hypothesis testing: you can do NHST with completely non-Gaussian statistics.)
You don't seem to understand my argument, so let me rephrase: The example about Native Hawaiians was meant to highlight the nature of conditional probabilities, and p-values are conditional probabilities. Just because p-values are below some threshold doesn't necessarily mean that the null hypothesis is incorrect and therefore should be rejected. Just because p-values values are high doesn't mean that the null hypothesis should fail to be rejected. P-values do not theoretically give that information. It doesn't even represent the probability of observing that value, since it's a conditional probability - as in, the probability of observing that value given that the null hypothesis is true, not the probability of observing that value. If the p-value is below 0.005, can you scientifically, theoretically conclude that the null hypothesis should be rejected? The probability of seeing a Native Hawaiian person given that the person is a US citizen is below the threshold of 0.005, but does that mean the conditioned part (the US citizen thing) should be rejected? Granted, it's hard to relate that example to actual experiments, but my argument is that p-values don't theoretically give any conclusions either way, and trying to make it "scientific" to draw conclusions by introducing thresholds to a conditional probability, no matter how strict, seems inherently flawed. Using it as a single metric among many, to use it as a tool for exploration makes sense to me. Even to make strong suggestions, sure, especially with all the controls RCTs put in. But to make hard conclusions, as in NHST? The approach itself doesn't have the theoretical power to do so.
> The researcher basically asks: How ridiculous would it be to believe the null hypothesis is the true answer, given the results we’re seeing?
but I think they (indirectly) correct it pretty well in the next section.
It isn't published yet, but Gelman and co's recent piece on this seems like an important contribution to the debate [1]. Short version: take the p-value down from it's pedestal, abandon the dichotomous view of is/isn't significant, and consider it with all the other evidence and data.
[1]: http://andrewgelman.com/2017/09/26/abandon-statistical-signi...
https://fivethirtyeight.com/features/science-isnt-broken/#pa...
My motivation to co-write was strongly influenced by being called out for p-jacking. Its remarkably easy to fall from grace. I did. walking back is very hard, when peer review doesn't want to have a polemical discussion.
> Yes, you’d want a very low p-value in a study that claims mental telepathy is possible. But do you need such stringent criteria for a well-worn idea? The high standards could impede young PhDs with low budgets.
Watching someone try to be Bayesian without using Bayesian techniques is painful.
> There are also new, advanced statistical techniques — like Bayesian analysis — that, in some ways, more directly evaluate a study’s outcome.
Almost as painful as watching Bayesian analysis described as "new."
Less snarky: not really
Also Bayesian analysis (as a mainstream stats technique) is relatively new as it has only become mainstream with the advent of cheap computing power.
https://psyarxiv.com/9s3y6
Personally I much prefer ROC curves, they are much better at showing the difference between distributions. Still, nothing beats the raw data and healthy skepticism.
That's a common misunderstanding of the p-value, but that's not what it means. (It means: assuming there is no difference, the probability that we'd nevertheless see a difference as big as we are seeing or even bigger is less than 5%).