For anyone wondering about the title, it's about p-hacking - picking and choosing variables to suit your objectives. Highly recommend trying the mini-game, it's fun!
They say a photo is worth a thousand words, and here, this demo is as convincing as a lot of the spilt ink regarding the replication crisis and social psychology. As someone who pursuing a PhD in the social sciences... This could be a useful tool for seniors majoring in the social sciences / who are taking stats classes.
Sigh. P hacking has long been a meme. Nicholas Cage and shark attacks for example. P values are useful though if the results seem at least plausible or can be explained by some underlying process. Like a low p value between calorie restriction and weight loss.
Shark attacks isn't really p-hacking though, that's about misinterpreting/misrepresenting a real correlation. P-hacking is finding a fake correlation due to chance because you tested a lot of different things in a smallish dataset, and then reporting that correlation as meaningful without mentioning your negative results or properly explaining your entire process.
This is what I don’t fully grasp about p hacking myself. Say you test variable abc and so on all the way to z. Variable z were found to be significant. Now imagine you had the same data set and just happened to test Z and stopped there, finding it significant.
According to the p hacking concept the former would be potentially unethical while the latter would be great science, however nothing about the underlying data or its association between variables has changed.
I know the exercise was to p-hack, but instead I decided to one-shot my attempt at the most reasonable model from first principals:
- given that we are looking at a national scale, use only national politicians
- use the components from Macroeconomics 101: exclude inflation as that’s on the Fed, exclude stocks as too conflated with FX and international investing alternatives
- don’t needlessly withhold data
Tried one hypothesis, so p-value of 0.04 is accurate. Still OK to explore if you Bonferroni correct the p-Val afterwards
An excellent demonstration of how easy it is to manipulate data to find significant results - it brings to light (again) the pervasive issue of data dredging in research, where researchers, intentionally or unintentionally, keep testing hypotheses until they find something publishable, thereby undermining the integrity of scientific findings and highlighting the importance of preregistration and replication studies.
How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?
Current attempts showing promise include initiatives like the Center for Open Science and journals such as eLife and PLOS ONE that emphasize methodological rigor over impact factor. Additionally, policies from organizations like the National Institutes of Health (NIH) mandating data sharing plans are gradually moving the needle toward these goals.
When you say that eLife is "very rigorous," could you elaborate on what specific aspects of their review process or criteria you find particularly stringent, and how those compare to the review processes of Nature/Cell?
eLife publishes reviews very prominently within articles, and they ask all reviewers to discuss their assessment after they have evaluated the article. I think this favors deep reviews.
In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion.
I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.
11 comments
[ 3.6 ms ] story [ 21.9 ms ] threadAccording to the p hacking concept the former would be potentially unethical while the latter would be great science, however nothing about the underlying data or its association between variables has changed.
- given that we are looking at a national scale, use only national politicians
- use the components from Macroeconomics 101: exclude inflation as that’s on the Fed, exclude stocks as too conflated with FX and international investing alternatives
- don’t needlessly withhold data
Tried one hypothesis, so p-value of 0.04 is accurate. Still OK to explore if you Bonferroni correct the p-Val afterwards
How can we shift the academic incentives away from "publish or perish" toward promoting transparency and rigorous methodology in research? Are any of the current attempts moving the needle?
I've found their reviews much more detailed and technical than Nature Medicine or Cell.
In Nature or Cell, I have sometimes seen 2-liners sent to authors as a review, or reviews where the reviewer clearly read the article for just 10 minutes before rushing an opinion.
I wish eLife doubled down on this and disclosed the reviewer names after e.g. a 2-year embargo. This would make reviews even better IMHO, more skin in the game.