Ask HN: Tools to evaluate scientific claims?
What statistical knowledge and training do I need to evaluate the legitimacy of scientific papers, esp. in the medical field?
Whenever I read about any scientific claims, I ignore the press and go straight to the original paper cited (if there is one, often it is misquoted). I then read the abstract and the testing methodology. If I can spot any issues in the methodology I usually stop reading, e.g. small sample size, obvious confounding variable, blatant causation/correlation errors. But if all seems well, that still doesn't tell you if the study's claims match the test results or if the threshold parameters make sense.
Given a basic stats background, how can I obtain a deeper, intuitive understanding of things like p-values (which seem to be outdated anyway) and other sample sizes. Thanks, HN.
9 comments
[ 4.2 ms ] story [ 26.6 ms ] threadIn the end, peer reviewers are only humans too, make mistakes, have a lot of other work to do, and don't even get paid for their review work. Recently we had an extensive discussion on HN about problems facing science as an institution (based on this article: http://www.vox.com/2016/7/14/12016710/science-challeges-rese...). Peer review was the fourth point on their list of problems.
It's always worth getting results out there even if they don't/can't live up to statistical certainty one might hope for. Every paper doesn't have to stand on its own and come to some undeniable ground breaking conclusion. Over time you'll hopefully collect enough published data to be able to do some more statistically powerful and useful meta-studies down the road.
None of which means there isn't a lot of statistics crap floating around, though.
Further down, in the Results & Conclusions, "association" implies "causal" when they mention "via an intrauterine mechanism?"
[^1]: https://news.ycombinator.com/item?id=12293675
[^2]: @fifteenforty comment, https://news.ycombinator.com/item?id=12295107
Link to original study: http://archpedi.jamanetwork.com/article.aspx?articleid=25432...
LA Times coverage: http://www.latimes.com/science/sciencenow/la-sci-sn-acetamin...
About p-values: they aren't exactly outdated, but they are the subject of a pretty fierce controversy. Many, if not most scientists still use them when doing statistical analyses because they are simple to apply and to understand and provide a quick metric for measuring the significance of a study's results.
Other scientists say that they are too simple and don't convey important information about the actual data (such as the spread). Apparently, there are more modern statistical procedures that do a better job than p-values do. (I'm not a statistician though, so don't ask me what these procedures are...) Also, p-values are all too often subjected to "p-hacking" - massaging the data until you get a statistically significant result (p <= 0.05). In fact, the very concept of significance is problematic. Originally, the p <= 0.05/0.01/0.005 significance limits were just approximate guidelines to help scientists interpret their data. Nowadays, they are often treated as definite boundaries of "truth". ("If my data gives a p-value of 0.049, the result is significant therefore my hypothesis must be true. If p=0.051, it is not significant, therefore my hypothesis must be wrong - or I must tweak my data until I get p=0.05.") This is obviously nonsense, yet a surprisingly common attitude (though not always as extreme as in my example).
As far as I personally am concerned, the real problem is not the actual p-values as such, but perhaps a lack of understanding of statistics by many scientists. (Coupled with the pressure exerted by journals that only want to publish "significant" results and so indirectly encourage p-hacking.)
I got my own statistics knowledge from a lecture series, a book on R, an ecology textbook and various articles...
Check your local university library if you can or google around. I'm sure you'll find something.