The dangers of fishing for p values (to focus on those paragraphs) are covered pretty extensively in undergraduate courses in, at the very least, economics and political science. Something happens between that point, and the point at which a researcher is trying to publish in a journal. I think it is fairly obvious that the problem is related to competition for funding and advancement. Of course this is probably a much more difficult problem to solve.
There's another potential source of the problem: the time gap in between those undergraduate lessons are learned in the abstract and the later point in time when the exercise if those lessons is required. Unexercised knowledge atrophies like any other muscle.
Perhaps rather than placing so much emphasis on memorization recall exams, undergraduate evaluation should be based on exercising the research and peer review process. The best way to learn research is to do research.
That's what we did in undergraduate. Practically every class I had required a research paper. We didn't always have to use statistics, but we were graded on methodology. Recall exams were rare, most exams involved interpretation and explanation. Of course I studied political science, things may be different in other fields.
As a statistican doing machine learning I think there is much more to it; You can do statistics with models. There is progress in statistics, like in all other sciences.
A catchy title, not explaining well what is really going on.
I always found the initial assumption that things are normally distributed quite odd.
For example if you work with latencies then the normal distribution is a terrible thing to use. The closer to zero you get the more skewed it has to be, simply because everything that can influence the result will mostly change it in one direction.
So in that particular case an Erlang distribution might be more applicable. And even that often falls short if you have periodic things (background tasks) occasionally messing with your latency, creating a multi-modal distribution.
The assumption is reasonable for a lot of cases because of the central limit theorem. In particular, it applies if the variable of interest is the sum of a large number of independent random variables. For example, the number of heads in N coin flips is the sum of the number of heads in each of N independent experiments each consisting of a single flip.
Of course, there are also a lot of important cases where that N isn't quite large enough to justify the normal distribution, or where the quantity of interest is not a sum of a large number of small effects, so the CLT doesn't apply at all.
Modern machine learning replaced it. Or rather, relegated statistics to a small subset of a wider field, only to be used for increasingly smaller subset of tasks.
Odd editorial. It goes from a not-terribly-good explanation of some criticisms of null-hypothesis significance-testing to an advertorial for his lab's particular variant on ensembles. Or am I missing something important here?
first thing we need is increased literacy in statistics.
if people actually understood they need to question the validity of statistics presented to them before making conclusions and based on those possibly faulty statistics, we'd be much better off.
Step 1: Establish disciplines that ridicule radical criticism.
Step 2: Hire undergraduate math/stats students who took criticism courses at gun point and hated them.
Step 3: ...
Step 4: "OMG Ponies and revolution"
Sir, you do not need a revolution, because it has already occured. You simply need employees who have read critical approaches to statistics AS AN INTEGRAL PART of their stats education. Those people are usually in
- feminist studies (did the requested stats revolution)
- gender studies (ibid, different name)
- women's studies (ibid, different name)
- lgbt studies (exploded feminist studies ftw)
- african american studies (ibid)
- critical studies (sociology, know of mainstream criticism only)
- american studies (good with convoluted language)
The scenario presented by the author seems more explicit and nefarious than the typical "p-hunting" that I've seen in academia. Though I've seen the situation he presents, successively trying different models or parameters until you find the expected effect, it's pretty obviously unethical.
What I've seen much, much more of goes something like this. Say your lab does research that requires a very large quantity of data that's very expensive to acquire. In academia, labor is relatively cheap and funding can be difficult to come by. So a dataset that consumed potentially 100's of thousands of funding dollars represents a significant resource investment.
So what do you do when the study is done? You try to figure out every way possible to reuse that data. And the most common way of doing that is running it using previous models looking for effects...enough people keep looking long enough and, voila!, you find a significance.
In the end, it's the same bias. But the motivation is different and when I've pointed out this is "p-hunting" most just give a blank stare.
14 comments
[ 2.7 ms ] story [ 40.0 ms ] threadI appreciate Prof. Van Der Laan's call for "quality" improvements, but as you point out, the incentives for "glory" trump "quality."
Perhaps rather than placing so much emphasis on memorization recall exams, undergraduate evaluation should be based on exercising the research and peer review process. The best way to learn research is to do research.
A catchy title, not explaining well what is really going on.
For example if you work with latencies then the normal distribution is a terrible thing to use. The closer to zero you get the more skewed it has to be, simply because everything that can influence the result will mostly change it in one direction.
So in that particular case an Erlang distribution might be more applicable. And even that often falls short if you have periodic things (background tasks) occasionally messing with your latency, creating a multi-modal distribution.
Of course, there are also a lot of important cases where that N isn't quite large enough to justify the normal distribution, or where the quantity of interest is not a sum of a large number of small effects, so the CLT doesn't apply at all.
if people actually understood they need to question the validity of statistics presented to them before making conclusions and based on those possibly faulty statistics, we'd be much better off.
Step 2: Hire undergraduate math/stats students who took criticism courses at gun point and hated them.
Step 3: ...
Step 4: "OMG Ponies and revolution"
Sir, you do not need a revolution, because it has already occured. You simply need employees who have read critical approaches to statistics AS AN INTEGRAL PART of their stats education. Those people are usually in
- feminist studies (did the requested stats revolution)
- gender studies (ibid, different name)
- women's studies (ibid, different name)
- lgbt studies (exploded feminist studies ftw)
- african american studies (ibid)
- critical studies (sociology, know of mainstream criticism only)
- american studies (good with convoluted language)
What I've seen much, much more of goes something like this. Say your lab does research that requires a very large quantity of data that's very expensive to acquire. In academia, labor is relatively cheap and funding can be difficult to come by. So a dataset that consumed potentially 100's of thousands of funding dollars represents a significant resource investment.
So what do you do when the study is done? You try to figure out every way possible to reuse that data. And the most common way of doing that is running it using previous models looking for effects...enough people keep looking long enough and, voila!, you find a significance.
In the end, it's the same bias. But the motivation is different and when I've pointed out this is "p-hunting" most just give a blank stare.