I have (almost exclusively in the finance space). I found it rejecting too many null hypotheses to work in practice (e.g. choosing a distribution that you could generate reasonable simulations with).
For choosing distributions, we ended up plotting observations versus various simulated realities and going with the "eye test". Now, of course, there will be those out that that say this is not rigorous enough, but for most phenomena in the social sciences its very hard to find a distribution that fits reality per the K-S test. So you're left with either bootstrapping (which has its own flaws) a distribution or choosing one that's good enough (e.g. passing the eye test).
I know of one book: Motulsky's Intuitive Biostatistics, very little formulas, much focus on upsides and downsides of various stats approaches, opinions etc.
That the K-S test works is some cute math by the father of modern probability A. N. Kolmogorov.
But that test and many more are part of non-parametric, that is, distribution-free hypothesis testing. That statistics has long been popular in the social sciences. A major theme in such statistics is permutations. Another major theme, and more recent, is resampling.
I first learned about such tests from a book that was sitting around the office, Sidney Siegel,
Nonparametric Statistics for
the Behavioral Sciences.
These tests are all one dimensional. Once I published a paper on a distribution-free test that is multi-dimensional -- my paper may remain the only such.
These days, see also the work of B. Efron and P. Diaconis.
This is a very nice review, but in practice I've found the K-S test to be much less useful than it initially appears:
1. Failing to reject the null hypothesis is not the same as accepting the null hypothesis. That is, concluding "these data are from some distribution X" is spurious.
2. There's a 'sweet-spot' for the amount of data. If you have too few samples, it's very easy to fail to reject; and if you have too many, it's very easy to reject (the chart at the bottom of the "Two Sample Test" section illustrates this).
3. The question "are these data from some distribution X?" is usually too strong. It's usually more informative to ask "can these data be modelled with some distribution X?"
Agree with you on all three, but specifically for 1., can you think of pathological pairs of distinct distribution that the test would often fail to reject?
The article says it's poor at detecting differences in the tails and much better at differences in the medians. So that's where I'd start to find problems.
Playing with the tails make all kind of mistakes possible, but that seems like a criticism that would apply to any attempt to identify a distribution based on sample.
if you are specifically testing against the normal dist the jarque bera test might be better, although also rather sensitive (prone to false negatives).
for two samples if you have enough data to bin, the chisq test is also available to you.
Nice review! I tried using the K-S test once for some of the old Matasano crypto challenges, to determine if the letter frequency of the text after running it through some deciphering algorithm was from the same distribution as the letter frequency of a sample of the English language. Couldn't ever get it to work, though... maybe that's an inappropriate use of the test, or maybe my sample (Pride and Prejudice, IIRC) was unrepresentative. In the end, simply computing the distance of the two letter frequency vectors (sum of squares) worked.
I did it all in ruby at the time, but it looks like rust may have some stats libraries now? Should give it a whirl again this time using rust, or maybe do tpatcek's new Stockfighter game instead.
The K-S test assumes the data comes from a continuous distribution, so count data would mess up the test's false positive rate. You could avoid that by doing a permutation test, or one of the variations of the K-S test designed to account for data with ties.
I've used the KS test (and several other nonparametric stats) in research a bit. It's a cool thing to learn about, though it's worth noting that a lot of applied research can make do with parametric approaches. Thank you, Central Limit Theorem.
22 comments
[ 4.3 ms ] story [ 36.4 ms ] threadI think it comes from 'shots fired in practice' vs. 'shots fired in anger' (i.e. in combat, for the purpose of killing).
For choosing distributions, we ended up plotting observations versus various simulated realities and going with the "eye test". Now, of course, there will be those out that that say this is not rigorous enough, but for most phenomena in the social sciences its very hard to find a distribution that fits reality per the K-S test. So you're left with either bootstrapping (which has its own flaws) a distribution or choosing one that's good enough (e.g. passing the eye test).
But that test and many more are part of non-parametric, that is, distribution-free hypothesis testing. That statistics has long been popular in the social sciences. A major theme in such statistics is permutations. Another major theme, and more recent, is resampling.
I first learned about such tests from a book that was sitting around the office, Sidney Siegel, Nonparametric Statistics for the Behavioral Sciences.
These tests are all one dimensional. Once I published a paper on a distribution-free test that is multi-dimensional -- my paper may remain the only such.
These days, see also the work of B. Efron and P. Diaconis.
More can be done.
HN discussion: https://news.ycombinator.com/item?id=10244950
1. Failing to reject the null hypothesis is not the same as accepting the null hypothesis. That is, concluding "these data are from some distribution X" is spurious.
2. There's a 'sweet-spot' for the amount of data. If you have too few samples, it's very easy to fail to reject; and if you have too many, it's very easy to reject (the chart at the bottom of the "Two Sample Test" section illustrates this).
3. The question "are these data from some distribution X?" is usually too strong. It's usually more informative to ask "can these data be modelled with some distribution X?"
What do you mean by this? You can choose any significance level you like.
if you are specifically testing against the normal dist the jarque bera test might be better, although also rather sensitive (prone to false negatives).
for two samples if you have enough data to bin, the chisq test is also available to you.
I did it all in ruby at the time, but it looks like rust may have some stats libraries now? Should give it a whirl again this time using rust, or maybe do tpatcek's new Stockfighter game instead.