[I'm assuming the author posted this. MidsizeBlowfish has no profile information.]
Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives?
BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)
It would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.
6 comments
[ 2.7 ms ] story [ 18.7 ms ] threadGood stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives?
My field is bandit algorithms, which are a generalisation of hypothesis tests. Here there are algorithms like Empirical Bernstein Stopping (http://icml2008.cs.helsinki.fi/papers/523.pdf) which achieve the same early stopping from a different approach. If this is of interest sign up to the course I'm running (http://noelwelsh.com/data/2013/08/16/free-bandit-course-draf...) -- I plan to cover this algorithm soon.
BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)
The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.