Just to be clear: I'm not necessarily suggesting that full multivariate A/B testing is the best thing to shoot for, I realize very well that especially in site design, different splits cannot be composed together.
However, related to a discussion this past week where it became clear that periodically checking confidence intervals until they "hit" was a terrible way to test because of the high likelihood of picking the wrong result, I am suggesting that the two-armed bandit problem should be considered mandatory knowledge for anyone working on A/B testing frameworks. That problem is literally what A/B testing tries to solve (try to make choices between two options to maximize payoff when you don't know the payoffs of either choice beforehand, taking into account the fact that trials spent building up certainty by testing the payoffs are sometimes wasted on the least valuable option).
I'd have linked to the two-armed bandit problem if there was a page on that, but it just redirects...
1 comment
[ 3.0 ms ] story [ 14.4 ms ] threadHowever, related to a discussion this past week where it became clear that periodically checking confidence intervals until they "hit" was a terrible way to test because of the high likelihood of picking the wrong result, I am suggesting that the two-armed bandit problem should be considered mandatory knowledge for anyone working on A/B testing frameworks. That problem is literally what A/B testing tries to solve (try to make choices between two options to maximize payoff when you don't know the payoffs of either choice beforehand, taking into account the fact that trials spent building up certainty by testing the payoffs are sometimes wasted on the least valuable option).
I'd have linked to the two-armed bandit problem if there was a page on that, but it just redirects...