This _looks_ like a very interesting take on sample sizes, but I'm not qualified to evaluate its technical merit.
However, as someone more on the creative, A/B testing-adjacent side of things, it's always boggled my mind that this isn't a settled issue.
Given the amount of money poured into testing these days, can anyone on the data science side give their 2c on why there are still novel approaches across tools and the industry with regards to samples sizes and significance?
Perhaps quantitative elements don't become hard norms in creative industries? Consider how much ad spend is burned in the first place without hard science to validate it.
That said the lack of norms in some other, more technical industries is surprising as well.
Cynical take: the advertising industry makes a lot of money by being a bit fuzzy about which parts and practices are valid, and which are complete bullshit.
Even if you assume best intentions, A/B testing often comes down to a variation of a t-test, the interpretation of which can be fuzzy too - https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf (TLDR - when it comes to statistical significance you can be either wrong or vague).
Why t-test in particular ? That would be applicable as long as one has two Gaussian populations with the same variance and potentially different means.
In my experience, those preconditions are violated more frequently than they are met, unless some extra steps are taken to make the data Gaussian with equal variance. Central limit theorem helps sometimes if the true variance of the populations are well behaved (not very large).
Not if it has a heavier tail, which is common in practice. It is disastrously brittle in those cases, mainly because sample variance is a lot of garbage in those cases.
Asymmetrical distributions don't fare well, in part because the mean probably isn't the thing you want to be comparing anyway. Mixture distributions or data contaminated with outliers are also trouble.
On the other hand, a bunch of simulation studies have found that t-tests work pretty well for symmetric distributions, even those with moderately heavy tails like a t(6) distribution. Moreover, the power declines a lot faster than the Type I error rate (which, for whatever reason, people seem to worry about more).
I've gotten a few paper reviews where someone has made the "but it's not normal" argument. Redoing the analysis with permutation or randomization tests (or both, for one stubborn reviewer) almost never changes the resulting p-values by more than a few percent. I don't know if this reflects the fact that they're usually quite similar or that we're just fairly judicious in how we analyze data.
The short and the simple (if somewhat controversial) answer is because the real world is complex and probability is a measure of subjective belief.
So without getting TOO philosophical, for cases which don't fall into relatively simple text-book examples (differences between coin tosses in two groups, random draws of cards, radioactive decay, well-defined distributions, that sort of thing...for which generally agreed upon settled best practice analytical solutions probably do exist), or which don't involve making some rather outrageous assumptions (lets assume away our non-survey error), you have in practice disagreements about what you're even experimenting over, what is held constant, what even are the significance of the results, what are the relevant distributions, what is the non-survey error, how to update your beliefs based upon the results, what are reasonable prior beliefs beforehand, real world restrictions on how you can design your experiment, etc.
It would be my contention that there's not a settled best practice because its inherently NOT settle-able. There is no objective answer to the reference class problem. There are multiple consistent hypothesis that can equally fit/explain data. Two reasonable rational minds can disagree over the implications of results. Options have trade-offs and there are differing levels of uncertainty over all things.
Of course, if I'm at work and I'm asked to design a sample, collection, or experiment, that's not what I say. I generally start at analytical solutions, and gradually step down into simulations and monte-carlo methods depending on how complex it gets. And i'll ask them what question they hope to answer, and what are the consequences of being right/wrong, and how much resources they have to expend in finding the answer and what are the restrictions on what they can do and what do they have to work with.
And people who ask me to design samples don't talk in languages of prior or subjective beliefs or refer to reference class problems or relevant distributions for a significance test or updating beliefs etc. Even though that's what I often know they're having problems with.
But this is hacker news, so maybe on here I can say: because the real world is really complex and a lot of it is inherently subjective.
(also, in the real world, a lot of people who ask for the a sample design REALLY don't like the amount of resources required for their 'officially' stated desired outcomes, so like a lot of real world practice and service delivery, best practice often becomes how to optimally deal with champagne tastes on a beer budget serving princes who ask for contradictory goals).
Totally get the "champagne tastes on a beer budget" thing too. Partly that's why I wonder about innovations that promise (or at least suggest) that you don't really need e.g. sample sizes that are as big as you may have thought. If (if!) that's true, that's amazing! That's a saving that everyone in business-driven online optimisation (for example) could benefit from... which also makes it seem unlikely. But as a non-expert it's impossible to sift the smart-but-wrong from the smart-and-correct takes, which makes the debates fascinating but frustrating!
This happens because no two businesses have exactly the same end goals and therefore there is no "one-size-fits-all" testing algorithm.
Some of our clients want to prioritize absolute certainty because they're testing something as fundamental as product positioning, while others want to test different colors and want a result better than control (irrespective of how much better it is).
Ultimately, you can think most A/B testing algorithms on a continuum of certainty and speed. You can't have both.
15 comments
[ 1.9 ms ] story [ 48.3 ms ] threadHowever, as someone more on the creative, A/B testing-adjacent side of things, it's always boggled my mind that this isn't a settled issue.
Given the amount of money poured into testing these days, can anyone on the data science side give their 2c on why there are still novel approaches across tools and the industry with regards to samples sizes and significance?
How come there isn't a settled best practice?
Perhaps quantitative elements don't become hard norms in creative industries? Consider how much ad spend is burned in the first place without hard science to validate it.
That said the lack of norms in some other, more technical industries is surprising as well.
In my experience, those preconditions are violated more frequently than they are met, unless some extra steps are taken to make the data Gaussian with equal variance. Central limit theorem helps sometimes if the true variance of the populations are well behaved (not very large).
Asymmetrical distributions don't fare well, in part because the mean probably isn't the thing you want to be comparing anyway. Mixture distributions or data contaminated with outliers are also trouble.
On the other hand, a bunch of simulation studies have found that t-tests work pretty well for symmetric distributions, even those with moderately heavy tails like a t(6) distribution. Moreover, the power declines a lot faster than the Type I error rate (which, for whatever reason, people seem to worry about more).
I've gotten a few paper reviews where someone has made the "but it's not normal" argument. Redoing the analysis with permutation or randomization tests (or both, for one stubborn reviewer) almost never changes the resulting p-values by more than a few percent. I don't know if this reflects the fact that they're usually quite similar or that we're just fairly judicious in how we analyze data.
For cases where one needs too test "is this bigger than that ?" I test for stochastic dominance rather than equality of mean.
So without getting TOO philosophical, for cases which don't fall into relatively simple text-book examples (differences between coin tosses in two groups, random draws of cards, radioactive decay, well-defined distributions, that sort of thing...for which generally agreed upon settled best practice analytical solutions probably do exist), or which don't involve making some rather outrageous assumptions (lets assume away our non-survey error), you have in practice disagreements about what you're even experimenting over, what is held constant, what even are the significance of the results, what are the relevant distributions, what is the non-survey error, how to update your beliefs based upon the results, what are reasonable prior beliefs beforehand, real world restrictions on how you can design your experiment, etc.
It would be my contention that there's not a settled best practice because its inherently NOT settle-able. There is no objective answer to the reference class problem. There are multiple consistent hypothesis that can equally fit/explain data. Two reasonable rational minds can disagree over the implications of results. Options have trade-offs and there are differing levels of uncertainty over all things.
Of course, if I'm at work and I'm asked to design a sample, collection, or experiment, that's not what I say. I generally start at analytical solutions, and gradually step down into simulations and monte-carlo methods depending on how complex it gets. And i'll ask them what question they hope to answer, and what are the consequences of being right/wrong, and how much resources they have to expend in finding the answer and what are the restrictions on what they can do and what do they have to work with.
And people who ask me to design samples don't talk in languages of prior or subjective beliefs or refer to reference class problems or relevant distributions for a significance test or updating beliefs etc. Even though that's what I often know they're having problems with.
But this is hacker news, so maybe on here I can say: because the real world is really complex and a lot of it is inherently subjective.
(also, in the real world, a lot of people who ask for the a sample design REALLY don't like the amount of resources required for their 'officially' stated desired outcomes, so like a lot of real world practice and service delivery, best practice often becomes how to optimally deal with champagne tastes on a beer budget serving princes who ask for contradictory goals).
Totally get the "champagne tastes on a beer budget" thing too. Partly that's why I wonder about innovations that promise (or at least suggest) that you don't really need e.g. sample sizes that are as big as you may have thought. If (if!) that's true, that's amazing! That's a saving that everyone in business-driven online optimisation (for example) could benefit from... which also makes it seem unlikely. But as a non-expert it's impossible to sift the smart-but-wrong from the smart-and-correct takes, which makes the debates fascinating but frustrating!
This happens because no two businesses have exactly the same end goals and therefore there is no "one-size-fits-all" testing algorithm.
Some of our clients want to prioritize absolute certainty because they're testing something as fundamental as product positioning, while others want to test different colors and want a result better than control (irrespective of how much better it is).
Ultimately, you can think most A/B testing algorithms on a continuum of certainty and speed. You can't have both.
More here: https://vwo.com/blog/multi-armed-bandit-algorithm/