tl;dr former Slack director of product outlines twelve principles for a/b type experiments with features, products, whatever. (ex Good experiments involve analyzing the results as opposed to 'ship/kill?') Each principle has a couple of sentences of elaboration and calls out the opposite style to avoid.
This is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.
This bit is crucial for running a good experiment:
> Good experiments use tight exposure groups
Not all your users will have the conditions necessary for the experimental treatment.
You should only look at changes in behavior where the treatment condition was true. Otherwise your experimental effect is diluted.
But your comparison between experiment and control groups must be neutral on the condition, that is, users in the control group would have seen the treatment if they were in the experiment group (counterfactual).
4 comments
[ 3.8 ms ] story [ 27.0 ms ] threadThis is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.
Kohavi's book will probably provide much more value than this kind of abstract post. See https://experimentguide.com/ for more details.
> Good experiments use tight exposure groups
Not all your users will have the conditions necessary for the experimental treatment.
You should only look at changes in behavior where the treatment condition was true. Otherwise your experimental effect is diluted.
But your comparison between experiment and control groups must be neutral on the condition, that is, users in the control group would have seen the treatment if they were in the experiment group (counterfactual).