HN Explorer

Poor Man's Template A/B Testing (in Django) (tech.matchfwd.com)

11 points by amcgregor ↗ HN
Ever wanted to A/B test your templates in Django and didn't know where to start? Here is a 50 line function as a drop-in replacement for render_to_response with full breakdown of how and why.

2 comments

[ 5.2 ms ] story [ 18.0 ms ] thread
DrDub
Everytime I read about A/B testing I remember this posting about why you do better than A/B using solutions to the multi-armed bandit problem: http://stevehanov.ca/blog/index.php?id=132
amcgregor
Convienently, the solution I wrote allows you to utilize two or as many templates as you want, add and remove to the directory as you wish. For us, setting off a handful of template options for two weeks then seeing which one has higher conversion is extremely simple (no need for 'reward' function as such) and effective (it's a fair distribution across the options). We explicitly wanted to avoid needing to track the number of times each possibility is viewed.

I think "better" needs to be defined. "More accurate results?" Possibly, but probably not by much. Less disturbance to general conversion rates (since the 'successful' case is presented most often barring random variability), sure. But the point is to try out every option, not try out the seemingly more successful option the majority of the time.