HN Explorer

Ask HN: A/B testing with only 1-2 sign-ups per month

2 points by timmorgan ↗ HN
I would like to employ A/B testing for my web service homepage and sign-up. My site only gets 1-2 sign-ups per month.

How long would I need to run the experiment to get a reasonably decent idea of which version is statistically better? (I have no statistical background/training.)

The wikipedia page (http://en.wikipedia.org/wiki/A/B_Testing) is very light on details.

Any help would be greatly appreciated.

4 comments

[ 4.1 ms ] story [ 17.0 ms ] thread
Alex3917
Based on your current traffic? About 55 years.
timmorgan
So you're telling me there's a chance... :-)
nate
Here's a great post on doing the math for this stuff

http://blog.asmartbear.com/easy-statistics-for-adwords-ab-te...

But yes, the bad news is it's going to take too long at your rate right now. I recommend looking at things like http://conceptfeedback.com to get feedback in lieu of being able to test right now. You could also consider running a poll with Mechanical Turk workers. The turkers would answer a poll like which design is better for about a $0.05 a person.

I've done the conceptfeedback.com for testing some new elements in http://tgethr.com before we had the traffic to start experimenting and the feedback was great.