Ask HN: Should I believe the results of our A/B testing?
So lets say we had 100 users reading one of our mentions on a blog that click through to our site to give the app a try...Does the download button or headline of the landing page (which we are A/B testing) really matter that much at this point? These users had a pretty clear goal of trying our app. The conversion rates from these visitors were around 75-85% across about 10 blogs (about 4,000 visitors in total). The app has no registration or credit card required barriers. The only requirement is that the user needs to have Firefox (which all of the blogs clearly mentioned before sending the users).
So should I be making judgment calls about the most effective headline or the color of the download button? Or should I just throw away this data and look at other traffic sources before making the decision of which headline to set as the next control?
1 comment
[ 3.9 ms ] story [ 9.3 ms ] threadIt sounds like the question you mean to be asking is more like: Do I have bigger fish to fry than playing with the color of the buttons on the site?