HN Explorer

Ask HN: High traffic sites use A/B testing. What can low traffic sites use?

6 points by briggers ↗ HN
Stated another way: why wait for your first few thousand visitors before making your first data-driven decision?

Is this a problem that needs solving?

[Edit: formatting]

7 comments

[ 2.8 ms ] story [ 34.8 ms ] thread
carbocation
You need data to make a data-driven decision. Can you elaborate on how you propose to solve the problem of "making a data-driven decision without sufficient data"? I'm pretty skeptical, as you can see.
briggers
In site usability, ecommerce, and conversion in general speed is well established as being very important. Google and others have made speed a core goal because their data shows that speed helps conversion.

However, I would suggest that site speed is merely a proxy for usability speed. ie. it's more important for conversion to have a form that is easily comprehended and filled out than it is to have a fast loading page.

What I'm working up to is the idea that usability speed might be a useful proxy for a high-conversion design. It's also data that can be measured accurately without bias, and can be generated in lieu of real visitors by throwing a horde of disinterested people from Mechanical Turk at (eg.) your sign-up page with a well-defined goal.

coopertmynas
You can't make a data based decision without data.

What you can do is test on an individual basis. Companies have been doing this long before computers came along. Look up videos by Steve Krug on how to properly run user testing.

There are a lot of tools geared toward this already. Some offer testers, others just give you the capability to test with your own anonymous users. My rule of thumb is if someone else is making money off it, there is still money to be made.

This kind of testing doesn't identify things like a red button converting better than a green button, but it can tell you if some users aren't seeing the button at all.

briggers
Right. Unless I've misunderstood you, this is the angle I am investigating with www.uithing.com

Thanks for the tip re: Steve Krug's videos.

Travis
In my most recent experience, I had so many different tasks to do that were more important than optimization. By the time I had the bare essential stuff done, I had enough data to start to make decisions on.
_delirium
It would be possible to at least improve the situation. An approach common in machine learning is, before a lot of directly-on-point data comes in, to bootstrap predictions using a background dataset of existing data, which can at least give you hints about the structure of the space, and where in it you might fall.

In this case, if you (or some service) had a large data set of user interactions collected from many other websites, then even though the first N data points on your site might not be enough to make strong conclusions on their own, they could be used to match your website up to scenarios in the background dataset that are most similar. There are a lot of different approaches, ranging from Bayesian ones where the background dataset is used as a prior in one way or another, to more clustering-based ones, where your small initial dataset is used to match you to a set of "similar, as far as user-experience lessons go" websites in the existing dataset.

pkamb
Write down the top 5 tasks people will be doing on your site.

Get 3 people, tell them to attempt to complete those tasks.

Watch.