HN Explorer

16 comments

[ 3.5 ms ] story [ 44.4 ms ] thread
Godhammer
The devil's in the details, but the angels are there, too.

Amazing how much impact these "minor" changes can have on the bottom line.

Thanks for the reminder to split test all things great and small.

Datasta
It's funny how that works. I suspect social proof was a distraction on that optin form.
swalberg
I wish I knew more about statistics. Punching just the two variants that made it to the end into a Chi square calculator gave me a 90% confidence, not the 95% he cited.

I also wonder about stopping the variation early, and even the conditions under which he stopped the test. http://www.evanmiller.org/how-not-to-run-an-ab-test.html talks about pitfalls when "testing until you reach confidence".

Are there any better references for these types of questions?

Datasta
I cited 95% confidence, which was calculated by Visual Website Optimizer. I wonder how they calculate their confidence level.
bermanoid
Never used Visual Website Optimizer, but chances are, whatever they do leaves you vulnerable to the flaw mentioned in the article that swalberg linked.

Basically, every time you peek at the results of a split test to see if the confidence level is high enough, you make your split test more prone to error.

There are ways to adjust for this which allow you to continuously monitor the test and stop if it hits a threshold, but it's much easier just to set a known number of samples to take and accept the confidence level that comes out then.

Datasta
Actually, I had planned to test 2,000 views from the outset. It went a little over because I didn't stop it fast enough. I do agree that you are vulnerable to influencing results if you stop the test as soon as you hit your confidence level
pilom
Not only is it a distraction. I see that and think "You are going to send me email that will be sent to 15,000 other people? That would just be spam for me! Why should I sign up if you are going to spam me?"
alain94040
Not sure if it's significant yet. What he really got was: 9 people click on the default form, 5 people on variation one, and 14 people on variation two. These are very low numbers no matter what way you look at them.
Datasta
Yes, those numbers are "lowish" but remember, it was only on 2,068 uniques. We plan on running the test again soon with another 2,000 uniques to see if timing had anything to do with it.
wayneyeager
I'm wondering if merely displaying a number is really "social proof" in the Cialdinian sense.

Possibly at one time, but now I think the "proof" part of social proof must be demonstrably not BS for it to work. In the same way that testimonials by "J.S., Ohio" have no real meaning anymore, but "Jared Sanderson, CEO of Whatever.com" would probably be taken seriously.

An interesting experiment would be to get permission from some of your better-known users to say something like "Join Bill Gates, Dalai Lama, and 14,152 others...". Possibly even better to use little avatars or pics of a bunch of people, ala Facebook Like or the old BlogLog widget. Who knows? I feel like that'd be a better thumbs up/down on social proof than just the number.

michael_dorfman
I found it very amusing that after the long post demonstrating that the "social proof" line was cutting his sign-ups in half, there was a box asking me to sign up which included the social proof.
Datasta
That's because I haven't split-tested that form yet. I'm currently preparing to run that one right now.
WillyF
It looks to me as though they should try the original headline without the social proof. Since the Control beat Variation 1, but Variation 2 beat both, that would seem to be a logical next step.
Datasta
I agree. That's on my agenda :-D
slimier
Agreed. I feel the majority of the science behind this is simply keep things clean and minimal. I would hypothesise 'Get Email Updates' would do better than 'Get Free Email Updates', too.

I don't believe a statistic offers any significant social proof. Lots of faceless people i don't know have signed up. So what?