How we split test our job listing titles with Optimizely (visual.ly)

Hilariously cool. Hopefully this report is the nail in the coffin of the "ninja" job posting fad.

Buttttt, since you're measuring clicks and not job candidates hired (the actual important metric) is there really a relevant takeaway from the data?

Hard to quantify but if this is proper conversion rate testing you've got to make sure you're optimizing around the proper metric.

(Addressing the owners of visual.ly)

I hope that "Chance to beat baseline" column isn't a naive measurement of "confidence" that's updated on every iteration.

I've mentioned this several times in threads such as these, but if you're blindly applying basic statistical analysis to a partially completed test, you are badly misinforming your customers.

Please read this: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Of course, if you are taking these results into account I apologise for mentioning this again and wish you every success.

One must love conclusions like this: "Even though the results aren't statistically significant (...), we were able to learn ..."