A/B Testing You'll Actually Use: Optimizely (YC W10) Launches Out of Beta (optimizely.com)
It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public!
Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.
We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback@optimizely.com and let us know what you think!
97 comments
[ 2.4 ms ] story [ 151 ms ] threadOur goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.
We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback at optimizely dot com and let us know what you think!
"Also on the front page: when you enter a URL and hit enter on your keyboard you are taken to some TechCrunch article :)" ... well scratch that, it does not happen any more.
Great job otherwise, I'm "selling" this to my boss tomorrow :)
We've actually just implemented Undo! It didn't make it into the demo video but if you click on the little upside triangle next to a variation's title (e.g. "Variation #1") you have an option to Undo or Redo. There are keyboard shortcuts as well!
As for the front page, is it possible you accidentally hit tab and then enter? If you hit enter right away, it works. If you hit tab, it changes focus to the first testimonial link on the page which is that TC article. We'll fix this so it doesn't happen in the future.
Thanks again for the feedback!
Based on the results in the promo video, looks like you should have your video autoplay! ;-)
That's exactly right! Glad you like it.
You can also start at a Silver or Gold plan for your 30-day free trial and downgrade at the end of your free trial if you like.
Great to see you guys out in the wild!
What VWO has that Optimizely doesn't
- VWO has (JavaScript based) A/B and multivariate testing and split URL (based on traffic redirect) on while Optimizely is strictly (JavaScript based) A/B testing. You cannot do multivariate testing with it.
- VWO has heatmap and clickmap reports while Optimizely doesn't
- VWO has various advanced features such as visitor segmentation, cross-domain tracking, testing behind login wall pages, test results notifications, variation screen shot generation, etc.
- VWO has team collaboration such as multiple-permission based logins and subaccounts. Optimizely doesn't have that.
- In VWO, for all plans, CEO equally shares the support work ;)
What Optimizely has that VWO doesn't
- Changing layout of the page. Optimizely has a drag and drop feature in its test designer which is useful if you want to test different layouts (say sidebar to the right). To achieve the same in VWO, you need to enter some CSS (say float:right v/s float:left) as there is no drag-drop support (yet)
- Since Optimizely exclusively focuses on A/B testing (and not MVT), in its test designer you can change several elements of the page (say headline, image and text) at once to create a variation. In VWO (currently), you need to change one element at once for A/B test and multiple elements for multivariate test.
- In optimizely, results update on the page in real time. In VWO, results are also realtime but you have to refresh the report to see latest results (no auto-refreshing)
I hope this was a fair comparision. I wish Optimizely team great luck for their product. I hope A/B testing market is big enough for multiple players to survive!
One benefit you forgot to mention is that your cheapest plan ($49) comes out to $0.0049 per visitor. Optimizely's cheapest plan ($19) comes out to $0.0095 per visitor.
One thing I think we do well is allow you to try Optimizely and create your experiment without needing to create an account or give us your email address. The only time you need to create an account is when you want to save.
We also offer cross-browser testing and uptime monitoring and reporting.
One small correction: we do support testing behind login wall pages. You just need to put the embed code on your page and the tool will load your site even if it's behind a login wall or on your own local intranet behind a firewall.
Our biggest competitor is non-consumption. I'm glad there are other startups out there helping to educate the market about the benefits of A/B testing. A rising tide floats all boats!
I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts I created a variation page on WordPress, identified the original one and the conversion page. On the site i created the test, started it and it's just working.
But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard?
Edit: clarified
The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target.
The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment.
We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger.
Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!
If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it?
Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
1. For anyone who is non-technical, this can be daunting. Are the snippets in the right places? Was this deployed correctly? Have I set up my conversion goals correctly (GWO only allows for a single goal)? More often than not, tasks like this are passed from marketing onto IT, who are usually overworked as it is.
2. Even for technical folks, deploying and verifying a new version of one's site just to enable (and subsequently disable) a test can be a big hassle.
In our experience, it's often the marketers, inside sales, and product managers in an organization who have the strongest interest in A/B testing. We're trying to eliminate the burden (and subsequent pain) that testing places on IT/engineering, freeing those stakeholders to test to their hearts' content! :)
I didn't write ABingo with a pain point included, but for Appointment Reminder it will be helpful to do AB testing in non-Rails marketing pages, too. I wouldn't append too much effort optimizing your solution for the needs of people who have written AB test frameworks though.
My guess is, similar tests favor the "non-test" or quicker result/load, though I bet it would take A LOT of views to irk this out. Google always talk about these test on shades of blue, I imagine those tests are similar in the number of runs they require (hint: a lot).
That being said, almost nobody split tests. It's a serious problem because marketing usually needs technical to split test, so even if Optimizely cuts your temporary conversion rate by 9% you will still make that back quickly after a couple of split test iterations.
My number one question for optimizely is when are they going to include some sort of call back api or something to track real conversions. The number one concern SaaS businesses have is visitor -> paid account not visitor -> free account or signup screen.
In a couple of spots, the edit HTML dialog overlaid the items I was trying to edit. I attempted and expected to be able to move the edit dialog around the page and out of the way of what I was editing.
On my own site, I was particularly wanting to play with colors/themes. Not sure what the best way to implement this is, but the only way I could do that through this interface was with style tags.
Awesome product! I will likely use it in the near future.
This certainly isn't perfect and we'll work to improve it. Would you mind sending me the URL you were trying and some steps to repro the frustration you had? Feel free to email us at feedback@optimizely.com
Thanks!
You specify how much traffic you want to allocate to each bucket. You can do this when editing an experiment and going to "Advanced Settings" :: "Percentage of Traffic to Variations"
Kidding aside, the product looks really great - congratulations on launching!
For a critique, I'll say that the intro video is very dry. A/B it with something less Ben Stein?
One thing I'm always interested in is pricing. How did you go about determining your various plans and rates?
Do you think either would help?
Also, would love to share how we went about determining our various plans and rates. We did a lot of customer development on this and it might be interesting to other entrepreneurs. Let me know if you are interested and we'll blog about it.
Now, a coding question. I'm wondering: For the all_experiments_json property of the main optimizely object, why is the value JSON-as-a-string, rather than just JSON? You're forcing yourself to parse it from a string to native JSON for no apparent reason. Just curious :)
As for your question, I think you're right that we could just write this as straight JSON without wrapping it in quotes. What are the benefits of doing this?
I tried just putting the all_experiments_json property in Firebug without wrapping it in quotes and I get a 'SyntaxError: invalid label' error. Looks like that is because the keys of the dicts are strings. Probably a way to generate this JSON server-side that makes Javascript happy.
One potential downside of making it just straight JSON is that this is generated by our Python template and if for whatever reason this isn't valid JSON it would throw a javascript exception when the file is loaded.
Right now our backend that generates this JSON does this:
The dump is the block of text that gets wrapped by '' in our template.Thoughts on a better way to do this?
Removing the quotes won't fix the problem, because you've also altered the way characters are escaped via the .replace() calls.
One potential downside of making it just straight JSON is that this is generated by our Python template and if for whatever reason this isn't valid JSON it would throw a javascript exception when the file is loaded.
I'm not familiar with Python, but there shouldn't be any reason why simplejson would return invalid JSON. In the event that it does, your script would still break, anyway. Granted, it won't be a javascript exception, but it would certainly fail elsewhere.
Like I said before, it really doesn't matter because it works :)
To clarify: If you want to run one experiment on a single dynamic page (not multiple dynamic pages) you can do that as long as the structure of the page does not change. The best way to find out is to just try it. Load one of these dynamic pages in the tool and create your variations. Then put the embed code on your site and click "Preview variation" from the variation drop-down menu (you can get there by clicking the small upside down triangle next to the variation title, e.g. "Variation #1"). If you have the embed code on your site this will show you the page on your live site. You do NOT need to start the experiment to do this.
Hope that answers your question
If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky.
You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around.
EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.
We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
True, but that doesn't help people who don't get that much traffic within a 30 day period.
We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
Offering refunds is great, but it's even better if they're not needed. :-)
Certainly agree! :)
Requiring the user to remember to cancel or downgrade seems potentially sneaky.
One caveat: I've seen successful trials that are a week or two long. At thirty days (and with users who aren't very committed), you might have an issue with users forgetting about the forthcoming charge, and then getting angry when it shows up on their bill next month.
Also, as someone that really wants to incorporate this tech into my larger scale marketing efforts I am put off by the pricing models to both of the above mentioned sources.
When only one of several ad campaigns are doing 100k visitors a day you can see how that pricing model does me no good. Particularly when you take into consideration that I never stop testing -something- on a campaigns landing page.
Also would love an API so that something like this could be tightly integrated into my custom conversion analytics solution.
Congratulations on your launch nonetheless.
Great feedback on the pricing model. I agree this isn't perfect. We do offer a 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason, we'd be happy to offer you a full refund for the month.
You think like a winner, my man. You had your opportunity to rag at the "competition" but you were wise enough to see the big picture. Cheers!
Overall, this looks like a great product. I know a number of people who find Google Website Optimizer complicated to use, and I would definitely recommend this to them as a simpler option. I love how slick the browser interface to edit pages is, and I think having the default 'engagement' metric so people can see results without having to set up a goal page is a brilliant idea.
There are a few things you have done which I would consider doing differently.
1. It looks like your mission is to make A/B testing really easy, but your pricing page at the moment doesn't really reflect that. Number of visitors tested is an easy metric, but one that it is hard for me to interpret without lots of knowledge of A/B testing. How many tests does this mean I can run how quickly?
I would also reconsider the additional features you offer in premium packages. Cross-browser testing sounds complex and makes me worry that your site edits will fail in IE6. I don't want to have to test it, I just want it to work. With uptime monitoring, what does this have to do with A/B testing? Bigger sites probably already have some form of monitoring already anyway, so it looks like they are going to pay for something they don't need. I think your core product is strong enough that you don't need to offer these.
2. Showing me the percentage significance level appeals to my inner stats nerd, but I suspect the sort of people I think will benefit most from Optimizely will have difficulty interpreting this number. What level is 'ok'? Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I need 5/5? Google deal with this very well with their bars which turn red or green when they reach significance.
3. The 'select container' option to expand the selection seems non-obvious, and isn't how multi-select works in any other interface I've seen. Maybe allow people to select multiple components and then take their deepest common parent?
There are also some additional features I personally would like to see
1. It would be great if you gave an estimate for how long until my experiment will reach an appropriate significance level (obviously based on % change and traffic seen so far).
2. I would like to be able to choose my conversion action by clicking on a form button or link in your page editor.
3. It would be amazingly useful to have some automatic suggestions for how a page could be changed - on many occasions I've seen people resist A/B testing because the options are so wide and they don't know what to do. Doing this for some simple suggestions sounds possible - e.g. making key links bigger and moving them up the page. Doing anything more sophisticated could be a good challenge though :-)
On pricing page: I agree this isn't perfect. What metric would you like us to segment our plans by ideally? We want to make this as simple as possible so folks know how to budget for this and so they can easily know which plan makes the most sense for them.
On additional features: Are there any other "value-add" services you think bigger companies might want besides just more visitors?
On 'select container': we realize this is a bit confusing and we're working on implementing a multi-select almost exactly as you described.
On choosing conversion action: right now we automatically track all reasonable conversion events on a page (clicks, form submissions, subsequent pageviews, custom events) and we want to make this even easier by allow you to explicitly create a custom conversion event to track by specifying it when you are editing the experiment.
On automatic suggestions for how a page could be changed: this is a hard one and we hope to get there eventually. In the mean time we're going to try to do a better job blogging about best practices and lessons we've learned working with our customers.
Thanks again for all the feedback!
One test, 30k visitors, x statistical significance - $100.
Perhaps with the option of only deleting variations on the fly that don't work.
Pete