As always, test test test. I tried this for BCC, repeatedly, and never got it to work.
Also as always: do not assume that you're seeing the result of any one element unless you are testing that element in isolation. You might think the lightbox element is the most prominent aspect, but the call to action also changed from "Start Optimizing Your Website" to "Create A/B Test ->"
You see that arrow there? That arrow, tested in isolation, has been worth 10% to me before in some circumstances. Yeah, it blows my mind, too. And there were many other things that changed.
Interesting, yes. Agreed that there are too many variables which we tweaked. But this was a two stage A/B test so if you remove this popup box and move back to traditional signup form, we saw conversions drop back to normal.
May be it was the arrow icon that did the trick but I am not complaining since we saw such good increase in conversions! ;)
I can't agree more with this. We've run A/B Testing on this kind of modal dialog and we did not see an increase in conversion. In the article too many variables changed at once.
Changing the "Watch the Video" button to a "Start Optimizing" button is a huge change in and of itself.
>Changing the "Watch the Video" button to a "Start Optimizing" button is a huge change in and of itself.
As I wrote in case study, this change did not lead to a significant increase in signups. After we introduced popup signup box, signups increased by 50%.
Not only that, but do not assume that what works for one audience automatically translates (which is when testing helps). Different websites/products should target differently.
So even if this worked for one of your sites, it may not be the best thing to do with another.
This reminds me of the draw of the "next article" dialogue that slides in on the bottom right of NYTimes articles - it makes for a very compelling click.
The (not so simple) key to these UX tricks is to do them in a way that they are a service to the user which also happens to be one of the points of Jakob Nielson's latest alertbox: http://www.useit.com/alertbox/hated-design.html.
It would be interesting to see how much of the value of this you could get without an overlay and by adding eye-catching dynamism to the signup dialogue - sliding the middle of the page away to reveal it for instance.
EDIT It would actually be interesting to see how much of this benefit would come from simply having the signup form itself on the frontpage - all previous designs required a click to reach it
There are probably some things that work (naturally, most products can be improved upon), but I doubt that it is a solution for everything. Already there was a comment in this thread from someone who claimed the popup forms did not work in his case.
Everybody is using the "you should follow me on Twitter" line on their blog, just because one guy made a successful A/B Test with that line.
It just seems that the A/B-Testing companies seem to promise a lot. Will everybody who uses visualwebsiteoptimizer see a 50% increase in signups (when they test the popup forms)? Apparently not... So it is a false promise of sorts.
I'm a bit troubled with my results being used to support that conclusion, in the matter of a pharmaceutical researcher who published "Penicillin is not an effective treatment for brain cancer" would be troubled to be cited in support of "Well, that tears it, medical research is a sham."
It seems to me that the issue isn't A/B testing, it is blind adoption of other people's recommendations. You certainly don't need A/B testing to give people suboptimal advice -- designers, developers, and marketers do it every day without any evidence whatsoever! ("Eat more bananas, they're good for brain cancer!" <-- the state of web design best practices and, ahem, not far off from the state of nutrition research)
Of course, I have no doubts that A/B Testing is useful, I take more issues with the way it is being marketed. Likewise a lot of SEO is probably useful, but there are also a lot of black sheep.
Btw. I don't like your analogy, I did not conclude from your example that A/B Testing never works. But one example is enough to show that it does not always work. Specifically, the claim "use A/B Testing and your signup rate will improve x%" is apparently not true.
The same thing happens with optimizing code for speed/efficiency/etc. People want rules of thumb and they don't want to hear the truth, which is "Rules of thumb are usually bullshit. You need to measure, change something, then measure again. Your system may or may not be the same as mine."
Totally agree. A/B testing is not a panacea and certainly not every one who uses VWO sees improvements. In fact, last to last week, we prominently featured an article on our blog by Noah of AppSumo who said only 1 out of 8 A/B tests produce results of any sort. Here is that article: http://visualwebsiteoptimizer.com/split-testing-blog/a-b-tes...
1 out of 8 A/B tests produce results and that's a reality. And only the positive results get published so you may get an impression that A/B testing is snake oil.
Reasons why such case studies are important is not because these will work for everyone, but may work. In fact, if I knew modal boxes work every time why would I even A/B test them?
Naturally, A/B testing is just one part of puzzle but discounting it is snake oil is not justified.
No offense intended, I am sure VWO is a great product. Haven't tried it yet, but have been meaning to. Also the blog is a joy to read. It was more expressing a general worry.
Yes, I can see that A/B testing is definitely getting attention and hype and some people may directly implement results from A/B testing (which defeats the original purpose).
It sounds like you think that the reported results of these A/B tests are snake oil, not A/B testing itself. I think most authors of A/B-testing-report articles would recommend testing such changes yourself, as opposed to accepting their findings as fact for every site in existence. Those who said the popup forms did not work for them only know that because they A/B tested them.
I think a healthy conclusion is "this worked for us, so it might be a good idea, but you should test it for your site". Wouldn't you agree?
Yes and no - you can not test it all, there might be a point of diminishing returns after a certain level of optimization that has gone into a site. Just as with SEO - SEO is not completely useless, yet there are people who promise too much or even hurt you.
I suspect with A/B tests you could also hurt yourself, for example if you measure the wrong thing.
I don't know, maybe I am just a little bit tired of all the simplified promises.
I think the snake-oil nature of (much of) SEO stems from the fact that it's difficult or impossible to measure in an accurate, unambiguous way. Since A/B testing includes measurement by its very nature, I suspect it will be more resistant to snake oil.
Why not go a step further and have the signup right there on the homepage? :-) Tumblr does it and I've been getting killer conversion rates on Ruby Weekly and JavaScript Weekly doing it too.
If your form is too big, I like Focus.com's approach of having the simplest part of the form on the homepage which then leads you into the full deal. I'd love to know how that's performing for them.
Everyone thinks these things are hot stuff lately. I hate them because I'm there to read the content I was linked to. I won't have an opinion of if I'm even interested in subscribing/donating/etc until AFTER I've finished reading it so I don't even read them. I just hit close.
Essentially, they piss off first-time visitors to increase repeat visitor signups.
I think you didn't read the case study. We don't popup the signup form as soon as homepage loads, rather we popup when people click on Start Optimizing button on the homepage.
From my standpoint, it's like trying to browse a store that first requires you to create an account before you can see what the true cost of the item is. I still have no idea what specific information is required to use (my information 'cost').
However, many people don't mind throwing an email/pass at anything that comes up so I can see why it works. I'd be interested to see if your signup conversion rates stay consistent though.
Note that this form pops up only after the user clicks on a button:
"Implementation wise, here is what we did: on homepage, whenever a visitor clicks on Start Optimizing Now button, we show a “popup signup form” which has only two fields (username and password)"
We converted ShrinkingApp.com to modal signup forms 6 months ago or so and saw a slight increase in conversions, nothing too dramatic. I can see how it would work from a simplicity standpoint (less page loads, quicker, etc.).
I've used similar functionality on a real estate website, allowing people to see that the information they want is behind the registration form. Previously, the client averaged 1-2 signups a day. After the change, they started seeing 6-7 per day. So this definitely can work - but as with every other user interface component you should use A/B testing and see if it works for you before committing to it 100%.
Having your call to action button below the fold is really bad practice. It's no surprise that your conversions increased when you moved the button up. Also, I would suggest putting your registration form directly on the homepage instead of requiring a person to click a button to see it.
> We used our new revenue tracking feature to see impact of “popup forms” on actual $$ made, however it is still to early to say anything.
I think this last sentence is key. Raise a barrier and of course the number will go up, but average value per signup will go down. This might be fine for freemium or very good email marketing, but it may also have a negative effect. Only further testing will reveal the complete funnel around total visitor value.
This is ridiculous! Of course more people will sign up if you make them only fill out two things and force the popup on them, but it completely devoids the concept of 'signup' of its meaning. What you should care about is how many people sign up and use your service. If I was presented with a signup popup like that I would just input a fake e-mail address, try it out once and never come back. That's not what a signup is supposed to be.
I think popup forms are annoying and spammy. I'll never use them on a website of mine.
Using them might result in more email signups, but I think the response to your email blasts will be worse off and result in a lot of immediate "unsubscribes".
Popups are a trend I hope goes the way of blinking text.
Is it considered acceptable practice to not include a 2nd password verification text field?
I can see that there's less friction in the process (1 less field to fill out). Perhaps since this is a specific lightbox call to action instance and not the traditional register page, there is no password confirmation text field (meaning the full fledged register page would have a pw confirm field).
I have not used a second password verification text field in some of my sites. While I have not tested it, I prefer it. Instead, if someone thinks they have mistyped their password, they might just retype the whole thing. Or, I give them an option to "show" or "hide" the password which toggles between password dots and normal characters.
I found that when I removed the second password field users thought they were signing instead of registering. This lead to users creating multiple accounts and panic-y emails about lost data.
38 comments
[ 3.2 ms ] story [ 110 ms ] threadAlso as always: do not assume that you're seeing the result of any one element unless you are testing that element in isolation. You might think the lightbox element is the most prominent aspect, but the call to action also changed from "Start Optimizing Your Website" to "Create A/B Test ->"
You see that arrow there? That arrow, tested in isolation, has been worth 10% to me before in some circumstances. Yeah, it blows my mind, too. And there were many other things that changed.
May be it was the arrow icon that did the trick but I am not complaining since we saw such good increase in conversions! ;)
Changing the "Watch the Video" button to a "Start Optimizing" button is a huge change in and of itself.
As I wrote in case study, this change did not lead to a significant increase in signups. After we introduced popup signup box, signups increased by 50%.
So even if this worked for one of your sites, it may not be the best thing to do with another.
The (not so simple) key to these UX tricks is to do them in a way that they are a service to the user which also happens to be one of the points of Jakob Nielson's latest alertbox: http://www.useit.com/alertbox/hated-design.html.
It would be interesting to see how much of the value of this you could get without an overlay and by adding eye-catching dynamism to the signup dialogue - sliding the middle of the page away to reveal it for instance.
EDIT It would actually be interesting to see how much of this benefit would come from simply having the signup form itself on the frontpage - all previous designs required a click to reach it
Everybody is using the "you should follow me on Twitter" line on their blog, just because one guy made a successful A/B Test with that line.
It just seems that the A/B-Testing companies seem to promise a lot. Will everybody who uses visualwebsiteoptimizer see a 50% increase in signups (when they test the popup forms)? Apparently not... So it is a false promise of sorts.
It seems to me that the issue isn't A/B testing, it is blind adoption of other people's recommendations. You certainly don't need A/B testing to give people suboptimal advice -- designers, developers, and marketers do it every day without any evidence whatsoever! ("Eat more bananas, they're good for brain cancer!" <-- the state of web design best practices and, ahem, not far off from the state of nutrition research)
1 out of 8 A/B tests produce results and that's a reality. And only the positive results get published so you may get an impression that A/B testing is snake oil.
Reasons why such case studies are important is not because these will work for everyone, but may work. In fact, if I knew modal boxes work every time why would I even A/B test them?
Naturally, A/B testing is just one part of puzzle but discounting it is snake oil is not justified.
Yes, I can see that A/B testing is definitely getting attention and hype and some people may directly implement results from A/B testing (which defeats the original purpose).
I think a healthy conclusion is "this worked for us, so it might be a good idea, but you should test it for your site". Wouldn't you agree?
I suspect with A/B tests you could also hurt yourself, for example if you measure the wrong thing.
I don't know, maybe I am just a little bit tired of all the simplified promises.
If your form is too big, I like Focus.com's approach of having the simplest part of the form on the homepage which then leads you into the full deal. I'd love to know how that's performing for them.
Essentially, they piss off first-time visitors to increase repeat visitor signups.
However, many people don't mind throwing an email/pass at anything that comes up so I can see why it works. I'd be interested to see if your signup conversion rates stay consistent though.
"Implementation wise, here is what we did: on homepage, whenever a visitor clicks on Start Optimizing Now button, we show a “popup signup form” which has only two fields (username and password)"
I think this last sentence is key. Raise a barrier and of course the number will go up, but average value per signup will go down. This might be fine for freemium or very good email marketing, but it may also have a negative effect. Only further testing will reveal the complete funnel around total visitor value.
Using them might result in more email signups, but I think the response to your email blasts will be worse off and result in a lot of immediate "unsubscribes".
Popups are a trend I hope goes the way of blinking text.
I can see that there's less friction in the process (1 less field to fill out). Perhaps since this is a specific lightbox call to action instance and not the traditional register page, there is no password confirmation text field (meaning the full fledged register page would have a pw confirm field).