Probably worse than most textbook corporate ads: the 'free guide' and 'read more' are too small to catch the eye, and the title is boring and generic even so even the sort of person that reads ads and is in the target audience might pass. Unsurprising the novelty ad performs better, but a better conventional one might too.
I'd wonder about click through vs. conversion and if there's a difference there.
Might just be because of novelty factor, but also may be tricking less sophisticated users because it looks different and those users may be less likely to convert.
It'd be interesting to see conversion results too.
I've seen variations of this article before. It's the novelty. "That's different... What would someone pay real money to advertise with something that looks so cheap and crappy? Click."
Their argument consists of linking an optimizely screenshot.
12 vs 24 clicks is not significant, it could’ve gone either way. Also given this minuscule sample, it’s easy to conduct p-hacking to get the desired outcome
Could you explain the calculations that lead to the claim that the result is not significant? From what I can tell, if we assume that clicking the ad is a weighted binary variable, eg. what is modelled as a "proportional distribution" there's a statistically significant difference between the two results. It's even pretty strong at P=0.006 (per https://epitools.ausvet.com.au/ztesttwo), but I might be missing something?
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
The author is using a tool designed for landing pages. The quality of the samples are going to be wildly different and that needs to be taken into account.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
As long as there's not a bias favoring one group in the collection of samples, statistically significant is statistically significant. There's no such statistical thing as a "tool designed for landing pages", but instead tools that compare occurrences in different population.
It’s statistically safest to live on Mars to avoid bear attacks.... the data matters. There’s limited real world application, and that’s what matters most. Semantics and arguing definitions isn’t useful.
You're handwaving about why there's some special definition of "significance" here, and when called on it, it comes down to "I don't feel like this is true".
Valid arguments, better formed, are: A) you're not sure it's representative of a real campaign, B) you're not sure it predicts the end-outcomes we really care about instead of some intermediate measure. Neither of these improve with more n, and so they're not significance related issues--- 4000 clicks doesn't improve either A or B.
I'm not off in the semantic weeds. And I think it's kind of embarrassing you're trying for a witty rejoinder instead of giving -some- kind of cogent argument.
Sibling comments are right on the technical side, but to help build intuition: Statistical significance isn't solely about 12 vs 24, you have to take into account the total population as well (5563 vs 4403). In particular, the smaller population had more than double the results - this is a hint that you have to do the math and can't just say "it's not significant", because it really well could be.
You can have a sample size of 5 and have it be statistically significant. Or maybe you are using some non-statistical measure of statistical significance?
The fact that it has a significant p-value is interesting, but the lack of information about how the author decided when to stop is suggestive of p-hacking (i.e. we don't know how many screenshots were taken, but we understand that the author posted only the most favourable one)
In the real world, we often have to work on a "preponderance of evidence" standard to actually get things done.
Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.
I think there can be prior that a professional looking ad can generate more clicks. Your argument shows a lack of statistical understanding - conditional on this data, the Bayesian approach would be to update the prior (whether A is better, or they’re equally as good) with the data collected. With such a small dataset, you might end up with a belief that there’s a 60% probability that B is better than A, but that’s not significant enough to conclude that B is in fact better than A, as you still have a lot of uncertainty.
With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.
I addressed in my second sentence that I disagree with that prior. I understand the statistics perfectly well.
And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.
P=.95 still is an arbitrary threshold, even if it's a commonly used one.
Well, with low probability events you can go a bit further with your back of the envelop calculations, because that means they're more or less Poisson distributed. The average is somewhere around 16 so that gives a standard deviation of 4.
So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.
Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.
So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.
I assume that one takes the uneven sampling into account? That should boost the significance a bit, though perhaps a bit more than I'd originally assumed. It remains a back of the envelop calculation after all.
I'm not sure the author really understands the math (from his comments) but he's right that it's significant, where significant means "I am at least 95% confident that the observed improvement is caused by this change". The test is fine (assuming he didn't just check the results every day until he found one that would make for a fun blog post).
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
I think the objection is about the phrasing that can be interpreted as "I will only be wrong in at most 5% of significant results", while what we mean is "if there is no effect, we will only get so large differences in at most 5% of cases".
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
> The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
Also when running campaigns, I’ve seen an ad suddenly get way more clicks than usual in a short timespan. Depending on how clicks are counted, just one user who gets click-happy could skew the result.
But in this case I suspect people were just curious why the ad looked like that and clicked to find out. Those may not be the people likely to convert.
Making people click is not the problem. People follow garbage click-bait links all the time. Design is not just about "making things look nice". It's just as much about creating identity. What does it matter if you bring users to your site if it's the wrong kind that does not convert to real customers? Seems like there's a particular type of business leaders that thinks marketing is just a numbers game.
Yeah exactly, another thing he could've done is put a girl in a bikini or a gross pic of toenails, a la Outbrain. That'll get people to click more than a "normal" ad but only the hackiest of hacks thinks that's the ultimate goal.
Exactly, which is why the most garbage click bait ads you will find on the internet are for websites that make their money on advertising rather than by selling products. Because those sites are just trying to get eye balls to increase impressions and thus make more money.
They don't have to worry about actually selling you a product.
> An ad that took me 15 seconds to create had a 2.5x better clickthrough rate than one done by a paid designer. If this were an actual campaign it would mean 2.5x more sales leads or user signups at a lower acquisition cost, on top of shipping faster with less overhead.
I find the paid-for design to be uninteresting and unattractive. If you were to ask me what I'd find when I clicked through, I would say that it's likely I'd be asked to sign up for a low-quality mailing list to receive a bland white paper.
The scribble version is, at least, novel. Still probably wouldn't click through, but at least it would grab my attention for a moment.
What was the bounce rate, avg visit time and conversion rate (email signup) between people who came from the non-designed vs. designed ad though?
A higher clickthrough rate isn't necessarily better if more of the people who came from the non-designed ad didn't do much or anything on the page and mostly clicked out of curiosity.
I've read somewhere that porn ads and political ads perform better when they don't look professionally made. If I were a publisher, I would be very selective about the quality of the ads being displayed. I would also be in the minority since even CNN has ugly ads at the bottom of every article ("You won't believe what she looks like today!")
The first thing I read on the hand drawn version was "free guide". In the "professional" version, my mind glossed over those words completely. I don't know why, other than maybe the pretty pictures at the bottom pulled my eye away.
It's old news that in copywriting, copy with some misspellings typically performs better. While you may lose some who see the ad as less trustful, people respond to ads that make them feel like they've discovered something secret, new or underground.
Ignoring the other points about bounce rate, brand identity, and statistical significance for a second, I think an aspect that's missed a lot is that good design accomplishes a goal. The hand-drawn ad, by this definition, stands out more than the "well-designed" one just by virtue of being visually distinct. Asking a firm to help you design an ad capaign should involve iteration, testing, and refinement as different ideas are tested on your audience. If you ask a firm to take some words and make them look good you're missing out on most of the design process.
The design of this site is unconventional by today's design standards, but it has pricing clearly presented and is 100x more memorable, just like the OP's example. (As a side note, sites like this make me miss the old days of the Internet when everything was unique.)
People are indifferent to car dealerships (They're all typically the same. Yes, luxo-brands will have free cookies and nicer waiting areas...), just like with insurance-- they are "boring" verticals so having something so out there works great. When you look at insurance companies, which companies have memorable marketing? Companies like Liberty Mutual with their vanilla "Save $200/year" commercials, or the Geico Gecko, "We are Farmers....", All State's Mayhem, etc.. The cheesier the better, since auto insurance is a boring, fungible product to consumers. Compare to another boring vertical like Men's Razors. They're essentially all the same. How are any of the marketing campaigns from P&G/Unilever actually differentiated enough to be memorable (ignoring the duopoly considerations)?
Having unfortunately worked in marketing, everyone seems to shoot for the middle-- and they end up being bland and forgettable. Chances are, you're not going to have a super memorable marketing campaign by doing something slightly different than everyone else. The companies I've worked with where their marketing is intentionally cheesy have always had better reception than the vanilla "professional" brands.
Help me understand this a bit -- why would a subreddit buy advertising? Is there a feedback loop or revenue share with mods?
Or is this purely a ponzi thing where you buy bitcoin worth 100 bucks, spend 100 bucks on advertising and end up with the same bitcoin now worth 1000 bucks.
Digital ads used to be noticeable (because flash and gifs). Google and fb ads are bland and don't stick out, and you don't even have an option. I wonder how advertisers missed this bit, novelty used to be the desired goal in old media advertising.
136 comments
[ 3.1 ms ] story [ 205 ms ] threadMight just be because of novelty factor, but also may be tricking less sophisticated users because it looks different and those users may be less likely to convert.
It'd be interesting to see conversion results too.
12 vs 24 clicks is not significant, it could’ve gone either way. Also given this minuscule sample, it’s easy to conduct p-hacking to get the desired outcome
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
Valid arguments, better formed, are: A) you're not sure it's representative of a real campaign, B) you're not sure it predicts the end-outcomes we really care about instead of some intermediate measure. Neither of these improve with more n, and so they're not significance related issues--- 4000 clicks doesn't improve either A or B.
I'm not off in the semantic weeds. And I think it's kind of embarrassing you're trying for a witty rejoinder instead of giving -some- kind of cogent argument.
It's a statistically significant difference.
The fact that it has a significant p-value is interesting, but the lack of information about how the author decided when to stop is suggestive of p-hacking (i.e. we don't know how many screenshots were taken, but we understand that the author posted only the most favourable one)
Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.
So... seems like useful data to me.
With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.
And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.
P=.95 still is an arbitrary threshold, even if it's a commonly used one.
So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.
Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.
So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
That seems like a highly significant result to me...
But in this case I suspect people were just curious why the ad looked like that and clicked to find out. Those may not be the people likely to convert.
They don't have to worry about actually selling you a product.
That is a false assumption.
Not all clickers are equal.
Not all leads are equal.
More is not necessarily better.
Better is better. Full stop.
The scribble version is, at least, novel. Still probably wouldn't click through, but at least it would grab my attention for a moment.
Use a design that costs me money to produce and doesn't work, or...
Use a design that does not cost me to produce and does work, and then find something new to do when the novelty wears off?
Not a tough call for me.
It communicates the useful part (in particular, the "call to action") clearly and instantly, without distraction.
There's more going into this than just the concept of writing on paper.
A higher clickthrough rate isn't necessarily better if more of the people who came from the non-designed ad didn't do much or anything on the page and mostly clicked out of curiosity.
I'll click on free.
Sure if your product is good some industries do not care how your design looks.
It's all down to the industry. If you were selling expensive water to schmucks, MS Paint simply will not suffice.
Very ugly site, but impossible to forget that branding. It's my understanding they do very well.
A lot of Indian websites look like this too (like paper advertising leaflets).
People are indifferent to car dealerships (They're all typically the same. Yes, luxo-brands will have free cookies and nicer waiting areas...), just like with insurance-- they are "boring" verticals so having something so out there works great. When you look at insurance companies, which companies have memorable marketing? Companies like Liberty Mutual with their vanilla "Save $200/year" commercials, or the Geico Gecko, "We are Farmers....", All State's Mayhem, etc.. The cheesier the better, since auto insurance is a boring, fungible product to consumers. Compare to another boring vertical like Men's Razors. They're essentially all the same. How are any of the marketing campaigns from P&G/Unilever actually differentiated enough to be memorable (ignoring the duopoly considerations)?
Having unfortunately worked in marketing, everyone seems to shoot for the middle-- and they end up being bland and forgettable. Chances are, you're not going to have a super memorable marketing campaign by doing something slightly different than everyone else. The companies I've worked with where their marketing is intentionally cheesy have always had better reception than the vanilla "professional" brands.
https://medium.com/@paulbars/magic-internet-money-how-a-redd...
Or is this purely a ponzi thing where you buy bitcoin worth 100 bucks, spend 100 bucks on advertising and end up with the same bitcoin now worth 1000 bucks.
Some donkeys look cute