The source shows he's also timing the pick, which I think is probably the more important dependent variable here. Doesn't matter what number you chose - it's how long it took you to find it, click it, and hit submit.
Indeed. The goal there is to see if people think about the choice, then second guess it before actually submitting.
Edit: Quickly skimming through the database, it also reveals "tab" loaders who open up a bunch of tabs and let them sit. (It couldn't possibly have taken someone 300s to make a decision!) Not the goal, and it does skew the data, but still interesting.
I hope he gathers data from different social groups. Would be interesting to prove that techie-types have a numerical bias different from marketing types.
I remember reading an article on this about year ago on HN. I think that the conclusion seemed to be that human prefer odd and prime numbers (at least in range 1 - 20).
I was just thinking it'll be 7. My own theory is most people don't want to be in the middle, so they go for the middle of the upper half. That's what I did anyway.
The other methods don't have a default. Sliders don't really present a good way of doing a "non-choice" default. I decided to leave it fixed at 1 to start to see if that particular method skews because of the default. (Not exactly precise and thorough, unfortunately. There could be more metrics, like starting position, but I wanted to keep it simple so I could get it up and running quickly.)
That's what I was thinking too. My choice was influenced at least slightly by where my mouse was at picking time. Would have been better to have a text entry field I think.
I wish this data could be analyzed- I was debating between three, five, and seven, and chose nine just to throw off Benford's Law. I really think it is the first time I have ever chosen nine.
There is actually a "mind trick" where you make someone choose a number between 1 and 4, guess it is 3, then make them choose a number between 1 and 10, and guess it is 7. It works, because there is a statistical bias toward these numbers, but the way you do the trick is also important (the way you gesture with your hands primes the person to answer with these numbers).
I've always loved such mind tricks, and I have done this one quite a few times. Before discovering HN, I used to go out a lot, and, when talking to cute girls, I've done this "trick" many times for giggles. Some girls really believed I was a psychic when I got it right :)
Funnily enough, knowing about this trick and the "statistical preference" for the number 7 made me choose 7 in this experiment, without thinking twice about it. I realized this afterward...
Are you trying to estimate the bias depending on the type of selector used (slider, input, options, ...)?
Very interesting concept :)
EDIT, forgot to add: if this is the case, have you thought of storing the type of selector in a cookie, so that refreshing the page gives always the same type?
Yeah, I considered a more involved way of choosing the interfaces to present — the server attempts to even out the distribution of interfaces, use cookies to limit, etc — but I also just really wanted to get this made in a couple hours. The current implementation is about as simple as it gets.
Yes? What do you mean by app? It's an app whose entire purpose is to collect data. I wouldn't think of it as a Trojan horse, since it doesn't pretend to be anything else. I guess it's Trojan in that it was made in Troy, NY.
The goal at first was to just see what numbers people gravitate toward. I've heard lots of conjecture about how people pick 7 or 3 or 4 more than others, and for a variety of reasons, but had a hard time finding actual demonstration of this. Then, while implementing a choosing system, the problem became: how do you present the information so as not to bias it? This is why there are four different ways of picking. There are also a couple other metrics being measured, including a difference in phrasing (Pick a number… vs Pick a random number…) which may be interesting.
Apologies for any bugs or general wonkiness. The whole thing was a ~2 hour impulse project.
PS: The data will absolutely be shared! Just need time to do a breakdown of all the different permutations.
I presume this is your site? Your design has a serious flaw: the number "1" is "selected" when you first view the page, which biases the results. You apparently tried to "fix" this by forcing the user to move the slider before the choice would be accepted, but that also biases the results.
The only way to really make it unbiased is with a text box. The next best thing would have been ten buttons in a row (but then you would have had to make sure that they didn't go off the right side of the screen).
I'd actually be interested in how different ways of presenting the choice affect the number chosen. It would be interesting of the author ran an A/B test with different methods of selection (text box, row of buttons 1-10, row of buttons in random order, slider, etc) to see what the differences would be.
I would imagine you would get a distribution favored to 3 and 7. We did similar research during a Cognitive Science class. We asked a number from 1 to 4, and got over 40% 3's.
We also did research to find favored Mastermind patterns. Bias was a large problem there too. When presented with colors, people would pick a single color more often or place the same colors next to each other. When presented with letters, people would try to spell out words.
Peculiar: In product pricing and conversion testing, prices with 7's and 9's seem to provide more favorable results. I believe this is akin to the favorite-color bias we happened upon (7 is my lucky number!), mixed with the slight confusing nature of calculating/rounding down a price ending in 7 or 9 (hey, it's still $2999, so just 2 grand and then some).
I would also guess that it's biased more toward 7 than toward 3. That's the conventional wisdom, isn't it? Below 4, 3; below 10, 7; below 20, 17; below 40, 37. Or has this bit of pop-math folklore been lost in translation?
Does your university have a significant Asian population?
Three (homonym with "alive") is a lucky number in Chinese culture. Four (homonym with "death") is an unlucky number. Apartment buildings built for Chinese persons often skip all floor numbers with "4" in them as well as all apartment numbers with "4" in them.
"I’m staying at a hotel right now, there’s no 13th floor because of superstition. But come on man, the people on the 14th floor, you know what floor you’re really on. If you jump out of the 14th floor hoping to kill yourself, you will die earlier." ~Mitch Hedberg
He makes a fair point though--if you're so superstitious that you're uncomfortable with the thirteenth floor, shouldn't you be uncomfortable about it regardless of its nomenclature?
Don't know about 3, but if a price includes lots of 8s, the seller is definitely aiming at Chinese buyers. Seeing this left and right here in Vancouver, and it looks corny at best. Like trying to lure Russians with a picture of vodka, or Americans with that of a cowboy hat.
This would drastically vary.
When A person tells B to pick a number there are various things going on in mind. Once its brought down to select from 1-10 the choices are narrowed down to a single digit number. Hardly anyone would choose 1 or 10. If someone's birthday comes in between those numbers, the choice becomes obvious, other wise its moves to the favorable number or a lucky number.
7 being the universal lucky number many believe in that. For numerlogy believers 3 comes to be a common lucky number amongst many. So the choice, not sure why 4.
Anyways the reasons vary alot.
Do keep us updated w/ the result want to know how mind works over numbers for all.
The only problem being, many would refresh and choose all numbers, why don't you add a simple cookie or ip restriction to allow to choose only once? This would reduce alot of fake entries.
Interesting idea, but the slider may not be a great interface. It stops the experiment being purely about numbers - I found I picked '5' because it was close to being nice and symmetrical.
I think that is a good datapoint, myself. If there is a predisposition for people to choose "5" under those circumstances, we will have learned something.
This would be a fantastic app to run A/B -like testing on to determine whether say a circular "picker" gives different results than a horizontal "picker". You could even tighten it down to see if a random distribution of numbers between 1 and 10 gives different results than a ordered distribution.
It fits the model perfectly and, at least to my perspective, very clearly.
I also took 5 because it was close. Also, since I know people tend to pick 7, I somewhat tend to don't pick 7 when I'm asked something :p You could maybe "randomize" the order of the numbers, or hack them up so, for instance, 7 is far left and 3 is far right.
No, the UI is different every time you refresh the page. There are 4 UIs, randomly presented.
I'll be fascinated to see how the different UIs skew the results. Of course, there are now so many variables in this "experiment" it'll be hard to make any concrete determinations.
If there are enough data points (which it seems like there is), and each UI is chosen a random 1/4 of the time, then it will be very easy to isolate and determine the effect of the slider UI. Problems would arise if the slider was chosen 60% of the time in Firefox, but only 30% of the time in Opera, and if Opera had a 20% higher population of hackers, who had a distribution centered around 6 as opposed to the aggregate population centered around 5; as is, it sounds like there's plenty of random sampling solving these complications.
Honestly I wasn't expecting to get so many number picks so fast. The plan was to start putting together an analysis page once I got enough data, which I thought would be a couple weeks. I'm going to try and get one together as soon as possible.
Edit: Just had to enable billing in AppEngine so it can keep going! Way more data than I ever expected.
An alternative idea would be to display a link to a page where the data will be displayed once it is ready. I agree that just getting a "Science thanks you" message feels a little like being cheated, but it didn't bother me because I knew I'd be coming back here and waiting for info about results.
I wonder if adjusting the the numbers would influence the choice. Instead of listing them 1-10, maybe try a run with the numbers not listed in sequential order.
Whenever I see these sorts of things I try to psych them out. In this case I deliberately thought of a number, discarded it, and repeated this process several times (rather than putting down 7, which was my first thought). So, unless you're trying to measure frequency of picked numbers when the pickers are trying to game the system, I don't think this will prove much. Nice interface though.
That's why one of the metrics is time, both from start of page to picking a number, then start to hitting choose. It's not perfect, but it'll hopefully be possible to differentiate between people who pick right away, and people who think about it.
Hey, so I submitted it to reddit thinking it might get you a few more responses. It kinda ended up being the number one link on the front page. Hope that didn't hurt too much.
This is where Appengine shines - it scales painlessly, so there won't be any hurt (apart from being billed for the resources).
This scalability comes at the cost of a wee paradigm shift from relational database to datastore mindsets, but is well worth it. Developers can do what they do best instead of having to become system architects and admins.
Thanks for submitting! The number of responses is vastly more than I expected, pushing 100k uniques now. AppEngine has handled it wonderfully. It blew past the free quota, so I did have to enable billing. But, it's still only pennies so far, and a quota reset is coming up soon.
The amount of data has been a challenge, since I have to put together more efficient stats tracking. A good problem to have, of course.
I was expecting to have to spam the crap out of my Twitter feed and leave links everywhere to get even 100 participants. Glad that wasn't the case!
After you've entered your unbiased data, make sure to refresh the page and see what else you could have been confronted with. You'll notice the different variables in the experiment - just make sure to do this after in order for it to not influence your decision.
Without something like facebook or twitter login, the system will be abused. There is no stopping someone who just reloads ( or even make an automated script heh ) and keeps on choosing, resulting in flawed data. Just saying.
Totally agreed. I'm relying on people not caring enough to abuse it — a bit risky, I know. I just wanted something simple that wouldn't require people to jump through hoops just to vote. Also, the IPs are now one of the metrics, for regional breakdown, so that could be used to spot obvious anomalies.
On Refreshing, it changes the interface, every time you get a different input method. I guess its that way so as to avoid being biased over any number by default.
An interesting extension of this could be to add a poll here on HN asking, "Which number did you pick?" and then comparing the poll results to the actual results.
Brilliant idea, but haven't you built in bias by unblinding the experiment? I followed the link, got one implementation and keyed in '5'. Then I read your write-up and realised that there are more implementations, reloaded the page and thought deeply about the number I would/should choose on the slider.
I would imagine that if you did this survey in a crowd of sports fans, you would get a different distribution based on town or player followed. Plus, if I remember right 7 and 4 and not terribly lucky numbers in China.
142 comments
[ 2.9 ms ] story [ 210 ms ] threadEdit: Quickly skimming through the database, it also reveals "tab" loaders who open up a bunch of tabs and let them sit. (It couldn't possibly have taken someone 300s to make a decision!) Not the goal, and it does skew the data, but still interesting.
I figured 9 had to be the least picked, but I'm starting to think I should have gone with 6...
I've always loved such mind tricks, and I have done this one quite a few times. Before discovering HN, I used to go out a lot, and, when talking to cute girls, I've done this "trick" many times for giggles. Some girls really believed I was a psychic when I got it right :)
Funnily enough, knowing about this trick and the "statistical preference" for the number 7 made me choose 7 in this experiment, without thinking twice about it. I realized this afterward...
Very interesting concept :)
EDIT, forgot to add: if this is the case, have you thought of storing the type of selector in a cookie, so that refreshing the page gives always the same type?
How many people leave it at 1? This is kinda neat.
Also, it was nice of science to thank me.
So I don't know.
The goal at first was to just see what numbers people gravitate toward. I've heard lots of conjecture about how people pick 7 or 3 or 4 more than others, and for a variety of reasons, but had a hard time finding actual demonstration of this. Then, while implementing a choosing system, the problem became: how do you present the information so as not to bias it? This is why there are four different ways of picking. There are also a couple other metrics being measured, including a difference in phrasing (Pick a number… vs Pick a random number…) which may be interesting.
Apologies for any bugs or general wonkiness. The whole thing was a ~2 hour impulse project.
PS: The data will absolutely be shared! Just need time to do a breakdown of all the different permutations.
The only way to really make it unbiased is with a text box. The next best thing would have been ten buttons in a row (but then you would have had to make sure that they didn't go off the right side of the screen).
We also did research to find favored Mastermind patterns. Bias was a large problem there too. When presented with colors, people would pick a single color more often or place the same colors next to each other. When presented with letters, people would try to spell out words.
Peculiar: In product pricing and conversion testing, prices with 7's and 9's seem to provide more favorable results. I believe this is akin to the favorite-color bias we happened upon (7 is my lucky number!), mixed with the slight confusing nature of calculating/rounding down a price ending in 7 or 9 (hey, it's still $2999, so just 2 grand and then some).
Three (homonym with "alive") is a lucky number in Chinese culture. Four (homonym with "death") is an unlucky number. Apartment buildings built for Chinese persons often skip all floor numbers with "4" in them as well as all apartment numbers with "4" in them.
I'd like to see one that starts on the 14th floor. I mean, if we're going to indulge arbitrary biases, let's go all-out.
And while we're at it, let's have at least a few cases where we number the ceilings, too.
He makes a fair point though--if you're so superstitious that you're uncomfortable with the thirteenth floor, shouldn't you be uncomfortable about it regardless of its nomenclature?
7 being the universal lucky number many believe in that. For numerlogy believers 3 comes to be a common lucky number amongst many. So the choice, not sure why 4. Anyways the reasons vary alot.
Do keep us updated w/ the result want to know how mind works over numbers for all. The only problem being, many would refresh and choose all numbers, why don't you add a simple cookie or ip restriction to allow to choose only once? This would reduce alot of fake entries.
Simple small project, I liked it :)
It fits the model perfectly and, at least to my perspective, very clearly.
Edit: Doh. Looks like you're doing just that.
EDIT: for clarity i used opera, then chrome then safar then ff. The UI is different for all!
I'll be fascinated to see how the different UIs skew the results. Of course, there are now so many variables in this "experiment" it'll be hard to make any concrete determinations.
But there will be data and it will be cool.
Instead I got a message "Science thanks you."
How about if you want something from the user, you give something back.
There you go, another smile :)
Edit: Just had to enable billing in AppEngine so it can keep going! Way more data than I ever expected.
http://www.reddit.com/r/science/comments/gclpv/pick_a_number...
This scalability comes at the cost of a wee paradigm shift from relational database to datastore mindsets, but is well worth it. Developers can do what they do best instead of having to become system architects and admins.
The amount of data has been a challenge, since I have to put together more efficient stats tracking. A good problem to have, of course.
I was expecting to have to spam the crap out of my Twitter feed and leave links everywhere to get even 100 participants. Glad that wasn't the case!
Referrer might be interesting information. Do geeks behave differently etc.
middle number: for the graphical choices as it's common/easier to get a valid first click in the middle of the slider.
high number: for the key entry as if most people are right handed it's easier to get to the high numbers instead of going across the keyboard.
all that is predicated on the fact that people are lazy instead of random when using this.