When you shake the bag, smaller candies settle down to the bottom, the larger ones get to the top. So even if someone tries to shake the bag to get a 'random' sample, they will be getting a biased sample if they pick all 5 candies from the same layer (usually the top).
On similar ground, this reminds me of Deming's red bead demonstration, relating statistics to corporations and management practices. Best explained dynamically: https://www.youtube.com/watch?v=JeWTD-0BRS4 (delightfully, this is posted by the Mayo Clinic).
Thanks for sharing this. Initially I found the video to be a bit dull but on reflection I think it's a brilliant demonstration on how companies tend to be managed in practise.
Exactly. Look into more of Deming if you liked that; he was brilliant all around and accurately evaluated and solved many management problems on a grand scale.
I would really have expected atleast a small number of people to have carefully looked to workout the distribution and whether there were a higher density of one type (e.g. small) at one position.
I imagine the use of a bag rather than a Jar as per the usual school fair game could make spotting these harder though (unless they're allowed to pick up the bag).
People forget that Galton's original example was with a group of people with good domain knowledge. It would be interesting to try the experiment on a group of old-school sweet sellers.
In this day and age nobody is going to hand the scale to the next group with the bag sitting on top of it? Its a bit cynical but I would expect that some pairs estimate to be really really close if not spot on as an effect of they just weighed the bag directly.
You could keep the bag on your desk and have students come up and sample from it and weigh them at their own desks, thus ensuring that the bag never comes close to the scale.
People routinely overestimate weights. Speaking from experience, it's possible to routinely win bets with an unmarked 2kg dumbbell (which weights "at least 5!")
All you have to do is use a scale which is small enough that it can't measure the weight of the complete bag.
I guess that still doesn't prevent someone from weighing all of the candies, one small batch at a time, and adding up the results. But hopefully in a classroom it would be obvious if someone was trying to cheat that way. I suspect that asking the students to work in pairs helps to discourage those kinds of shenanigans, too.
By that logic, shouldn't they be guessing too low? I think the point is they realise there are different sizes, but consistently overcompensate for this effect.
> Now call out to the students who are sitting near where you hid the envelope: “Um, uh, what’s that over there . . . is it an envelope??? Really? What’s inside? Could you open it up?” A student opens it and reads out what’s written on the sheet inside: “Your guesses are all too high!”
Maybe it's because I recently did some reading about magicians, but if I were one of the students I would be thinking that he could have any number of hidden envelopes with different predictions, and he just chose the one that ended up being correct. Of course, I'm deliberately missing the point of the story.
The point is that they all guessed low. The weight of the bag never changes, and there's no pretense that the teacher didn't know how much the bag weighed all along. You don't even need the envelope - you could just weigh the bag at the end. Same effect.
"he just chose the one that ended up being correct"
Yes, but it wouldn't be so impressive if he were to choose the envelope which said "your guesses were all wrong, and normally distributed about the actual value".
Correlations between participants (such as due to a systematic bias such as a selection bias) is one of the most common ways for 'wisdom of crowds' to fail. So I'm not sure what your point is.
My point is this accidental selection bias doesn't depend on the crowd- you could just as easily make this mistake if you are 1 person in a room by yourself. Some examples of what I would call unwisdom of crowds might be the biasing feedback effect a prediction market's current price has on people's judgement, or how those with the most money (or the with the smallest marginal value of a dollar) exert a larger influence on a prediction market's predictions, if they are implemented that way.
This post is also from a teacher of statistics. Sure, you can weigh the whole bag to get the total (or average) weight; how do you propose to measure an aspect of an entire population, say one as large as a country? Sampling has its uses, when done properly. Besides, as others have pointed out, most people are notoriously bad at estimating weight.
The "wisdom of crowds" part of the title is a bit unfortunate. While there can be systemic problems with that type of approach, this does not really demonstrate them.
What is demonstrated is when you give the students an algorithm for a biased estimator, the estimate they get is biased. This is good; empirical demonstration is useful... but it isn't the wisdom/unwisdom of crowds, really.
edit: good responses! Unfortunately I don't have enough time right now to properly clarify how/why I'm looking at it this way.
I think the point is that you are taking the weights of the candies as proxies for the guesses of individuals - the (sum * 20) procedure is effectively an average-based estimate, like with the 'traditional' wisdom of crowds type approach.
He is then simply showing that for certain populations such approaches won't necessarily give good expected results, and so one should always exercise caution - whether the population is sweets or people's guesses at the weight of a cow is essentially immaterial.
But that misses the whole point of why the "wisdom of crowds" actually works when it does. To wit; the individual biases and estimation errors can tend to cancel out. This is exactly the opposite of the demonstration, which is a systemic, biased estimator.
No, I didn't miss that. It the lack of "crowd" that is the problem. There is no crowd, really, there are a bunch of people performing a sampling algorithm. But crowds don't work that way.
This is a perfectly good demonstration of bias in statistical estimators. It just doesn't have anything to do with crowds, their wisdom, or their unwisdom.
I would dispute that he (or I) have missed that point - he is simply giving an example as to why one should, as is always the case when undertaking statistical study, make a reasonable attempt to understand the population from which you are sampling.
The wisdom of crowds effect is not a 'strong result', and it can be hard to work out whether it will apply in any given situation, and so it is prudent to exercise some caution, and to teach people why said caution is warranted, which is the point of his little game.
The 'wisdom of crowds estimate' (take 5 samples and average) isn't a biased estimator, because averaged across all realisations it would produce a correct result. The bias creeps in mostly because of careless assumptions on the part of the samplers.
You said "that misses the whole point of why the "wisdom of crowds" actually works when it does" - I guess all he's saying is that it doesn't always work, which I presume you agree with, and he's giving an example of why. To flip your argument around: saying that the wisdom of crowds works when it works is simply tautological.
edit: just to clarify, the "crowd" here is the group of sweets, not the participants. The participants are multiple realisations of the estimator, at least that is my interpretation (because if not, then I agree with you - the participants aren't a "crowd" in that sense).
re your edit: yes it was the latter interpretation that I had a problem with. The "group of sweets" == "crowd" doesn't really work for me, but I'll have to think about why.
Yeah I only realised that you were interpreting it differently when I saw your reply to jimhefferon. If you do figure out what bothers you about the sweets==crowd interpretation then do please reply - statistics is so slippery that it takes a lot for me to feel fully confident with any particular idea.
Ok, I think the simple version of this is: You are basically constructing a situation where the dominant bias is the same for all of the measurements (e.g. pairs of students). So you have a problem with your method of sampling being biased in a predictable way. So far, so good and this is an interesting lesson.
But they are all using the same estimation method from this biased information, and they are all (by construction) creating the same bias, and that is very much not crowd-like behavior.
Applying this as an example of the problem with "wisdom of crowds" seems to really miss the point to me, because you created a situation where they didn't behave like a crowd.
On further reflection, I have to concede that it is not a particularly good example of the wisdom of crowds idea.
Whilst I still think you can interpret his experiment as showing something about how and why WoC can go wrong, essentially along the lines I previously argued, it is not obvious, and there is some subtlety in the interpretation.
Your original phrasing "the "wisdom of crowds" part of the title is a bit unfortunate" seems most appropriate; indeed the word "crowd" doesn't appear in the rest of the piece.
Seems fair to me. Note that he published all the estimates and then had people make predictions as a group, which is pretty much the way to extract wisdom-of-crowds effects.
I agree the estimator is biased, but in good part because of how people pick 5 from 100.
We are all biased in some ways. If a group shares the same biases, then a wisdom-of-crowds approach will yield a wrong answer in which everybody is confident. I think it's a fine lesson for people trying to work in this fashion.
From the instructions: "Explain that their goal is to estimate the total weight of all the candies in the bag. They can choose their 5 candies using any method–systematic sampling, random sampling, whatever. Whichever pair guesses closest to the true weight. they get the whole bag!"
They knew their goal was to estimate the bag, and they were allowed to pick any five candies they wanted. If somebody picked in proportion to frequency, they'd be fine.
My chemistry teacher pulled a much better example of "Wisdom of crowds".
We were told to measure exactly 25 ml of both water and alcohol, mix them, then report the volume.
It was a competition to see which team got the closest to 50 ml...
So we all reported our values on the board at the front for comparison.
At the end, the team closest to 50 was declared "A Liar" because alcohol and water do not mix to the same volume as their constituent parts, thus the final volume should have been closer to 46ml and the team closest to 46 was awarded as the winner of the competition.
Blew my 15 year old mind and I never biased my answers towards my bias again.
48 comments
[ 3.2 ms ] story [ 125 ms ] threadSo nobody in the class just waits until you're not looking and weighs the whole bag with the scale?
Or simply a scale incapable of measuring a weight much larger than the heaviest piece of candy.
http://en.wikipedia.org/wiki/Granular_convection
That's how you increase fruit content in your müsli - shake the bag, scoop from the top and leave oats to others :)
On similar ground, this reminds me of Deming's red bead demonstration, relating statistics to corporations and management practices. Best explained dynamically: https://www.youtube.com/watch?v=JeWTD-0BRS4 (delightfully, this is posted by the Mayo Clinic).
I imagine the use of a bag rather than a Jar as per the usual school fair game could make spotting these harder though (unless they're allowed to pick up the bag).
As you pass the 1.5 kg bag to the next group, it's easy to see that that it's not 3-4 kilos as you have just guessed.
I guess that still doesn't prevent someone from weighing all of the candies, one small batch at a time, and adding up the results. But hopefully in a classroom it would be obvious if someone was trying to cheat that way. I suspect that asking the students to work in pairs helps to discourage those kinds of shenanigans, too.
The first reaction I think any class I was in would have to this demonstration would be to figure out how we're being cheated.
Given a bag with a random sampling of candy and being told to 'pick 5 pieces' I doubt I would choose 5 of the same large candy bars.
It seems highly surprising that 100% of the time this is done you don't have a single pair of students reaching just a bit further into that bag.
It doesn't even seem to be a particularly impressive demonstration.
Maybe it's because I recently did some reading about magicians, but if I were one of the students I would be thinking that he could have any number of hidden envelopes with different predictions, and he just chose the one that ended up being correct. Of course, I'm deliberately missing the point of the story.
Yes, but it wouldn't be so impressive if he were to choose the envelope which said "your guesses were all wrong, and normally distributed about the actual value".
http://phenomena.nationalgeographic.com/2013/01/31/the-real-...
This post is more "ask a bad question, get a bad answer".
What is demonstrated is when you give the students an algorithm for a biased estimator, the estimate they get is biased. This is good; empirical demonstration is useful... but it isn't the wisdom/unwisdom of crowds, really.
edit: good responses! Unfortunately I don't have enough time right now to properly clarify how/why I'm looking at it this way.
He is then simply showing that for certain populations such approaches won't necessarily give good expected results, and so one should always exercise caution - whether the population is sweets or people's guesses at the weight of a cow is essentially immaterial.
This is a perfectly good demonstration of bias in statistical estimators. It just doesn't have anything to do with crowds, their wisdom, or their unwisdom.
The wisdom of crowds effect is not a 'strong result', and it can be hard to work out whether it will apply in any given situation, and so it is prudent to exercise some caution, and to teach people why said caution is warranted, which is the point of his little game.
The 'wisdom of crowds estimate' (take 5 samples and average) isn't a biased estimator, because averaged across all realisations it would produce a correct result. The bias creeps in mostly because of careless assumptions on the part of the samplers.
You said "that misses the whole point of why the "wisdom of crowds" actually works when it does" - I guess all he's saying is that it doesn't always work, which I presume you agree with, and he's giving an example of why. To flip your argument around: saying that the wisdom of crowds works when it works is simply tautological.
edit: just to clarify, the "crowd" here is the group of sweets, not the participants. The participants are multiple realisations of the estimator, at least that is my interpretation (because if not, then I agree with you - the participants aren't a "crowd" in that sense).
But they are all using the same estimation method from this biased information, and they are all (by construction) creating the same bias, and that is very much not crowd-like behavior.
Applying this as an example of the problem with "wisdom of crowds" seems to really miss the point to me, because you created a situation where they didn't behave like a crowd.
Whilst I still think you can interpret his experiment as showing something about how and why WoC can go wrong, essentially along the lines I previously argued, it is not obvious, and there is some subtlety in the interpretation.
Your original phrasing "the "wisdom of crowds" part of the title is a bit unfortunate" seems most appropriate; indeed the word "crowd" doesn't appear in the rest of the piece.
I agree the estimator is biased, but in good part because of how people pick 5 from 100.
We are all biased in some ways. If a group shares the same biases, then a wisdom-of-crowds approach will yield a wrong answer in which everybody is confident. I think it's a fine lesson for people trying to work in this fashion.
They knew their goal was to estimate the bag, and they were allowed to pick any five candies they wanted. If somebody picked in proportion to frequency, they'd be fine.
We were told to measure exactly 25 ml of both water and alcohol, mix them, then report the volume. It was a competition to see which team got the closest to 50 ml...
So we all reported our values on the board at the front for comparison.
At the end, the team closest to 50 was declared "A Liar" because alcohol and water do not mix to the same volume as their constituent parts, thus the final volume should have been closer to 46ml and the team closest to 46 was awarded as the winner of the competition.
Blew my 15 year old mind and I never biased my answers towards my bias again.