> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected .
They guessed I was 'Under 60', which I am, but over 50% of people fit into the category of under 60... so the fact they can guess this with only 50% of accuracy doesn't really feel right?
GP is saying it's even more than 50/50. Less than half of people are >60.
A more extreme example would be if it guessed: "You are younger than 99 years old."
I think they're saying over or under 60 is completely arbitrary when judging only from 'random' answers. It could be over or under a month, or 1000 years, the result would still be 50/50 because the ability to create a sequence that appears random has no link to age.
I also guess that the segment of the population interested in a self-described "digital publication that makes data fun" probably skews even younger than the population. FWIW I was guessed to be over 60 but am not.
Yeah, most of us here got “over 60” because we inputted data with repeated values, because we know real random data has repeated values. I’m not sure if we overdid it or the study has a weird metric of “looks random”.
Hm I'm curious about the complexity measurement for the coin tosses. How does it work?
There is suspicious clustering in the plot, at complexity slightly lower and higher than -1. Complexity -1 is also more sparse than seems reasonable from the shape/density of the cloud.
I think, that's a common misconception about randomness: It doesn't mean the number has to change with every roll of dice. On the contrary the longer the sequence large clusters of the same number will occur.
It's maybe not so counter-intuitive as the Birthday paradox, but maybe already showing how bad our intuition is at grasping randomness. Initially I had hoped about research on that, which is probably very hard. - Or not? Couldn't you look at the n-gram distributions of the data and look how much that is "random", i.e. are people avoiding clusters harder than they should?
This is what surprised me the most. It's admittedly been a long time since I took a stats class, but the trend line hardly looks like a trend. There hardly seems to be any correlation at all on either of the two charts with trend lines
They are equally likely, yes. But, the questions here pertain to whether or not an other person would perceive your sequence as random.
It wasn't too long ago there was an article on HN (can't find it now) describing what feels random to people.
Essentially, a sequence feels more random the harder it is to explain.
So, HHHHHHHH doesn't feel random because it's summarized as 8 H's and HTHTHTHT doesn't feel random either because it is HT repeating. Strings that are only really communicable by repeating the string verbatim feel the most random.
But if this is about perceived randomness shouldn’t people also guess which sequence was created by a human vs sequences created by some algorithm?
I understand that to score randomness you probably would create n-grams of the characters and look if these would be equally distributed. But for such short sequences it feels hard to do. Maybe a statistician can explain this?
For me, using my right thumb on a smartphone seems enough to skew the randomness. Just by doing it again with my index finger (after writing most of this comment), I raised it from being “more random” than 18% to 84%.
'Perceived as random' seems like a pretty junk measure of other humans' efforts at producing randomness. Garbage in, garbage out. Surely analysing this tells you absolutely nothing about anything?
I would understand if the actual measurement is not perceptual but mechanised, i.e. "how small can we compress this stream of random choices using our best known compression methods" or something. (But then a stream of 10 symbols is surely not enough to show you the humans.)
It gets pretty theoretical, but basically it's estimating the Kolmogorov complexity by looking at the size of the Turing machine required to generate a particular string, rather than Shannon entropy or implementations of common compression techniques.
> But if this is about perceived randomness shouldn’t people also guess which sequence was created by a human vs sequences created by some algorithm?
It's about what the test person think another human will perceive as random so there's a layer of indirection there. If these guesses you suggest would help the study or not I really can't say.
Yes the visual layout and input method is an interesting bias. If you imagine drawing a line between all choices on the number pad (which I do), I realised my answer avoided doubling-back on itself. My 'random' shape looked more like a nicely distributed squiggle, which is less random.
The measure of randomness chosen in this particular paper appears to be an approximation of the Kolmogorov-Chaitin complexity adapted for small integer/binary sequences.
This effectively looks at how easy it would theoretically be to compress/describe the data, for instance HHHHHHHHHH would be low complexity as it could be encoded as '10 H's'.
If something is truly random, it shouldn't be possible to encode it due to the pigeon-hole principle.
> If something is truly random, it shouldn't be possible to encode it due to the pigeon-hole principle.
This statement is obviously untrue.
“Random numbers” don’t really exists. The original authors were right about that. Every number/sequence is equally likely to occur. There’s even an XKCD about this [0].
I guess what you mean is: If you have a process that generates sequences randomly, most of those sequences are expected to compress badly.
This is a key point in the article. The researchers (original) did not want to throw out the data that doesn't "seem random", this group argues in favour of doing so.
> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just "made up" by somebody
This explanation leads me to think that the decision of what is random in the study is based on human perceptions of randomness, not actual statistical randomness. Although any sequence is equally (un)likely to be rolled, 1111111111 would stand out from the other sequences much more than 3156263441.
No, they're just saying that no person who was following the instructions would produce a sequence of all heads or all tails as a "random" sequence, so they're throwing out those two specific sequences.
Look at the graph of responses. There are a few clear clusters and lines outside of the main, statistically random cluster. Those other ones can be dropped.
> Sure, a probability of having many tails continously is low but how is it not random?
Randomness is never about the output; it is a property of the source.
So what is not random, is that the frequency of people that output repeating sequences is higher than the quotient of that sequence among all possible sequences of the same size.
The issue of the initial paper is that it claims to conclude that people aged > 60 are less random, when they simply output a fully-repeating sequence with a slightly higher rate (which is not surprising taking into consideration that there are much, much fewer of them).
P(all heads | bad faith) ≫ P(any given "random"-looking string | bad faith)
therefore
P(bad faith | all heads) ≫ P(bad faith | any given "random"-looking string)
So if you want to exclude bad faith responses, the best strategy (by Neyman–Pearson, if you want to think of it that way) is to remove "all heads" responses (and similarly for "all tails").
Very cool website. Kind of "someone is wrong on the internet" crazy levels of effort! Really I think you can see that there's no trend just by looking at the graph. Always be suspicious of graphs that look like they have no trend by eye but have a solid trend line superimposed.
Would be nice if they defined "complexity" somewhere. I think the sequence lengths are too short to distinguish true random number generators from poor random number generators.
In other words what kind of graph would you see if you used a real coin? Although I guess if you sample over enough people then it doesn't matter.
My idea of what random actually looks like has been affected a lot by generating random numbers with a computer. They just don’t actually look that random.
I read an anecdote about the iPod shuffle (hey kids - it was a music player with no screen so you could not choose songs directly) - they initially set it to be genuinely random in the way it chose the next song - people didn’t like it. It didn’t _feel_ random to pick a song you only just listened to again. So they had to make an algorithm that was sort-of-random but with some constraints to make it feel how we expect randomness to be.
I mean, that makes sense. What I want when putting a music player to "shuffle" is not "give me something unpredictable" -- that's fundamentally what randomness means. What I want is "give me something new". Something new is not something random, it's something _different_ from before, if reasonably possible.
Which to a machine may as well be the same thing in either phrasing. You want something different from what you just listened to. To it, anything not 'that song' is different and 'new' potentially if also not 'just listened to' within a certain set amount of songs. Even without that certain set of songs being logged and considered; any picking of a different song from the last is verifiably random.
Think of it all like a deck of cards. Shuffle is apt in that sense. You don't expect to see double aces each time you pick through the shuffled deck of cards, but sometimes you do. Sometimes, you also find double jacks, queens and kings; in a row. Sometimes you don't. That deck could be shuffled by the worlds best trick shufflers. Still gonna get doubles now and then.
True Randomness is not really technically possible. At least, not with our current technologies available; and we have a lot of aces up our sleeves.
The best we can manage for randomness right now, is creating random strings of numbers to serve as the seed for new randomness. At least, if I understand correctly. If I do, then this is why cryptography is so damn important for us in the computational side of things. Network Security requires randomness.
If I understand you correctly, I think you missed my point. You're explaining how with true randomness, you get different stuff most of the time and the same stuff some of the time. That is true. But it's not what people want when they press shuffle. What people want is something _different_, and giving the same song twice is not something different. As another commenter wrote, giving multiple (different) songs after each other from the same album would even be undesirable, even if that could occur perfectly well with random shuffling.
You would expect a shuffled deck of 52 unique cards. Not a deck of three 5 of spades. Likewise with a playlist: if I shuffle a playlist of 52 songs, I want those 52 songs to be played in a random order. Not for a random song to be played each time but a random shuffle of that list.
Not just that though. If you have an iPod filled with 20 albums from your favorite artist and 1 album from 5 others, you wouldn’t be happy even with random excluding previous.
The iPod shuffle thing wasn't really about randomness, it was a UX failure. "To shuffle" means a specific thing. If I ask you to shuffle a deck of cards and give it to me so I can draw them one by one, I very much don't expect you to put each card back in the deck and re-shuffle it before every time I draw a new card.
In casual language, random means not "uniformly random", but something more like "without a discernable pattern". Playing a song from the same album is the start of a discernable pattern.
I used to be a game designer, and I worked on a lot of games with randomness mechanics and I analysed a lot of player feedback. How people at large perceive randomness is NOT what randomness is, of course. A task to create something that is random, and a task to create something that people perceive is random are two very different tasks.
I've had to deal with a similar issue for a product. We ended up summarising it in a fuzzy way that people's minds have a notion of 'micro' and 'macro' randomness.
In the case the coin flip, all heads or all tails is perfectly fine and will happen in macro random, ie if you took macro to mean > 1 million rolls let's say. But in micro random (the 12 rod we experience in real time) if that were to happen we'd feel uncomfortable and immediately assume it was cheated even if it was a product of true randomness because macro random suffers the same problems as very large numbers of slices of time
Spotify had a blogpost about it back in the day[1]. It was based on prior work that's a bit more general[2]. The basic idea was that you don't really want to randomize, you want to distribute.
Those regression lines are absolutely laughable. Point cloud has line going through it, and beforehand I have no clue where the line will be (other than that it will go through the center, presumably). Even if there is a statistically significant effect, its meaning will be a rounding error.
I understood my task as convincing another _human_ that it was randomly generated. Since N was low for each of the tasks, I was deliberate about sometimes having repeated values, and not ensuring that every option was picked an equal number of times, since that looks suspiciously algorithmic. Apparently I'm over 60.
I got the same answer as you, over 60, the first time as I was also very deliberate then went back and did it again like a 3 year old, jabbed anywhere and got a higher random result. Maybe there is something to the study?
Picking evenly seems to consistently produce higher "randomness" scores than picking unevenly or using an RNG. I wonder how this algorithm would rank random sequences vs shuffled linear sequences.
The fundamental issue with a randomness metric for sequences is that an idealized independent generator will under-perform vs a constrained generator that excludes low scoring sequences.
As I understood it, their entire point is that younger people are not better at generating random sequences than older people, so guessing someone's age based on their complexity (randomness) score is completely unreliable.
Towards the bottom of the page they said they've only guessed age correctly 51% of the time, which lines up with there being no correlation between age and ability to generate random sequences
My point isn't the age result that I mentioned. (I believe their claim that it's bogus.) It's that the instruction to click "as randomly as possible" is ambiguous so at best they're measuring an average of the behaviours they think they are.
They are not, these are the instructions from the reproduction
> Tap a sequence of 10 dice rolls. Make it look as random as possible; another person should not be able to tell if you made it up or if it was from real dice rolls.
And this is the excerpt from the study they mention
> Click on a number between one and six as randomly as possible to produce the kind of sequence you'd get if you really rolled a die [...]
I made the same mistake as thombles, the new instructions make it sound like the objective is to trick a human. The original clearly states the objective is to be random.
They are not the same objective, as humans are terrible at recognizing randomness.
> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody.
I have a really hard time rationalizing why you would leave that part out of your quote and drew the conclusion you did. The original task was clearly also about creating patterns that a human would recognize as random.
For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to think that such a sequence is a likely random sequence. Therefore, the replicating authors are fully justified to eliminate those poor faith answers. The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant.
This is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this.
I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter.
You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data.
But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.
It seems to me that if we reject a subset of experimental samples because they look like bad data (e.g. extreme outlier caused by sensor malfunction) we are still keeping all the bad data we are unable to recognize as such (e.g. sensor malfunctions producing less extreme data), which introduces a bias.
Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.
It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis.
At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.
I probably should have clarified that I was responding to the content of the parent comment rather than the submission itself.
I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic.
Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it.
It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis.
You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.
Since two people have come to the same misunderstanding, I must have worded my argument inadequately.
Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study.
But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.
If you're going to exclude bad faith answers, I think you should exclude all of them. But I don't think you can do that. Is HTHTHTHTHT a bad faith answer? Always or only sometimes? We're trying to infer the test subject's intent from their answer, and that's fundamentally impossible I think.
I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less random'. If old test subjects show more bad faith, I think it's not really wrong to say older people are less random. And it does have predictive power.
Arbitrarily (because they is no way to do it subjectively) excluding some answers and not others has, I think, a greater risk of skewing the results.
However, the study concluded that person's ability to produce randomness peaks at 25. An increase in showing bad faith doesn't tell us anything about the ability to produce randomness if desired. Thus, if we accept the bad faith answers as part of the data, the conclusion of the study becomes incorrect, at least in wording.
It’s hard to believe the original authors made their argument in good faith. They probably ran the numbers with the filters and saw they wouldn’t have a paper that way.
> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected .
Yeah... you thought I was 60. Seems from the comments this is a common thing.
You might want to check your algorithms. But then again, you do say in the end of the results that you need more 60+ year olds to help make this more accurate.
Also, a bone to pick. You claim that people get less random as they get older over 25, with 25 being the peak. I would wonder if maybe that has some correlation with the brain finally fully developing from adolescents into true/full adulthood. (Remember folks, we do call people 'young adults' for a while in their 20's.)
Also, while you make that claim about people as they get older, I still managed to get a coin flip result that was 13% more random than others at my age group of 33. Or something like that. I forget how it was worded exactly off hand this moment without going and checking again in my history. My point here is this.
If randomness declines with age past 25, but my score at age 33 is 13% more random than others in the same age range; then is it truly declining for everyone equally or is it just some people more rapidly than others?
I think this maybe correlates potentially with the findings of the trend disappearing once the non-random data is removed. (The all heads/all tails results.)
Anyways. With all this said, I do agree you need some more participants above the age of 60. Have you considered using facebook at all?
Did we read the same article? They aren't claiming those things at all. Those are the claims of the original study, which are being disputed by this attempt at reproduction. The writers suspect those claims to be false due to the choice of the original study to not remove likely intentionally non-random data.
I believe the 13% stat you saw is that your score had a higher random score than than 13% of other participants, so not very random.
> As for our initial idea to make an age-guessing game, we have guessed right 51% of the time. Pretty much what we had expected .
I don’t know what the “guess” was for others. But for me it guessed “are you under 60?” If that’s what it’s doing guessing above or below 60, then I think it’s amazing they, only getting 51% right. I would expect that a strategy of ignoring data completely and always guessing under 60 would be significantly better.
Very interesting, and I think this part sums up the crux:
“The researchers believe that you can only analyze the raw responses because, statistically, any sequence is equally likely to occur, so where do you draw the line?”
I’d say that, as it is a psychological study, making claims about a human behavior, treating humans as pure random number generators without considering _intent_ is a mistake.
It is entirely possible that older people fill in more “questionable” responses because they can’t be bothered with the study, and that this causes the “decline” in ability as people age.
But we don’t know for sure, because it was never investigated. Thus, the biggest problem is the original study not even bringing this into light, even though it appears the original authors were aware of it.
One thing many surveys/studies do is to include "trap" questions (I'm sure there's a real name for 'em) which disqualifies any participants that answer them incorrectly.
I think they are being overly kind. The conclusion itself shouldn’t be sensitive to just removing the all H if all T answers. Since their trend disappears from removing just those answers, were only left with the far more mundane “older people are more likely to write all Hs or Ts”. The true conclusion was hidden by the averaging that goes on when you make a best fit line.
The more surprising part to me was that quote about the 2015 study that tried to reproduce 100 studies and managed to replicate 39. That's honestly way better than I expected for psychology.
I feel like a lot of the comments here are written after only taking the test and many are not reading the rest of the article.
The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.
But their data doesn't make sense to be personally...
Only 5% of their dataset is above the age of 60, making their claim that they are getting 50% of their guesses wrong seem like they are calculating it wrong. Surely their cut-off should be at the 95th percentile of the data?
They shouldn't be guessing 'under 60' the same proportion of times as 'over 60', because their population is mostly under 60.
Again though, they are arguing that there is no correlation between randomness and age. This was just a demonstration that when they use randomness to predict age, the results are wrong 50% of the time-- which is precisely in accordance with their hypothesis
Yeah but their guess shouldn't be wrong 50% of the time as again that means that they can’t have picked the 95th percentile result! Because it’s 50:50 I’ll assume that they are assigning people scoring higher than average the “under 60” category - which is obviously incorrect. Otherwise how do they pick the cut off?
To explain with another example - let's say that I have a dataset of 100 people's scores at golf (no handicaps) and I know that 5% of them are pro-players and others are 'advanced amateurs'. Because of this I might take the top 5 scores and guess that they are pro's and assign the others the guess of 'advanced amateur'.
Now let's say that there was actually no correlation between people's scores at golf and their 'pro' status - what accuracy would I expect in the above experiment? The answer is actually closer to 90% 'accurate guesses' than 50%! (Although obviously - that's 90% accurate based on random chance).
Now if someone told me they got 50% of the guesses wrong at this task, that implies that they guessed that the top 50% of those golfers were pro rather than picking the top 5% of scores, and I would question the methodology.
This % is similar to the dataset in the webpage - I downloaded it, filtered out exclusions and c4% of the valid responses are 60 or over.
If I inherently pick a small population (i.e. over 60's are c4% in this dataset) and I am guessing wrong 50% of the time, it means that my cut-off is incorrectly calibrated. Their score cut-off should, at worst, be picking the wrong 4% and missing another 4%.
Am I going crazy? It seems logical to me, but to be open maths isn't my strong point. I just know that if I designed the guessing rule, I would be getting more than 50% (my algorithm would be 'if the users average score across the three tests is less than -1.5, assign 'over 60' and that would get c95% accurate guesses, albeit it would still not prove anything and I agree with the authors overall premise!).
In your golf example, making that guess requires an additional knowledge of what "pro" means and it's frequency among golfers. The data doesn't know that just like the randomness data doesn't know that most humans are younger than 65 years old. If you really want to figure out how predictive the data is, you shouldn't include considerations like that in your model. I get what you're saying but ultimately I don't think their goal was to make the most accurate prediction, they wanted to make one that illustrated their point by basing their guess off the data alone.
The calculation involves knowing the age of the sample population though (if you don’t know the ages of your sample, how do you work out what the cut off is at 60 years?).
If I don’t know how many golfers are pro, I simply cannot estimate if it is 100 golfers that are pro or 0 (unless it’s a real gap in scores). Making an assumption that 50 are pro is no more valid than 0 or 100.
If you take the average score of 100 people and say that you estimate anyone scoring below the average is above 60, you are going to be wrong regardless of if your hypothesis is valid or not.
Putting that up and saying “see, it’s wrong 50% of the time!” doesn’t make sense when your calculation is incorrect.
In order to calculate the cut-off correctly they either need to take the 95th percentile result, or pick a sample where 50% of people are over-60 and 50% are under 60 and take an average of that.
Using a dataset where 95% of people are under 60 and then picking the mean clearly isn’t going to work.
I'd have read it if it weren't white text on a pink background. I'm not going through the trouble of pulling it up in a browser and undoing what they presumably did on purpose. Then to complain that people don't read the whole thing?
It picked me as under 60 though I am actually over. Does it give any more granular age guesses or is it stuck on over 25, under 60, over 60? I may retake this in the morning to see if anything changes for me or for the test.
360 comments
[ 4.6 ms ] story [ 345 ms ] threadThey guessed I was 'Under 60', which I am, but over 50% of people fit into the category of under 60... so the fact they can guess this with only 50% of accuracy doesn't really feel right?
(i.e. there is no statement that the 51% figure is age adjusted)
I'm probably just being silly here...
"As for our initial idea to make an age-guessing game, we have guessed right NaN% of the time. Not as good as we had expected ."
There is suspicious clustering in the plot, at complexity slightly lower and higher than -1. Complexity -1 is also more sparse than seems reasonable from the shape/density of the cloud.
It's maybe not so counter-intuitive as the Birthday paradox, but maybe already showing how bad our intuition is at grasping randomness. Initially I had hoped about research on that, which is probably very hard. - Or not? Couldn't you look at the n-gram distributions of the data and look how much that is "random", i.e. are people avoiding clusters harder than they should?
It wasn't too long ago there was an article on HN (can't find it now) describing what feels random to people.
Essentially, a sequence feels more random the harder it is to explain.
So, HHHHHHHH doesn't feel random because it's summarized as 8 H's and HTHTHTHT doesn't feel random either because it is HT repeating. Strings that are only really communicable by repeating the string verbatim feel the most random.
I understand that to score randomness you probably would create n-grams of the characters and look if these would be equally distributed. But for such short sequences it feels hard to do. Maybe a statistician can explain this?
For me, using my right thumb on a smartphone seems enough to skew the randomness. Just by doing it again with my index finger (after writing most of this comment), I raised it from being “more random” than 18% to 84%.
I would understand if the actual measurement is not perceptual but mechanised, i.e. "how small can we compress this stream of random choices using our best known compression methods" or something. (But then a stream of 10 symbols is surely not enough to show you the humans.)
It gets pretty theoretical, but basically it's estimating the Kolmogorov complexity by looking at the size of the Turing machine required to generate a particular string, rather than Shannon entropy or implementations of common compression techniques.
It's about what the test person think another human will perceive as random so there's a layer of indirection there. If these guesses you suggest would help the study or not I really can't say.
This effectively looks at how easy it would theoretically be to compress/describe the data, for instance HHHHHHHHHH would be low complexity as it could be encoded as '10 H's'.
If something is truly random, it shouldn't be possible to encode it due to the pigeon-hole principle.
See http://www.scholarpedia.org/article/Algorithmic_complexity
This statement is obviously untrue.
“Random numbers” don’t really exists. The original authors were right about that. Every number/sequence is equally likely to occur. There’s even an XKCD about this [0].
I guess what you mean is: If you have a process that generates sequences randomly, most of those sequences are expected to compress badly.
[0]: https://xkcd.com/221/
Could someone explain why putting sequences of same choice is invalid?
Sure, a probability of having many tails continously is low but how is it not random?
This explanation leads me to think that the decision of what is random in the study is based on human perceptions of randomness, not actual statistical randomness. Although any sequence is equally (un)likely to be rolled, 1111111111 would stand out from the other sequences much more than 3156263441.
It's not that the single values sequences are invalid, it's that they're occurring with a greater frequency than by chance
The players are just trying to sound smart basically
Randomness is never about the output; it is a property of the source.
So what is not random, is that the frequency of people that output repeating sequences is higher than the quotient of that sequence among all possible sequences of the same size.
The issue of the initial paper is that it claims to conclude that people aged > 60 are less random, when they simply output a fully-repeating sequence with a slightly higher rate (which is not surprising taking into consideration that there are much, much fewer of them).
P(all heads | bad faith) ≫ P(any given "random"-looking string | bad faith)
therefore
P(bad faith | all heads) ≫ P(bad faith | any given "random"-looking string)
So if you want to exclude bad faith responses, the best strategy (by Neyman–Pearson, if you want to think of it that way) is to remove "all heads" responses (and similarly for "all tails").
Would be nice if they defined "complexity" somewhere. I think the sequence lengths are too short to distinguish true random number generators from poor random number generators.
In other words what kind of graph would you see if you used a real coin? Although I guess if you sample over enough people then it doesn't matter.
I read an anecdote about the iPod shuffle (hey kids - it was a music player with no screen so you could not choose songs directly) - they initially set it to be genuinely random in the way it chose the next song - people didn’t like it. It didn’t _feel_ random to pick a song you only just listened to again. So they had to make an algorithm that was sort-of-random but with some constraints to make it feel how we expect randomness to be.
Think of it all like a deck of cards. Shuffle is apt in that sense. You don't expect to see double aces each time you pick through the shuffled deck of cards, but sometimes you do. Sometimes, you also find double jacks, queens and kings; in a row. Sometimes you don't. That deck could be shuffled by the worlds best trick shufflers. Still gonna get doubles now and then.
True Randomness is not really technically possible. At least, not with our current technologies available; and we have a lot of aces up our sleeves.
The best we can manage for randomness right now, is creating random strings of numbers to serve as the seed for new randomness. At least, if I understand correctly. If I do, then this is why cryptography is so damn important for us in the computational side of things. Network Security requires randomness.
A human pressing "shuffle" usually doesn't want randomness. They want pleasing _variation_. See e.g. the "Comparison" heading here: https://blog.demofox.org/2017/10/20/generating-blue-noise-sa...
In the case the coin flip, all heads or all tails is perfectly fine and will happen in macro random, ie if you took macro to mean > 1 million rolls let's say. But in micro random (the 12 rod we experience in real time) if that were to happen we'd feel uncomfortable and immediately assume it was cheated even if it was a product of true randomness because macro random suffers the same problems as very large numbers of slices of time
[1]: https://engineering.atspotify.com/2014/02/how-to-shuffle-son... [2]: http://keyj.emphy.de/balanced-shuffle/
Those regression lines are absolutely laughable. Point cloud has line going through it, and beforehand I have no clue where the line will be (other than that it will go through the center, presumably). Even if there is a statistically significant effect, its meaning will be a rounding error.
The fundamental issue with a randomness metric for sequences is that an idealized independent generator will under-perform vs a constrained generator that excludes low scoring sequences.
Towards the bottom of the page they said they've only guessed age correctly 51% of the time, which lines up with there being no correlation between age and ability to generate random sequences
> Tap a sequence of 10 dice rolls. Make it look as random as possible; another person should not be able to tell if you made it up or if it was from real dice rolls.
And this is the excerpt from the study they mention
> Click on a number between one and six as randomly as possible to produce the kind of sequence you'd get if you really rolled a die [...]
I made the same mistake as thombles, the new instructions make it sound like the objective is to trick a human. The original clearly states the objective is to be random.
They are not the same objective, as humans are terrible at recognizing randomness.
> so that if another person is shown your sequence of digits from 1 to 6, he/she should not be able to tell whether these numbers were produced by a real die or just “made up” by somebody.
I have a really hard time rationalizing why you would leave that part out of your quote and drew the conclusion you did. The original task was clearly also about creating patterns that a human would recognize as random.
For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to think that such a sequence is a likely random sequence. Therefore, the replicating authors are fully justified to eliminate those poor faith answers. The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant.
Edited for clarity.
I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter.
You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data.
But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.
Original hypothesis: Old people are worse at giving random responses.
Alternative hypothesis: Old people are more likely give bad faith responses.
This review is suggesting the AH is equally good at explaining the data as the OH.
At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.
I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic.
Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it.
It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis.
You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.
Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study.
But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.
Agree with your post, but don't believe that the authors paper made this claim unless I am missing something?
I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less random'. If old test subjects show more bad faith, I think it's not really wrong to say older people are less random. And it does have predictive power.
Arbitrarily (because they is no way to do it subjectively) excluding some answers and not others has, I think, a greater risk of skewing the results.
Yeah... you thought I was 60. Seems from the comments this is a common thing.
You might want to check your algorithms. But then again, you do say in the end of the results that you need more 60+ year olds to help make this more accurate.
Also, a bone to pick. You claim that people get less random as they get older over 25, with 25 being the peak. I would wonder if maybe that has some correlation with the brain finally fully developing from adolescents into true/full adulthood. (Remember folks, we do call people 'young adults' for a while in their 20's.)
Also, while you make that claim about people as they get older, I still managed to get a coin flip result that was 13% more random than others at my age group of 33. Or something like that. I forget how it was worded exactly off hand this moment without going and checking again in my history. My point here is this.
If randomness declines with age past 25, but my score at age 33 is 13% more random than others in the same age range; then is it truly declining for everyone equally or is it just some people more rapidly than others?
I think this maybe correlates potentially with the findings of the trend disappearing once the non-random data is removed. (The all heads/all tails results.)
Anyways. With all this said, I do agree you need some more participants above the age of 60. Have you considered using facebook at all?
I believe the 13% stat you saw is that your score had a higher random score than than 13% of other participants, so not very random.
I don’t know what the “guess” was for others. But for me it guessed “are you under 60?” If that’s what it’s doing guessing above or below 60, then I think it’s amazing they, only getting 51% right. I would expect that a strategy of ignoring data completely and always guessing under 60 would be significantly better.
Hypothesis: the older a person is, the more likely they are to understand that HHHHHHHHH is a statistically likely outcome from true randomness.
https://www.incibe-cert.es/sites/default/files/blog/comproba...
“The researchers believe that you can only analyze the raw responses because, statistically, any sequence is equally likely to occur, so where do you draw the line?”
I’d say that, as it is a psychological study, making claims about a human behavior, treating humans as pure random number generators without considering _intent_ is a mistake.
It is entirely possible that older people fill in more “questionable” responses because they can’t be bothered with the study, and that this causes the “decline” in ability as people age.
But we don’t know for sure, because it was never investigated. Thus, the biggest problem is the original study not even bringing this into light, even though it appears the original authors were aware of it.
Here is the paper: https://www.nature.com/articles/nature.2015.18248
The authors of the website are stating that they believe the study is wrong. The below/above 60 answer is showing you it’s incorrect half of the time along with data backing up the claim.
> we decided to reduce our experiment to three tasks because of attention spans (not yours, it is exceptional if you are reading this).
Only 5% of their dataset is above the age of 60, making their claim that they are getting 50% of their guesses wrong seem like they are calculating it wrong. Surely their cut-off should be at the 95th percentile of the data?
They shouldn't be guessing 'under 60' the same proportion of times as 'over 60', because their population is mostly under 60.
To explain with another example - let's say that I have a dataset of 100 people's scores at golf (no handicaps) and I know that 5% of them are pro-players and others are 'advanced amateurs'. Because of this I might take the top 5 scores and guess that they are pro's and assign the others the guess of 'advanced amateur'.
Now let's say that there was actually no correlation between people's scores at golf and their 'pro' status - what accuracy would I expect in the above experiment? The answer is actually closer to 90% 'accurate guesses' than 50%! (Although obviously - that's 90% accurate based on random chance).
Now if someone told me they got 50% of the guesses wrong at this task, that implies that they guessed that the top 50% of those golfers were pro rather than picking the top 5% of scores, and I would question the methodology.
This % is similar to the dataset in the webpage - I downloaded it, filtered out exclusions and c4% of the valid responses are 60 or over.
If I inherently pick a small population (i.e. over 60's are c4% in this dataset) and I am guessing wrong 50% of the time, it means that my cut-off is incorrectly calibrated. Their score cut-off should, at worst, be picking the wrong 4% and missing another 4%.
Am I going crazy? It seems logical to me, but to be open maths isn't my strong point. I just know that if I designed the guessing rule, I would be getting more than 50% (my algorithm would be 'if the users average score across the three tests is less than -1.5, assign 'over 60' and that would get c95% accurate guesses, albeit it would still not prove anything and I agree with the authors overall premise!).
If I don’t know how many golfers are pro, I simply cannot estimate if it is 100 golfers that are pro or 0 (unless it’s a real gap in scores). Making an assumption that 50 are pro is no more valid than 0 or 100.
If you take the average score of 100 people and say that you estimate anyone scoring below the average is above 60, you are going to be wrong regardless of if your hypothesis is valid or not.
Putting that up and saying “see, it’s wrong 50% of the time!” doesn’t make sense when your calculation is incorrect.
In order to calculate the cut-off correctly they either need to take the 95th percentile result, or pick a sample where 50% of people are over-60 and 50% are under 60 and take an average of that.
Using a dataset where 95% of people are under 60 and then picking the mean clearly isn’t going to work.
It picked me as under 60 though I am actually over. Does it give any more granular age guesses or is it stuck on over 25, under 60, over 60? I may retake this in the morning to see if anything changes for me or for the test.