Wonder how many people here immediately knew what the rule was?
This is the standard example of to right way to test a hypothesis/theory and the power of Confirmation Bias, testing sequences that are invalid under the theory instead of testing what you think is correct.
I'm curious to see more about the distribution of questions and answers people had, and how the HN population may differ from the NYT's. There will certainly be self selection bias here, but if you're willing to share how you did with others, please enter it here:
https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
People familiar with unit testing and test driven development will feel at home with this kind of puzzle. That doesn't mean that they will be less biased in social/political decisions, it just means that this test will fail to prove a point.
Seriously, though, it seems a bit of leap from the existence of confirmation bias to explaining away the public outpourings of US Politicians about their financial crises and foreign policy disasters - in the absence of better data as to just why the given statements were made ascribing this to confirmation bias seems itself open to accusations of confirmation bias! :)
That's exactly what I was thinking. I (sometimes) follow TDD, and I applied it to this problem. I made sure to include negatives, 0, positives, and include primes here or there to help avoid issues with multiplication/exponentiation. After a few of these, I felt pretty confident that the rule was simple.
To talk about a certain fraction of real numbers you have to have a distribution over them. In general we take the uniform distribution if no distribution is explicitly given. That doesn't work for real numbers (it doesn't even work for natural numbers). (See https://math.stackexchange.com/questions/14777/why-isnt-ther...)
If there's no implicit default distribution, we have to pick on. I can pick one where they cover an arbitrary high percentage of real numbers..
Down the rabbit hole of pedantry: we don't need a distribution, just a measure, if we want to talk about how many reals it accepts. The Lebesgue measure is implied on the Reals if none is given, and the computable reals have measure zero.
We can't reasonably talk about a percent coverage, since the Lebesgue measure of the reals is infinite, but as a non-technical description, 'zero percent' is morally equivalent to saying it only covers a measure-zero set.
Based on the context of the question and the UI of the testing interface, fractions seem unlikely to be an intended part of the question. I likewise wouldn't bother testing unicode U+216x roman numerals.
It refused to accept both fractions and imaginary numbers. I did test negatives and zeros since that was really the only remaining set I could think of.
Most importantly I used about 6 tests (3 right 3 wrong) to come up with the answer and then did another 17 looking for the trick. After all, it couldn't just be that simple right?
The presentation of the problem -- and I know that trusting the problem state is unwise sometimes in cognitive-bias tests, since many such tests are actually designed to be "we said we were asking X but actually meant Y" -- indicated a simple rule, rather than one which would behave differently on different classes of numbers.
So after the tests listed above I felt confident enough to guess.
I tried floating point numbers. Also, at 28 decimal places, the test breaks; it's not arbitrary precision. So, technically, the answer isn't simply "any ascending sequence of numbers".
I actually avoided going that far to avoid getting bad data. I was trying to answer the question, "What does the experimenter THINK his rule is?" rather than what will the computer do. Since the computer can't be infinite, it will inevitably fail with overflow, underflow, and such.
I was relieved, in fact, when it worked with negatives and floats in a "safe" range.
I also tested with 1,1,2 and 1,2,2 to make sure that the required increase applied to ALL of the values, not just a specific pair.
I too tried to test if it was only one pair that was significant. However I grew impatient and didn't try to come up with more tests when I thought I had a sufficient answer to explain my most vexing observation (negative, positive, positive out of combinations involving negative numbers).
The observation to brainstorm for ways of proving that a statement is in fact wrong, and exhausting them, is such an eloquent way of wording the hunt for a negative.
The slightly-shorter 0.60000000000000000, 0.60000000000000001, 0.60000000000000002 will break it too, for what it's worth. If you punch those in to a Javascript console, you can see that the FP representation of all three is 0.6.
Yes I did, though not to 28 places as tptacek had done. With 2 to 4 place floats, including negatives, the test worked as expected.
Like others here have said it wasn't a particularly hard "rule" to figure out. Easy to immediately rule out geometric relationship as in 2^y, which didn't leave a whole lot of possibilities to test. For the commenters here, I'd attribute ease of finding the solution to familiarity with the kinds of problems that programming presents.
Which leads to the idea there's value in learning even the rudiments of programming. Logically, it should encourage better problem-solving skills in general. We might think there this has important implications for our educational systems. But I know, that's probably not realistic at all.
I know the question isn't directed at me, but I thought you might want to know. I couldn't try negatives or floating point on the iPhone. The keypad didn't have the option.
I was just writing this out! I believe practice in unit testing is what let me get this question right. I almost submitted n^1, n^2, n^3 before I realized I didn't get any wrong answers and something didn't feel right. I credit this 'instinct' to having written thousands of tests, and trying to make them fail to make sure my tests weren't lying to me.
It's funny. Even though I got the right answer, looking back, I still see many holes in my tests. I only tested positive and negative integers. I didn't even think to try fractions, decimals, hex values, text, etc. and I was already expecting a confirmation bias test.
I mean, it also illustrates how training and systematic reasoning can improve these things. Whether explicitly or implicitly, I picked up certain skills and procedures for problem solving (from programming and math contests) that I now use by default. Trying a bunch of examples, coming up with a hypothesis, trying to disprove it, testing edge cases…
This doesn't mean I always use these—at the very least, I have to explicitly jump into "problem solving" mode—but it means they can be useful.
It's still a meaningful difference, and could very well apply to lots of things beyond this kind of puzzle.
Exactly. I'm just as 'No' averse as the next person but the 'No' I'm averse to is the one where you make wrong assumptions and it comes back to haunt you afterwards.
Your last question is "have you seen a test about confirmation bias before?" But the text of this test says that it's about "why no-one likes to be wrong", which means it's pretty obviously about confirmation bias (and therefore that the test-taker should be wary about just confirming their first intuition).
Intersting that the split of correct/wrong answers from the HN crowd is 78%/22% - the exact opposite of the general population : 22% / 78%! The HN community does think different :)
In the sense that they guessed the rule correctly. The correct answer was the dominant hypothesis in my mind before I tried any sequences, and others here report the same.
It just means that this test was described at least several times in different "computer" media over last few years and almost everyone has read about it.
Same thing with all "logic" puzzles.
It's probably not "fair" to say I got it in zero... but I did. :)
Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.
Similar story here. I got it in zero because this problem shows up at the early part of HPMOR.
I suspect that the basic idea behind it is about right (people who insist on failures before committing to a theory will probably do "better"). But it seems to me that this test will be best at selecting people who've seen it before and can pretend they didn't (or even remember to ask negative questions when someone asks you to guess three numbers to get the job).
Yeah, I'd like to see other sorts of questions that you can only get right by looking for disconfirming evidence, but that don't have anything to do with choosing a sequence of numbers.
3 9 27 yes (is it exponential series?)
4 16 64 yes (is it only odd numbers?)
5 7 9 yes (is it any numbers of the same parity?)
6 7 8 yes (is it any set of increasing numbers?)
6 7 6 no (just to confirm that it's x<y<z, and not something like x<=y<=z)
My answer was: "The sequence is of an increasing real variable, where each subsequent value is greater than the preceding value. It's monotonically varying."
When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not.
Although I used the wrong term, it's strictly increasing.
There's a selection bias - those of us who got it right are more likely to fill that out (:
As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.
It would be relevant to include a histogram of #yes_answers - #no_answers in the summary, to test whether people are biased towards positive rather than negative tests. I think the raw data suggests that it does although I totally failed to create a histogram in google spreadsheets within 5 minutes.
I've given the test to various people since then and never once came across someone who'd guess it right away, or within a short period of time. The breaking time came usually after a few minutes when they gave up and started throwing out random numbers that coincidentally did not meet the rule. Once you hit the first "No", it took a very short time to figure out the rule for almost everyone.
The official answer to this puzzle makes a huge assumption: that there is one correct answer. There is not one correct answer. (x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer, at least as automatically checkable, and there's an infinite number of such tuples. To find a "no" you're reduced to random guessing. That's not a puzzle, that's crap. The confirmation bias material might be true, but the puzzle does not illustrate it.
The puzzle is not "Find a rule that matches everything you tested". It's figure out what rule they are using. It may be true that there are multiple ways to state the rule, such as "x_2 >= x_1 + 1 and x_3 is >= x_1 + 2" but they are equivalent.
Try the sequence: 0,0,0. It would give "yes" for (x,2x,4x), but the actual rule gives it "no".
Your task is to learn a class of objects by example. Without seeing any negative examples, you're unlikely to learn the right class (which is unique, it is precisely the class of increasing sequences of length 3). (x, 2x, 4x) is not "a correct answer", as it does not describe the class you were supposed to learn to distinguish.
Well, yes, in this situation random guessing would be very likely to provide an immediate check on the solution.
(1x, 2x, 4x), as indicated in the video below, is not sufficient. It represents a subset of the values that are valid.
Think of it this way. When asked to write a unit test, do you only test the positive outcomes? No, you test to make sure the failures are as you expect as well. Otherwise, you are likely to have what you think is a failure end up as a success.
> (x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer
I think this logic is a bit wonky - if there are sequences that get a "yes", but don't match (x, 2x, 4x) then the correct rule cannot be (x, 2x, 4x), can it?
It's amusing that you went to a website on confirmation bias, did the puzzle incorrectly, presumably read the material on confirmation bias, but still suffer from the effects of confirmation bias.
Amusingly human, if I may add. Reading "Thinking, fast and slow", by Daniel Kahneman, one key idea that I got was that even knowing against biases you are very, very likely to suffer from those biases. Disheartening results were gathered from studies done on well-trained psychologists and people prepared for the experiment, to no avail. Can't remember the details right now, but just read the book, it's awesome. Another good one was "Influence" by Cialdini, but they gave you tips on trying to avoid those biases that, upon reading Kahneman, I don't think anymore that are very useful.
From a Computational Learning Theory perspective, we are faced with an infinite hypothesis space, with an infinite VC dimension. So yes, there's not much strategy that can be employed here.
But, the fact that there is one correct answer is not really an assumption that the puzzle makes, it is information we have been given:
> We've chosen a rule that some sequences of three numbers obey -- and some do not.
This simply means that the solution is realisable. No matter how many ways (in English) we have to describe that solution it is still the same solution.
I've read this comment three times and I still can't figure out what you're trying to say. It sounds like you're arguing that all "puzzles" should exist on finite domains so that a brute force solution exists.
"(x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer, at least as automatically checkable."
Well, no: you can type 1, 2, 3 into the system and it will tell you "yes", but your rule says that it should tell you "no".
It is crucial to the definition of "correct answer" here that your rule should not just say "yes" only for tuples which the widget also says yes to, but also your rule should say "no" only for tuples which the widget also says no to. That is what the puzzle means when it's asking, "can you guess the rule that we've created?"
This makes it very, very different from what I think you're thinking about, which is situations where someone tells you, "what is the next number in this sequence? 4, 7, 13, 25, ...?" where technically there are an infinite number of rules which will generate those 4 numbers first and an arbitrary number afterwards. Technically one of them is "simplest" in the sense that it can be expressed in 7 symbols, but in general it's a complicated problem and there is no best solution.
"To find a 'no' you're reduced to random guessing. That's not a puzzle, that's crap."
In many ways it still is a puzzle but the space that it lives in is richer. If you think about typical "puzzles" they're things like: "here's a grid with some spaces filled in with numbers,
Each number is a block in a block wall. We want you to turn this into a block maze so that each 'block wall' (set of blocks connected by adjacency) contains exactly one numbered-block whose number says how many total blocks are in the wall. Furthermore the path (non-block space) of the block maze should be connected and should not contain any 'rooms' -- that is, any 2x2 or larger segments of open space."
This 7x7 grid has 10 spaces which are known to be blocks and exactly 12 more blocks scattered in the remaining 39 spaces, so just by those factors alone we're searching only (39 choose 12) ~= 3.91 billion possibilities; we can also use a quick heuristic to identify 6 places which must be "space" to break apart adjacent numbered walls, removing 91% of that search space.
The puzzles, "I have a set of integers where inclusion in the set is governed by a short rule, you can ask me any integer and I will tell you whether it is in my set", by contrast, have an infinite search space. This means that any solution is going to be more interesting, as will the means for checking that solution's validity. You could require, for instance, a Haskell expression of 140 characters or fewer which turns a nonnegative Int named `n` into a Bool, to be judged as "valid" or "invalid" if it properly filters `[1..10000000] :: [Int]`. You could even give the first 100 numbers in the set, e.g.:
In this case that's pretty much enough to see the general pattern; the verification covers 10 million bits while the 140-character limit probably limits your search space to 1000 bits or so, so it's going to...
The idea isn't to come up with tuples that satisfy the predicate. The idea is to figure out what the predicate is in the first place.
Also, you're not in any way, shape, or form reduced to random guessing. If you have an idea of what the rule might be, you build a counterexample. There's a ton of value in _trying_ to get a no but getting a yes instead.
It's not an assumption; the article tells you up front that there is one correct answer.
It is an analog for how science works. When it comes to a natural phenomenon, humans can come up with multiple explanations that fit a given set of observations, but presumably (I mean, this is a basic tenet of science) nature only works in one consistent way.
Thus, the importance of a falsifying test. You form a hypothesis based on the initial observations (in this case, the number sequence 2, 4, 8), and then you propose a test that could falsify your hypothesis.
The trick is that a hypothesis can fail in several ways. It can be outright wrong, like saying "the rule is that the numbers decrease from left to right." That's obviously just wrong.
But it can also be too specific, like saying "the rule is that the exponent increments by one with each step to the right." That matches the given evidence, and tests with other base numbers will succeed too. But it's over-fitting.
Here's a concrete example: a man wearing a red shirt drops a weight and measures gravitational acceleration as 9.8 m/s^2. So he formulates a hypothesis that gravity always produces an acceleration of 9.8 m/s^2 in the presence of a red shirt.
And if he always wears a red shirt, and always tests gravity on the surface of the Earth, he'll always find supporting evidence for that hypothesis.
But of course we know that gravitational acceleration varies depending on mass and distance, and that it's the same no matter what color your shirt is. But he would only find that out if he varied his experiment beyond what his hypothesis predicts.
With your example it's easy to lose the distinction between "would eventually terminate if you had a fast computer and a lot of time" and "never terminates even in theory." Here, it definitely looks like it should always run two iterations to matter what the numbers are (as long as they're finite), but it doesn't.
Edit: I think this is the code that actually reads the numbers the user enters, see [0]
function l(){
var a=h.exec(m[1]),f=null,g=null,n=null;
return a&&(null!==a[1]&&a[1]&&(f=parseInt(a[1],10)),
null!==a[2]&&a[2]&&(g=parseInt(a[2],10)),
null!==a[3]&&a[3]&&(n=parseInt(a[3],10))),
new e(f,g,n)
}
Edit(2): Actually, I'm not so sure that's the correct code at all. They NYT game is capable of parsing floats correctly (e.g. it accepts 1.1, 1.2, 1.3 as a "Yes") so it's not just using parseInt.
var rightWrong = (inputData[0] < inputData[1]) & (inputData[1] < inputData[2]) ? right : wrong;
With a variable declaration on line 545 being
var inputData = [NaN, NaN, NaN],
revealed = false,
right = "<p class = 'g-answer g-yes'>Yes!</p>",
wrong = "<p class = 'g-answer g-no'>No.</p>";
And `inputData` is changed on text input on line 662
$("#g-input input").each(function(i) {
var val = $(this).val();
inputData[i] = $.isNumeric(val) ? Number(val) : NaN;
});
It uses the `Number()` function to convert from the input text to an actual number, so it can convert any number format defined by ES5[1] or ES6[2]. So in ES6 you can use binary (0b, 0B) and octal (0o, 0O) formatting along with exponential (1e-2) and hex (0x, 0X). Binary and octal works for me currently on Chrome 43 OS X.
* The number may have optional sign and digits after a decimal point, and may use exponential notation. Example: (-1.2e1, .0E+0, 1.e-3) => "Yes". As seen in the second and third number here, there may be no digits before or after the decimal point, but both at the same time (i.e., ".0" and "0." parse but not ".").
* If the number begins with "0x" or "0X" it is read in hexadecimal, where the digits a-f may be in either case. Hexadecimal notation must not be accompanied by decimal point, sign, or exponential notation.
* No whitespace is permitted within the numeral, even between the sign and the digits as in "+ 11", but both tabs and spaces may be used before and after the numeral without changing its value. In particular, by using a input of the form "1 " it is possible to make rectangular display empty while still parsing it as number. Note that pressing "Check" leads to the numbers being displayed in the rectangle in exactly the same way as they were displayed in the text box, which may depend on the position of the cursor in the text box.
ETA: Also, you mentioned rounding, but there is also exponent overflow and underflow. The application refuses to parse numbers greater or equal to 1.7976932e308. It parses arbitrary negative exponents fine, but it does not recognize that 1e-324 is greater than 0.
A test engineer walks into a bar. He orders a beer. He orders two beers. He orders 999999999 beers. He orders 1.00001 beers. He orders -42 beers. He orders 1048576 beers...
The purpose is not to see if people use randomized inputs, or even if they're capable of analyzing data. It is to see if people are willing to challenge their own notions. The "misleading" sample is then just right.
Even in real life, no discrete categorization hypothesis should be accepted if you've only seen data matching one of the categories.
>In order to prove their point effectively without falling in the same trap they are pointing, they should conduct the same experiment with a random example each time, not one especially created to mislead the experimentee.
This is the point though - we all have preexisting beliefs, and we often don't critically question those beliefs. The [x,2x,4x] pattern injected into our minds by nytimes is playing the part of the preexisting belief. The fact that most people go forward with their theory without properly testing it is the confirmation bias. Totally randomizing the numbers wouldn't be able to make this point because three random numbers usually won't have a clear looking pattern (and hence wouldn't give us an obvious preexisting belief).
> In such problems, it provides more information to test your hypothesis with data matching your hypothesis than with data not matching it.
I'd argue this is often not the case. For example, in the article's Iraq example, looking for evidence that there were no WMDs would have cost very little compared to even a short war. Instead, Bush's administration searched for positive evidence anywhere they could, even from questionable sources.
There is no penalty to being told "no". That also applies to "I think I know the answer". There is no penalty for being told "no" there as well, so once you have a reasonable guess why not check it? Disagree with their analysis.
There is an implied penalty because it says about the answer box "Make sure you’re right; you won’t get a second chance" in contrast to "You can test as many sequences as you want" about the sequences.
This is absolutely correct. Perhaps Hacker News culture is such that they realize such a warning can be overcome through a refresh, or possibly an incognito browser window if a cookie is preventing a second guess. Still though, within the rules of the game, there is no drawback of testing negative sequences, but there's a clearly defined drawback of entering an incorrect answer.
Zendo is fantastic. Watching people unaccustomed to its sort of problem-solving struggle with and improve their methodologies for being good at the game is really enlightening.
I guessed correctly with only 2 nos. Since there is no penalty for guessing incorrectly here, I felt safe enough with my theory. I might have checked for more nos, if I had to announce my theory publicly (Twitter, comment, etc). However, I also knew about Confirmation Bias beforehand.
You can enter all kinds of crazy random sequences which only have The Rule in common and get a yes, which seemed to be enough assurance. If you're trying to get it to say "no" but failing, is that still confirmation bias? Doesn't sound like it.
I also guessed correctly with only 2 nos -- my real confirmation came from the fact that "-1, 0, 300000000" (or some number of 0s) was correct, meaning it couldn't be any really meaningful sequence.
It's surprising how far you can go by such heuristics. I run an IRC bot that uses matches like this (I'm working on a proper solution right now though) to parse natural language queries and I managed to trick few people into thinking they're talking with human. As long as it's ok for 90% of most common cases, people often won't notice.
I've made a robot that screams[0]. That is, just outputs a random string of "AAAAaaa" when its name is mentioned or when somebody else screams (four or more A's).
What is surprising is how basically 15 lines of python implementing these rules, invokes a very real emotional response in a lot of people :-)
It only means non-decreasing in the context of a monotonic function, where the definition, I believe, is that the derivative of the function is never <0.
"We’ve chosen a rule that some sequences of three numbers obey — and some do not. Your job is to guess what the rule is."
My mistake was to assume that choosing the first number uniquely defines the next ones in the sequence. Since, you know, like all the sequence puzzles I've seen before worked like that, and I didn't read it rigorously enough. Oh, by the way, the doubling thing is wrong if you use negative numbers (wrong as in it gives false positives, instead of just false negatives). But the problem definition doesn't even tell what set of numbers we're operating on.
Finding the rule the sequences obey is impossible since it could be that all cases follow a simple rule except for one triplet which you're unlikely to find. It's trivially easy to fool the user into finding a wrong rule.
Same here - I was also guilty of feeling a bit clever and avoiding the obvious "oh, the numbers double every time" answer. So, when I found out that if the spelling of each number was one letter longer (4 = four letters, 8 = five letters, 11 = six) I got smug, and didn't bother testing further.
That was a hypothesis I considered, as was x,2x,4x. Invalidated both of those, and ultimately drew the intended conclusion (which is not quite "correct", given javascript numeric precision issues).
Most people will make assumptions about what's required based on previous experience. And hardly anyone will have previous experience where a question formatted in this way isn't asking you to find a series rule.
That's not quite the same thing as confirmation bias. With a bias you're just as likely to discount significant evidence as you are to mismodel the problem space.
It is true that, in general, this quiz is impossible to guess, for the reasons you explain.
But in this case -- which is the main point of the article -- it was actually a trivial rule. No tricks, no special cases. The real purpose wasn't for you to find the actual rule, but to learn about your biases. The only trick here lies in the human mind, and its tendency to validate patterns (and claim an early victory) instead of trying to refute them.
Yeah I thought it would be something like "Numbers must strictly increase unless the first number is 15326". Then the point of the article would be that some government rules are not well defined.
I got the part about increasing eventually, but I thought the third number also had to be the sum of the other two. I came up with the sum idea after trying (3 6 9), so only 2 tests. The idea that they had to be increasing came later. I don't have it open but I'm pretty sure one of my tests was (1 2 5) which should have tipped me off... in conclusion yes, I'm probably dumb.
I know you're joking, but I think this is important:
Failing the test does NOT mean a person is dumb. The point of the article is that confirmation bias seems to be a fundamental default in the way everyone thinks. Certain people with specialized training in inductive problem solving (scientists etc.) have learned to compensate.
If folks think it's an issue of intelligence, then they might be willing to think "but not me, because I'm smart." (After all, many programmers believe that they are smarter than the average bear). But while programmers are well-trained to think carefully about sequences of numbers, they might be as susceptible as anyone else to confirmation bias in other areas.
I wonder how much games like Twenty Questions plays into conditioning towards this kind of approach, since the implication there tends to be that the fewer questions you ask, then the better you've done.
Like the other comment said, this is not about being dumb. It's about human psychology: you'd rather not be wrong, so you jump at the possibility of being right, and "forget" to take the logical step of actively trying to disprove your rule (1 2 5 can be disregarded because you weren't trying to disprove the rule).
Putting your engineer hiring notices under the hood is becoming a common "Easter-egg" practice. It's also a form of targeted recruitment advertising - the only people who see it are your target audience.
The attached reading material is interesting, but this question is too similar to problems where you guess the next one in the sequence and none are missing.
A rule where the numbers are increasing does not explain why 3 or 5 or 6 is missing from the sequence in that version of the question that is much more common.
Having just re-read HPMOR a few weeks ago, I could answer it right away, but there was actually a difference from HPMOR's version, which required three positive increasing numbers, while this allowed negatives.
Neat - as others have pointed out, I feel that being familiar with unit testing would help in this situation. Having negative test cases is just as important, if not moreso, than having "happy path" tests.
I entered A, B, C. The form didn't let me submit the sequence, so they are doing some verification that the entries are numbers. Rational numbers work, though: 1.1, 1.2, 1.3.
I think that the "quick" adjective in the title is purposeful misleading. You are supposed to learn quickly the most general rule, but that is not so easy because there are many possible rules that could fit such a pattern. It seems that you should be rewarded for solving the puzzle quickly and then you fall in the trap.
I propose to change the title to "A puzzle to test your Generalization Abilities", and state clearly that you should try to find the most general rule that satisfies all patterns you can think of. In that case, I would expect the conclusion and results of the experiment to be completely different. So to summarize: the so "quick" adjective in the title has a very strong anchor effect.
Edit: changed for grammar and to express more clearly what I think.
Maybe it should be "a puzzle many people already know the answer to" in which case the conclusion and results are already obviously biased.
I was able to solve the puzzle without testing any numbers at all. Which really skews the relevance of "only nine percent of people saw three 'no's before answering."
I "got lucky" in the sense that my experiences have primed me to recognize that particular kind of question. If the article had actually modified the question framing at all, rather than just copying the existing one, it maybe wouldn't have worked that way.
It's the same reaction I have to the Monte Hall problem: I don't have to think or be clever to get the right answer. So naturally you couldn't effectively teach me anything by simply posing the question, and any information you collect by doing so won't accurately reflect what I learn or how well I think. You'd just be testing topical familiarity.
The question could have read the exact same way, with a different final rule as the answer. Without testing at all, you wouldn't/couldn't know that...and so you didn't solve it..you just guessed...
310 comments
[ 3.9 ms ] story [ 273 ms ] threadThis is the standard example of to right way to test a hypothesis/theory and the power of Confirmation Bias, testing sequences that are invalid under the theory instead of testing what you think is correct.
The result summary is visible here: https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...
The raw answers are visible here: https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1J...
Personally, I spent most of my answers playing with the inputs. The form was happy to report that "1e1, 15, 0x10" was a valid sequence. :D
Seriously, though, it seems a bit of leap from the existence of confirmation bias to explaining away the public outpourings of US Politicians about their financial crises and foreign policy disasters - in the absence of better data as to just why the given statements were made ascribing this to confirmation bias seems itself open to accusations of confirmation bias! :)
At that point, I correctly answered the question.
But it accepts zero percent of real numbers.
To talk about a certain fraction of real numbers you have to have a distribution over them. In general we take the uniform distribution if no distribution is explicitly given. That doesn't work for real numbers (it doesn't even work for natural numbers). (See https://math.stackexchange.com/questions/14777/why-isnt-ther...)
If there's no implicit default distribution, we have to pick on. I can pick one where they cover an arbitrary high percentage of real numbers..
We can't reasonably talk about a percent coverage, since the Lebesgue measure of the reals is infinite, but as a non-technical description, 'zero percent' is morally equivalent to saying it only covers a measure-zero set.
Most importantly I used about 6 tests (3 right 3 wrong) to come up with the answer and then did another 17 looking for the trick. After all, it couldn't just be that simple right?
So after the tests listed above I felt confident enough to guess.
I was relieved, in fact, when it worked with negatives and floats in a "safe" range.
I also tested with 1,1,2 and 1,2,2 to make sure that the required increase applied to ALL of the values, not just a specific pair.
The observation to brainstorm for ways of proving that a statement is in fact wrong, and exhausting them, is such an eloquent way of wording the hunt for a negative.
Like others here have said it wasn't a particularly hard "rule" to figure out. Easy to immediately rule out geometric relationship as in 2^y, which didn't leave a whole lot of possibilities to test. For the commenters here, I'd attribute ease of finding the solution to familiarity with the kinds of problems that programming presents.
Which leads to the idea there's value in learning even the rudiments of programming. Logically, it should encourage better problem-solving skills in general. We might think there this has important implications for our educational systems. But I know, that's probably not realistic at all.
4,6,8 Y 1,1,1 N 1,2,3 Y 1,6666,8777 Y 1,0,1 N 3,2,1 N 3,2,3 N 5,6,4 N 7,5,6 N
But I totally see why it would cause you to pause and rethink your original idea :).
This doesn't mean I always use these—at the very least, I have to explicitly jump into "problem solving" mode—but it means they can be useful.
It's still a meaningful difference, and could very well apply to lots of things beyond this kind of puzzle.
I don't specifically recall seeing one, but it is likely that I have.
FWIW, I've seen this kind of game before and I was expecting it to be something simple.
> Remarkably, 78 percent of people who have played this game so far have guessed the answer without first hearing a single no.
Some of those 78% probably got it right, and some of the remainder would have got it wrong.
In what sense? How did they know that e.g. (1, 1, 1) didn't work, unless they tried it?
How many "I know the answer, but can I find a flaw in their code" questions did you ask?
It's probably not "fair" to say I got it in zero... but I did. :)
Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.
I suspect that the basic idea behind it is about right (people who insist on failures before committing to a theory will probably do "better"). But it seems to me that this test will be best at selecting people who've seen it before and can pretend they didn't (or even remember to ask negative questions when someone asks you to guess three numbers to get the job).
When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not.
Although I used the wrong term, it's strictly increasing.
As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.
Right? What answer did you give?
Yes, double the previous number
Yes, Each number much larger than the previous
Yes, sequence must always increase by 1
Yes, Powers of 2
Yes, h = 2n, i = 2(n+1), j = 2(n+2) . n is an integer
Yes, Powers of 2
8 4 2 (no) 1 2 3 (yes) 1 1 1 (no) 1 100 123 (yes) 1.0 1.1 1.2 (yes)
answer: incrementing numbers
If anyone's tallying
If anyone's tallying
Could you perhaps move to a bar chart instead of pie charts?
I've given the test to various people since then and never once came across someone who'd guess it right away, or within a short period of time. The breaking time came usually after a few minutes when they gave up and started throwing out random numbers that coincidentally did not meet the rule. Once you hit the first "No", it took a very short time to figure out the rule for almost everyone.
Try the sequence: 0,0,0. It would give "yes" for (x,2x,4x), but the actual rule gives it "no".
(1x, 2x, 4x), as indicated in the video below, is not sufficient. It represents a subset of the values that are valid.
Think of it this way. When asked to write a unit test, do you only test the positive outcomes? No, you test to make sure the failures are as you expect as well. Otherwise, you are likely to have what you think is a failure end up as a success.
I think this logic is a bit wonky - if there are sequences that get a "yes", but don't match (x, 2x, 4x) then the correct rule cannot be (x, 2x, 4x), can it?
1, 3, 5, 7
what comes next? 9 right? Or is the sequence generated by 2n − 1 + (n − 1)(n − 2)(n − 3)(n − 4) for n ∈ N. Then we've got 33.
"among all hypotheses consistent with the observations, the simplest is the most likely"
33 is correct, but it's less likely to be the basis for the generation of the sequence.
Your answer of (x, 2x, 4x) proves the puzzle illustrates the confirmation bias, at least in your case.
Does the unit test that confirms your function returns the expected result given one set of arguments prove it correct?
Related: read The Checklist Manifesto.
Not true. What if x is negative?
But, the fact that there is one correct answer is not really an assumption that the puzzle makes, it is information we have been given:
> We've chosen a rule that some sequences of three numbers obey -- and some do not.
This simply means that the solution is realisable. No matter how many ways (in English) we have to describe that solution it is still the same solution.
"(x, 2x, 4x) gives you a "yes" every time, therefore it is a correct answer, at least as automatically checkable."
Well, no: you can type 1, 2, 3 into the system and it will tell you "yes", but your rule says that it should tell you "no".
It is crucial to the definition of "correct answer" here that your rule should not just say "yes" only for tuples which the widget also says yes to, but also your rule should say "no" only for tuples which the widget also says no to. That is what the puzzle means when it's asking, "can you guess the rule that we've created?"
This makes it very, very different from what I think you're thinking about, which is situations where someone tells you, "what is the next number in this sequence? 4, 7, 13, 25, ...?" where technically there are an infinite number of rules which will generate those 4 numbers first and an arbitrary number afterwards. Technically one of them is "simplest" in the sense that it can be expressed in 7 symbols, but in general it's a complicated problem and there is no best solution.
"To find a 'no' you're reduced to random guessing. That's not a puzzle, that's crap."
In many ways it still is a puzzle but the space that it lives in is richer. If you think about typical "puzzles" they're things like: "here's a grid with some spaces filled in with numbers,
Each number is a block in a block wall. We want you to turn this into a block maze so that each 'block wall' (set of blocks connected by adjacency) contains exactly one numbered-block whose number says how many total blocks are in the wall. Furthermore the path (non-block space) of the block maze should be connected and should not contain any 'rooms' -- that is, any 2x2 or larger segments of open space."This 7x7 grid has 10 spaces which are known to be blocks and exactly 12 more blocks scattered in the remaining 39 spaces, so just by those factors alone we're searching only (39 choose 12) ~= 3.91 billion possibilities; we can also use a quick heuristic to identify 6 places which must be "space" to break apart adjacent numbered walls, removing 91% of that search space.
The puzzles, "I have a set of integers where inclusion in the set is governed by a short rule, you can ask me any integer and I will tell you whether it is in my set", by contrast, have an infinite search space. This means that any solution is going to be more interesting, as will the means for checking that solution's validity. You could require, for instance, a Haskell expression of 140 characters or fewer which turns a nonnegative Int named `n` into a Bool, to be judged as "valid" or "invalid" if it properly filters `[1..10000000] :: [Int]`. You could even give the first 100 numbers in the set, e.g.:
In this case that's pretty much enough to see the general pattern; the verification covers 10 million bits while the 140-character limit probably limits your search space to 1000 bits or so, so it's going to...The idea isn't to come up with tuples that satisfy the predicate. The idea is to figure out what the predicate is in the first place.
Also, you're not in any way, shape, or form reduced to random guessing. If you have an idea of what the rule might be, you build a counterexample. There's a ton of value in _trying_ to get a no but getting a yes instead.
It is an analog for how science works. When it comes to a natural phenomenon, humans can come up with multiple explanations that fit a given set of observations, but presumably (I mean, this is a basic tenet of science) nature only works in one consistent way.
Thus, the importance of a falsifying test. You form a hypothesis based on the initial observations (in this case, the number sequence 2, 4, 8), and then you propose a test that could falsify your hypothesis.
The trick is that a hypothesis can fail in several ways. It can be outright wrong, like saying "the rule is that the numbers decrease from left to right." That's obviously just wrong.
But it can also be too specific, like saying "the rule is that the exponent increments by one with each step to the right." That matches the given evidence, and tests with other base numbers will succeed too. But it's over-fitting.
Here's a concrete example: a man wearing a red shirt drops a weight and measures gravitational acceleration as 9.8 m/s^2. So he formulates a hypothesis that gravity always produces an acceleration of 9.8 m/s^2 in the presence of a red shirt.
And if he always wears a red shirt, and always tests gravity on the surface of the Earth, he'll always find supporting evidence for that hypothesis.
But of course we know that gravitational acceleration varies depending on mass and distance, and that it's the same no matter what color your shirt is. But he would only find that out if he varied his experiment beyond what his hypothesis predicts.
That also means that for(i=0;i<j;++i){} doesn't necessarily terminate for an arbitrary j smaller than infinity, which I find hilarious.
Either way, though, it is definitely hilarious!
Nedit: One of the other responses nailed it. IEEE standards on 64 bit fp ops
[0] http://a1.nyt.com/assets/interactive/20150612-151638/js/foun...
on line 588 is the comparison
With a variable declaration on line 545 being And `inputData` is changed on text input on line 662 It uses the `Number()` function to convert from the input text to an actual number, so it can convert any number format defined by ES5[1] or ES6[2]. So in ES6 you can use binary (0b, 0B) and octal (0o, 0O) formatting along with exponential (1e-2) and hex (0x, 0X). Binary and octal works for me currently on Chrome 43 OS X.[0] http://graphics8.nytimes.com/newsgraphics/2015/06/16/puzzle/...
[1] http://www.ecma-international.org/ecma-262/5.1/#sec-9.3.1
[2] http://www.ecma-international.org/ecma-262/6.0/#sec-7.1.3.1
* The number may have optional sign and digits after a decimal point, and may use exponential notation. Example: (-1.2e1, .0E+0, 1.e-3) => "Yes". As seen in the second and third number here, there may be no digits before or after the decimal point, but both at the same time (i.e., ".0" and "0." parse but not ".").
* If the number begins with "0x" or "0X" it is read in hexadecimal, where the digits a-f may be in either case. Hexadecimal notation must not be accompanied by decimal point, sign, or exponential notation.
* No whitespace is permitted within the numeral, even between the sign and the digits as in "+ 11", but both tabs and spaces may be used before and after the numeral without changing its value. In particular, by using a input of the form "1 " it is possible to make rectangular display empty while still parsing it as number. Note that pressing "Check" leads to the numbers being displayed in the rectangle in exactly the same way as they were displayed in the text box, which may depend on the position of the cursor in the text box.
ETA: Also, you mentioned rounding, but there is also exponent overflow and underflow. The application refuses to parse numbers greater or equal to 1.7976932e308. It parses arbitrary negative exponents fine, but it does not recognize that 1e-324 is greater than 0.
Even in real life, no discrete categorization hypothesis should be accepted if you've only seen data matching one of the categories.
This is the point though - we all have preexisting beliefs, and we often don't critically question those beliefs. The [x,2x,4x] pattern injected into our minds by nytimes is playing the part of the preexisting belief. The fact that most people go forward with their theory without properly testing it is the confirmation bias. Totally randomizing the numbers wouldn't be able to make this point because three random numbers usually won't have a clear looking pattern (and hence wouldn't give us an obvious preexisting belief).
> In such problems, it provides more information to test your hypothesis with data matching your hypothesis than with data not matching it.
I'd argue this is often not the case. For example, in the article's Iraq example, looking for evidence that there were no WMDs would have cost very little compared to even a short war. Instead, Bush's administration searched for positive evidence anywhere they could, even from questionable sources.
In my logs most people seem to have gotten it right, though presumably that's because it was linked from LessWrong.
For an actually-fun game like this, see https://en.wikipedia.org/wiki/Zendo_%28game%29.
No surprise there, given that this experiment is discussed in Sequences, the link to the post present in the article you link to :).
I guessed correctly with only 2 nos. Since there is no penalty for guessing incorrectly here, I felt safe enough with my theory. I might have checked for more nos, if I had to announce my theory publicly (Twitter, comment, etc). However, I also knew about Confirmation Bias beforehand.
You can enter all kinds of crazy random sequences which only have The Rule in common and get a yes, which seemed to be enough assurance. If you're trying to get it to say "no" but failing, is that still confirmation bias? Doesn't sound like it.
var probablyWrong = ["doubl", "expon", "multipl", "^", "", "power", "two", "2", "twice", "as big", "nth", "rais"];
var seemsRight = ["larger", "increas", "greater", "small", "less", "big", ">", "<", "go up", "ascending"], weaselWords = ['but ', 'not ', 'odd'];
Been expecting something more interesting than that
What is surprising is how basically 15 lines of python implementing these rules, invokes a very real emotional response in a lot of people :-)
I named it "Wilhelm".
[0] inspired by this comic http://gunshowcomic.com/513
The sequence is not monotonically increasing.
It only means non-decreasing in the context of a monotonic function, where the definition, I believe, is that the derivative of the function is never <0.
http://www.amazon.com/Mistakes-Were-Made-But-Not/dp/14915141...
My mistake was to assume that choosing the first number uniquely defines the next ones in the sequence. Since, you know, like all the sequence puzzles I've seen before worked like that, and I didn't read it rigorously enough. Oh, by the way, the doubling thing is wrong if you use negative numbers (wrong as in it gives false positives, instead of just false negatives). But the problem definition doesn't even tell what set of numbers we're operating on.
Finding the rule the sequences obey is impossible since it could be that all cases follow a simple rule except for one triplet which you're unlikely to find. It's trivially easy to fool the user into finding a wrong rule.
Good article - and a humbling experience. :)
Most people will make assumptions about what's required based on previous experience. And hardly anyone will have previous experience where a question formatted in this way isn't asking you to find a series rule.
That's not quite the same thing as confirmation bias. With a bias you're just as likely to discount significant evidence as you are to mismodel the problem space.
While that's true, the real thing being tested is the user's readiness to prove themselves wrong.
But in this case -- which is the main point of the article -- it was actually a trivial rule. No tricks, no special cases. The real purpose wasn't for you to find the actual rule, but to learn about your biases. The only trick here lies in the human mind, and its tendency to validate patterns (and claim an early victory) instead of trying to refute them.
Were you testing a pre-supposed hypothesis that confirmed itself?
Failing the test does NOT mean a person is dumb. The point of the article is that confirmation bias seems to be a fundamental default in the way everyone thinks. Certain people with specialized training in inductive problem solving (scientists etc.) have learned to compensate.
If folks think it's an issue of intelligence, then they might be willing to think "but not me, because I'm smart." (After all, many programmers believe that they are smarter than the average bear). But while programmers are well-trained to think carefully about sequences of numbers, they might be as susceptible as anyone else to confirmation bias in other areas.
A rule where the numbers are increasing does not explain why 3 or 5 or 6 is missing from the sequence in that version of the question that is much more common.
So, after all, I think I fall in the trap of confirmation bias that the sequence must consist of numbers only.
>We’ve chosen a rule that some sequences of three numbers obey
>Now it’s your turn. Enter a number sequence in the boxes below, and we’ll tell you whether it satisfies the rule or not.
After that, I went with doubling and it worked 3 times with various sizes of number, so I went with that.
Double bonus points if you can guess what I was testing with the first sequence (which the given numbers do satisfy).
But, in case no-one guesses, the answer is (rot13):
erirefr nycunorgvpny beqre
https://youtu.be/vKA4w2O61Xo
Edit: changed for grammar and to express more clearly what I think.
I was able to solve the puzzle without testing any numbers at all. Which really skews the relevance of "only nine percent of people saw three 'no's before answering."
It's the same reaction I have to the Monte Hall problem: I don't have to think or be clever to get the right answer. So naturally you couldn't effectively teach me anything by simply posing the question, and any information you collect by doing so won't accurately reflect what I learn or how well I think. You'd just be testing topical familiarity.