Edit: I just had a look at the median word count for adults who took the survey. It's around 27,000. I wonder whether that's true or not.. it seems to me that I'm lacking.
Perhaps the most interesting part of how they're doing it is that they have an algorithm that doesn't rely on the results of other test takers, except for the eventual statistics.
I got 25,600 but that was below the median of 27,123. I didn't select words I recognized but couldn't define and I double checked some definitions. I think I have a decent vocabulary. It's like I'm competing with the High IQ Society book club.
You need a test where the definition is part of the test. Otherwise anyone can just check boxes.
Just from getting some friends to do this it seems to me the median score overall and the median score for each age are a bit inflated. Just my thoughts, but I think people aren't being 100% truthful. Although I may just have a poor vocabulary http://testyourvocab.com/?r=36208
Ah, that's why dictionary.com didn't know what "splarge" meant. Apparently "sparge" is a sprinkling - why would home brewers know this? Genuinely curious here.
My result[1] isn't too bad but there's still quite a few words I can learn especially at the end. I did cheat a little since I learned a few of the more curious looking ones when this showed up on /b/ 2 nights ago.
Obvious point : The only people interested in finding out their scores will be the kind of people who think their vocab is something worth competing on. There's no way this is a fair sample across all English-speakers.
I wouldn't say the ONLY people. This could be the next 'IQ test' where it seems the near opposite, the lesser intelligent ones are who are obsessed with them.
Going by magazines, common forwards and so on it seems the average man loves a good self-quiz.
But to remove some of the bias, we'd need approximately equal numbers of above and below average people to be going to the trouble of answering the quiz.
And in the back of my mind, I thought that the bulk of quiz magazines were sold to women, no? (Though I'm probably being too pedantic...)
I will be a lot more interested in the Brazilian site when it goes live! I'm not making any particular effort to expand my English vocabulary (having a smaller vocabulary is not among the top ten reasons my writing in English is worse than Shakespeare's) but my Portuguese vocabulary is muito terrivel. It would be great to have a way to measure my progress.
Edit: Oh. The Brazilian site is just a Portuguese version of the instructions for the English site? :(
I only scored 24k which seems low based on the statistics at the end. I also only selected words that I absolutely knew the definition of, even though some I think I knew based on the root.
Memorizing trivia words is just something that has never interested me. Instead I keep a thesaurus and dictionary handy at all times :)
ObScore: 36000, 32yo male US-english native speaker. There were a few more I'd seen but wasn't really sure enough to define. I maxed out the SAT verbal section (and got 1 wrong on math, which was enough to drop to 780/800) back in the 1990s.
And yet not a slythy tove in sight, gyring or otherwise.
The thing that always surprises me with this sort of test is how many words I half-know: I recognise the word, I have a (correct) general sense of what it means, but either I couldn't articulate an accurate definition or the definition I would give is an uncommon usage and I hadn't come across the more common meaning(s).
Following the rules strictly (so I didn't tick the half-known words), I came in slightly below average for my age, assuming the trend continues beyond the 32 years that their table currently gives.
For comparison, if I also included the words I half-knew, I gained nearly 4,000 new words and went up about 20%.
What difference does it make? The site doesn't say what it means in everyday life. I'm guessing if you exclude high achieving SAT vocab nerds, it finds the difference between people who care about the meaning of each word and people who will guess through context because they have no patience for a dictionary. Or people who don't read fancy texts, like the Scarlett Letter for example, after failing to read that I stopped reading books.
Not sure I buy the results though. I would think that the rate of increase would start to decrease quite significantly after high school/college but it appears to stay pretty much linear throughout the data.
Site creator here -- That has been the most interesting finding so far -- I certainly didn't expect it. Unfortunately I haven't had enough participation yet from children, or older folks :), to see how it continues to extend in either direction.
I got 37,300. They claim this is not quite 95th percentile, which I am a tad skeptical accurately represents my vocabulary-size percentile relative to the general population. Perhaps this survey is being forwarded around unusually literate people at the top end, or more than 5% of responders are cheating. Where are the fake words to catch cheaters? I Googled a lot of what I didn't recognize, and everything I checked was real.
Let's think about it logically. If you were to actually test the users instead of asking them 'which of these do you know?', the ONLY option is multiple choice since analysing text-field input from the user to determine if their definition is correct is at best extremely difficult, and more than likely impossible ... right ?
I agree that the quality of the result is highly dependent on the honesty
of the person tested, but with multiple choice, wouldn't you be able to
deduct the meaning of the word from choices you are given?
E.g.: One of the words I encountered in the test was 'terpsichorean'. While I knew that Terpsichore is one of the Muses, I did not know which one and left the box unticked. Had there been multiple choices, I might have guessed the correct solution.
True, but it's still much better than just relying on people submitting accurate results without even testing them ... at least with multiple choice everyone is being tested to approximately the same metric, and you can get a more accurate percentile
To my defense, they are both derived from 'deducere', to lead away, and 'were not distinguished in sense until the mid 17th cent' according to my system’s dictionary
Actually, the SAT also employs multiple choice, and quite successfully. You see, multiple choice can not only give you hints, but can also be used to lead you astray. In the end, it balances out.
I ticked 'terpsichorean', because it made me think "dreamy travelogue writing of a scenic beach with either terpsichorean sky or sea, it means X looks a pale shade of blue-green".
Google tells me it means dancing so I was way way off (maybe conflating turquoise and cerulean?).
But I have no way of knowing how many words that I feel comfortable defining are actually nowhere near correct, so to be any kind of accurate, they need to do some verification of correctness. All 'honesty' means is 'don't deliberately cheat' not 'don't be dumb'.
I had "discomfit" as one of my words that I wasn't clear of the definition of, I was pretty close when I looked it up but couldn’t have guaranteed it. It's probably easily confused with discomfort ... which made me think that this needs to be a little more tested. Commonly misread words could easily inflate scores.
However, I think a multiple choice test could also inflate scores unless the definitions were very cunningly constructed.
I scored 75-80th percentile (32,800) which surprised me. It seems quite a lot of words, for one. For another I consider my vocab' to be very good and I don't think I'm being bigheaded in that. Ergo I expected to be ranked higher.
On the second page there was an entire column of words of which I recognised only three sufficiently to provide a guaranteed accurate definition. One of that column was terpischorean, another tatterdemalion.
Whilst looking up tatterdemalion I found little use of it after the 1930s except as a proper noun (a Marvel Comics character for example). What I did find however is that Google Books is useless for finding dates. One citation from an author Sir Edward Bulwer Lytton is given a date of 1999. That's a reprint date, the author died in the 19th century.
Eliezer, I think you appreciate some of the ideas behind the FAQ I'll repost here with adaptation to the current situation:
VOLUNTARY RESPONSE POLLS
As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make such an inference. That's because the data are not from a random sample of the relevant population. One professor of statistics, who is a co-author of a highly regarded AP statistics textbook, has tried to popularize the phrase that "voluntary response data are worthless" to go along with the phrase "correlation does not imply causation." Other statistics teachers are gradually picking up this phrase.
-----Original Message----- From: Paul Velleman [SMTPfv2@cornell.edu] Sent: Wednesday, January 14, 1998 5:10 PM To: apstat-l@etc.bc.ca; Kim Robinson Cc: mmbalach@mtu.edu Subject: Re: qualtiative study
Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.
I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll
is a spectacular historical example of a voluntary response poll with a HUGE sample size and high response rate that didn't give a correct picture of reality at all.
When I have brought up this issue before, some other HNers have replied that there are some statistical tools for correcting for response-bias effects, IF one can obtain a simple random sample of the population of interest and evaluate what kinds of people respond. But we can't do that here on HN, nor can we for the online vocabulary estimation.
Another reply I frequently see when I bring up this issue is that the public relies on voluntary response data all the time to make conclusions about reality. To that I refer careful readers to what Professor Velleman is quoted as saying above (the general public often believes statements that are baloney) and to what Google's director of research, Peter Norvig, says about research conducted with better data,
I'm questioning the way respondents are classified.
I chose 'Canada' as my region since I'm from Montreal. My first language is English and I'm fluent French. I did the first half of elementary school in French. Firstly non-Quebec anglophones tend to have better grammar and larger vocabularies than anglophone Quebecers. Secondly it doesn't take into account that English can be a 3rd language . Most immigrants to Quebec are required by law to attend French language elementary and high schools (there are exceptions). Immigrant children who's first language isn't English or French (the majority) take on two new languages, English being their 3rd after French. English tends to be the social language for many.
Montreal has a strong tech industry employing bilingual/multilingual people many of which read HN and possibly took part in the survey. My gut feeling is that English speaking Quebecers are skewing the stats. More granular control over region will be useful; show some insight to this reality.
Note: I traveled through China and south east Asia last year and found the quality of English to be much better than I expected. Considering Indochina ruled by the French I didn't find a person who could speak it. To possibly classify any country as "non-English-speaking" is kind of silly. Every country is "other-English-speaking" but then again it's a subjective classification isn't it. Doesn't China have the largest English speaking population now...
Site creator here -- you're right, survey participants are incredibly literate. I suppose that's Internet users in general, disregarding YouTube commenters :), or else the particular people who have spread the test, or are interested in taking it. Average verbal SAT score on the site is 700 (out of 800), far above the population's average of around 500.
Right; most people with "normal" or worse vocabularies are emphatically not going to see a "test your vocabulary" link and say "Hey, instead of doing something fun, let's see how poor my vocabulary really is! Then I'll tell all my friends!"
And when they see some egghead friend on Facebook has posted their vocabulary score and is challenging them to respond... they'll roll their eyes, and move on to their Farmville updates.
Don't get me wrong -- I love these things, and it came back with 37K for me -- but there's no way I'm posting that score, or even the link, to Facebook. I know how to maintain friendships, and saying "look how smart I am; I'm probably smarter than you" does not figure into it.
I can't imagine that the percentiles are reflective of the general population...I got 27,000 which it claims is the median score, and from practical experience my vocabulary is quite a bit larger than most people's. If this is measuring, say, the HN crowd, then perhaps it's accurate.
I found the same thing. I suspect a combination of the early respondents being quite a bit above average and the possibility that some people are checking words they don't actually know, or simple believe they know a correct definition for.
I think it would be a good idea to weight the survey with some test questions that ask if you know a definition to some of the less common words and then ask you to pick a correct definition from a list of 5 with 4 incorrect answers. At least this way they can approximate how much someone may exaggerate their knowledge.
However as someone who answered as honestly as I could (without spending the time to verify my definition of each word) it is cool to know what my personal vocabulary is.
I only got ~22k and I managed to score in the top 15% on the GRE verbal portion not too long ago. So either people who take the GRE are on average below median or the results aren't quite accurate.
Likewise. I answered honestly and got a little over 22k. I've always tested extremely well on verbal portions of standardized tests and feel that my vocabulary is well above normal but this would put me at about the 20th percentile.
I just counted, and you've used at least 12685 dictionary words in HP:MOR [1]. I was expecting it to be more to be honest, but it didn't seem right not to post just because it didn't match my expectations.
It certainly seems unlikely that someone who can produce an excellent 500,000 word work of fiction (aside: thanks very much by the way,) in addition to reams of technical writing, has a vocabulary not in the 95th percentile of the population. OTOH, HP:MOR has fewer words in it than I expected, and even the upper bound of 14795 seems low. Maybe the working is wrong; it's shown below.
There's a huge difference between writing something targeted at a selected audience, in a given time period, limited range of topics the characters will discuss, etc. and listing out all of the vocabulary words you know personally across all domains of knowledge. Even though HP:MoR Harry has a broad vocabulary, for example, and may use some words not all readers will know, there are still tons of topics (with their own specialized vocabulary...) that will never come up in the written storyline -- even if Harry would know them well.
Harry also presumably does some practical limiting of his vocabulary in conversation, because only shared vocabulary is useful if you're trying to actually communicate and don't want to stop to give definitions all the time.
It might be more interesting to compare the unique word count of HP:MoR against some of the "real" Harry Potter books, if you can get your hands on the text.
Thanks to the internet†, I can reveal a surprise: with the same methodology, the dictionary words in the seven real Harry Potter books concatenated is 19,245 and the total unique words is 21,441.
Total word count is 1,122,131 which is longer than HP:MoR by a factor of three. Plotting mean unique word count for the whole, halves and quarters of MoR gives a fit of uniques=168*length^0.3357, which makes sense given Zipf's law. That formula predicts about 18,050 words for a work of the same length as the original HP.
(Edit to add obvious test in the other direction.) The first 386,829 words of the original HP contain 12,255 unique words. The last 386,829 words contain 13,635 uniques. So, its comparable but perhaps slightly more varied (MoR had 12,685).
In light of those figures, is it possible Eliezer's vocabulary is less good than he thinks (Dunning-Kruger)? Especially as the Harry Potter book were written for children and presumably edited as such.
On the other hand, the fact that Eliezer seems to have used fewer words in his writing than you'd expect if his vocab was excellent doesn't mean that his known vocab is poor — he might just not use all the words he knows in writing.
Additionally, given the success of J K Rowling as an author, you might expect her vocabulary to be excellent, so it is conceivable that he's good and she's better.
† I have all the Harry Potter books on a shelf at home. Is torrenting the pdfs at work so I can word count them infringing copyright? I could have done it manually, it just would have taken longer.
I thought Eliezer's Lesswrong sequences might give different results. Applying your tests to those (from http://jb55.com/lesswrong/), I get 257,646 total words, 11,666 unique dictionary words, and 12,721 unique words (I'm surprised there aren't more unique words, given that the quantum physics sequence is in there).
168*257,646^.3357 = 11,010, so the sequences seem to be at about HP level.
Excellent work, by the way; thanks for the analysis.
By contrast, I scored 23,300 putting me on par with the average result for a 15 year old, if this test is to be believed. I was brutally honest about not selecting words I only recognised and might be able infer meaning from in context, but couldn't articulate a clear definition for (of which there were a surprising number).
And yet I look at the vocabulary used in the comment posts of those claiming high scores, and wonder how they ever scored so highly (accepting that comments are not neccessarily reflective of ones general writing or vocabulary). I believe that as this test is so open to cheating, that using responses to it as the corpus for determining median scores renders the entire exercise completely meaningless.
None of the words were domain-specific; they were all general vocabulary. You're a polymath, but a lot of your knowledge is focused into technical areas that weren't represented on the test.
lots of people here are saying they scored lower than what they expected, and that maybe other people cheated. that could be it, but it could also be that hacker news folks tend to be overconfident. this would match the stereotype of this group being mainly male nerd entreprenuers, which could score worse on things like this but perceive themselves to score much higher (a feeling not a fact backed by studies that i can remember). who knows; just voicing this thought since no one has mentioned it yet.
I got just under the median on this test, but I scored around the 99th percentile in the SAT verbal, and I feel it's reasonable to say the two test approximately the same things. It seems unlikely that I've slipped so far in just two years :)
I never really considered Hacker News to be full of overconfident people. If anything, to me being part of HN is a humbling experience. It reminds me that there are so many people out there that are smarter or more experienced than me, and I've seen other people say similar things.
I'm pretty much in the same boat. This scored me below median, but I scored in the 99th percentile on the SAT verbal as well (although that was a decade ago for me).
I think one contributing factor is the structure of the questions. If you ask me if I KNOW at least one definition for the word mawkish for example, I'll choose no I don't know a definition of it. I do however have a good enough feel for the word, that I can almost guarantee I can get a SAT style analogy with it correct, or if you gave me a multiple choice selection of definitions I can probably pick out the right one. I don't consider either of those skills equivalent to actually knowing a definition of a word.
Same with me (SAT)... It says I have below the median vocabulary. They should have at least had some sort of base standard to compare the voluntary data with so that they could measure their sample drift.
Hacker News is _very_ full of over-confident people, believe me.
During a discussion of 'whether open source contributions were being overly important to job seekers' a while back, a surprising number of HN commenters automatically put themselves in the role of employer, peering dubiously over their glasses at me. Many of these commenters were hilariously under-qualified to be taking on that kind of role with respect to anyone.
I don't think SAT verbal is nearly as intensely focused on obscure vocab. I didn't do it, but did do a GRE verbal back in 1994 to get into grad school. For what it's worth, my score on this and my GRE verbal were quite consistent.
Correct. Most of the folks in question had filled in enough to be able to be straightforwardly findable in terms of linkedin, blogs, etc. It wasn't rocket science.
i scored just over 90th percentile and i promise i didn't lie. it seems to me that a lot of the words were very old-fashioned - they were the kind of words i learnt from context while reading books by people like dickens. they're not words i would use in normal conversation, or when writing.
anyway, what i'm saying is that i suspect sat verbal is testing something quite different - isn't it much more aimed at ability to use the language rather than whether you can recognise obscure words?
[what surprised me the most was how graded it was - on both tests there was a pretty clear cut-off point where the words became unknown. i didn't expect it to be that ordered.]
I doubt that many people are cheating, but the "percentiles" are basically meaningless because the survey obviously self-selects for 1) people who are on the Internet, 2) people who are on Internet sites where this survey will get reposted 2) people who suspect they have pretty good vocabularies (who is going to take a non-mandatory 'test' that doesn't even earn them any Facebook Credits to inform them they aren't so bright), etc.
End result: substantial inflation of scores relative to what you'd see if you gave the same survey to the general population (and assuming everyone is relatively honest).
That's possible, even likely. Another explanation would be that tests like this tend to be biased towards people with very evenly distributed interests, preferrably with a literary, classical bias or background. Someone who knows 10,000 words from one or two specialzed disciplines will always score worse than someone who diligently read through all the high school text books and topped it off with some gutenberg.org.
Think I have the lowest score here. 16,400 words. English is not my native language but I speak English daily and I wouldn't say my English is bad. Pretty disappointed with the score and also surprised the median is way way higher than I expected.
Edit: And, also to add, I followed 2 criteria for whether I know the word or not.
1. What's the absolute definition?
2. And can I find the equivalent or meaning of it in my native language? (which is Tamil, an Indian language, if anyone cares.)
I think a lot of the words are ones that you will only ever encounter in reading fiction (and flower/older fiction at that). Also, keep in mind that this test doesn't include field-specific technical jargon.
No one getting high scores did so because of or despite their education. They did it by having the sort of brain that happens to retains words like uxoricide and reading copious quantities of material in which such abstruse words are employed.
I got a high score and went to public schools in the US, and didn't finish college either.
At one point, the sun was generally widely accepted to revolve around the earth.
It's likely that education improves one's vocabulary, however there are a lot of variables there. People who go to good schools likely come from families where learning is important in any case, are wealthier, etc...
(Just over 30,000.) Personally, I learned far more of those words from reading George R.R. Martin or Guy Gavriel Kay than anything in my liberal arts education.
At a certain point, somewhere in the middle ranges of these scores, formal education at even the best schools will stop having any effect on your vocabulary. It becomes more a matter of where someone's interest lies. Do they read? A lot? Outside of the usual curriculum?
And at the upper end, it's more about personal predilections than anything else. Are rare words like little gems that you save and collect?
Actually, I immediately knew what "uxoricide" meant, not because I could recall seeing the word, but because of my education, which included four years of Latin in high school. ("Uxor" is Latin for "wife", and "-icide" is a very common suffix for "killing".)
So, I beg to differ. Education can definitely help. Bravo to you, though, for building your vocabulary on your own.
Self-selected survey respondents != actual population.
Though I suppose, also -- IQ doesn't say much at all about what your vocabulary will be. If you don't read much, or mostly read popular literature, it doesn't matter how clever your are; your vocabulary won't be any bigger than the words you've encountered enough to form (or look up) a definition.
Though it's also worth noting that beyond a certain point, building a broader vocabulary isn't very useful; when you communicate, you generally need to limit yourself to vocabulary your audience knows. Likewise, when you're reading, it's pretty common that the authors using particularly obscure vocabulary are using it to muddy their meaning, not clarify it.
330 comments
[ 2.3 ms ] story [ 271 ms ] threadEdit: I just had a look at the median word count for adults who took the survey. It's around 27,000. I wonder whether that's true or not.. it seems to me that I'm lacking.
(My result:) http://testyourvocab.com/?r=35795
Also worth seeing: http://testyourvocab.com/hard.php
You need a test where the definition is part of the test. Otherwise anyone can just check boxes.
Anyway, my score: http://testyourvocab.com/?r=36192
[1] http://testyourvocab.com/?r=36123
Going by magazines, common forwards and so on it seems the average man loves a good self-quiz.
And in the back of my mind, I thought that the bulk of quiz magazines were sold to women, no? (Though I'm probably being too pedantic...)
"The companion Brazilian project can be found at howmanywords.com.br"
Which right now just says "Calcule o tamanho do seu vocabulário em inglês / Coming soon"
Edit: Oh. The Brazilian site is just a Portuguese version of the instructions for the English site? :(
Memorizing trivia words is just something that has never interested me. Instead I keep a thesaurus and dictionary handy at all times :)
ObScore: 36000, 32yo male US-english native speaker. There were a few more I'd seen but wasn't really sure enough to define. I maxed out the SAT verbal section (and got 1 wrong on math, which was enough to drop to 780/800) back in the 1990s.
The thing that always surprises me with this sort of test is how many words I half-know: I recognise the word, I have a (correct) general sense of what it means, but either I couldn't articulate an accurate definition or the definition I would give is an uncommon usage and I hadn't come across the more common meaning(s).
Following the rules strictly (so I didn't tick the half-known words), I came in slightly below average for my age, assuming the trend continues beyond the 32 years that their table currently gives.
For comparison, if I also included the words I half-knew, I gained nearly 4,000 new words and went up about 20%.
What difference does it make? The site doesn't say what it means in everyday life. I'm guessing if you exclude high achieving SAT vocab nerds, it finds the difference between people who care about the meaning of each word and people who will guess through context because they have no patience for a dictionary. Or people who don't read fancy texts, like the Scarlett Letter for example, after failing to read that I stopped reading books.
Maybe working on that English Minor is panning out...
Not sure I buy the results though. I would think that the rate of increase would start to decrease quite significantly after high school/college but it appears to stay pretty much linear throughout the data.
I scored far lower than I would have expected, which although it hurts my ego a bit, I can easily dismiss because of the nature of this test.
In all the exams I had in my life, from 1st grade to BS in CS, I only had one multiple choice test.
E.g.: One of the words I encountered in the test was 'terpsichorean'. While I knew that Terpsichore is one of the Muses, I did not know which one and left the box unticked. Had there been multiple choices, I might have guessed the correct solution.
BTW, the word is 'deduce' ;)
To my defense, they are both derived from 'deducere', to lead away, and 'were not distinguished in sense until the mid 17th cent' according to my system’s dictionary
Google tells me it means dancing so I was way way off (maybe conflating turquoise and cerulean?).
But I have no way of knowing how many words that I feel comfortable defining are actually nowhere near correct, so to be any kind of accurate, they need to do some verification of correctness. All 'honesty' means is 'don't deliberately cheat' not 'don't be dumb'.
However, I think a multiple choice test could also inflate scores unless the definitions were very cunningly constructed.
I scored 75-80th percentile (32,800) which surprised me. It seems quite a lot of words, for one. For another I consider my vocab' to be very good and I don't think I'm being bigheaded in that. Ergo I expected to be ranked higher.
On the second page there was an entire column of words of which I recognised only three sufficiently to provide a guaranteed accurate definition. One of that column was terpischorean, another tatterdemalion.
Whilst looking up tatterdemalion I found little use of it after the 1930s except as a proper noun (a Marvel Comics character for example). What I did find however is that Google Books is useless for finding dates. One citation from an author Sir Edward Bulwer Lytton is given a date of 1999. That's a reprint date, the author died in the 19th century.
VOLUNTARY RESPONSE POLLS
As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make such an inference. That's because the data are not from a random sample of the relevant population. One professor of statistics, who is a co-author of a highly regarded AP statistics textbook, has tried to popularize the phrase that "voluntary response data are worthless" to go along with the phrase "correlation does not imply causation." Other statistics teachers are gradually picking up this phrase.
-----Original Message----- From: Paul Velleman [SMTPfv2@cornell.edu] Sent: Wednesday, January 14, 1998 5:10 PM To: apstat-l@etc.bc.ca; Kim Robinson Cc: mmbalach@mtu.edu Subject: Re: qualtiative study
Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.
http://mathforum.org/kb/thread.jspa?threadID=194473&tsta...
For more on the distinction between statistics and mathematics, see
http://statland.org/MAAFIXED.PDF
and
http://escholarship.org/uc/item/6hb3k0nz
I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll
http://historymatters.gmu.edu/d/5168/
http://www.math.uah.edu/stat/data/LiteraryDigest.pdf
is a spectacular historical example of a voluntary response poll with a HUGE sample size and high response rate that didn't give a correct picture of reality at all.
When I have brought up this issue before, some other HNers have replied that there are some statistical tools for correcting for response-bias effects, IF one can obtain a simple random sample of the population of interest and evaluate what kinds of people respond. But we can't do that here on HN, nor can we for the online vocabulary estimation.
Another reply I frequently see when I bring up this issue is that the public relies on voluntary response data all the time to make conclusions about reality. To that I refer careful readers to what Professor Velleman is quoted as saying above (the general public often believes statements that are baloney) and to what Google's director of research, Peter Norvig, says about research conducted with better data,
http://norvi...
I chose 'Canada' as my region since I'm from Montreal. My first language is English and I'm fluent French. I did the first half of elementary school in French. Firstly non-Quebec anglophones tend to have better grammar and larger vocabularies than anglophone Quebecers. Secondly it doesn't take into account that English can be a 3rd language . Most immigrants to Quebec are required by law to attend French language elementary and high schools (there are exceptions). Immigrant children who's first language isn't English or French (the majority) take on two new languages, English being their 3rd after French. English tends to be the social language for many.
Montreal has a strong tech industry employing bilingual/multilingual people many of which read HN and possibly took part in the survey. My gut feeling is that English speaking Quebecers are skewing the stats. More granular control over region will be useful; show some insight to this reality.
Note: I traveled through China and south east Asia last year and found the quality of English to be much better than I expected. Considering Indochina ruled by the French I didn't find a person who could speak it. To possibly classify any country as "non-English-speaking" is kind of silly. Every country is "other-English-speaking" but then again it's a subjective classification isn't it. Doesn't China have the largest English speaking population now...
Doesn't matter. Look at the density instead.
And when they see some egghead friend on Facebook has posted their vocabulary score and is challenging them to respond... they'll roll their eyes, and move on to their Farmville updates.
Don't get me wrong -- I love these things, and it came back with 37K for me -- but there's no way I'm posting that score, or even the link, to Facebook. I know how to maintain friendships, and saying "look how smart I am; I'm probably smarter than you" does not figure into it.
I think it would be a good idea to weight the survey with some test questions that ask if you know a definition to some of the less common words and then ask you to pick a correct definition from a list of 5 with 4 incorrect answers. At least this way they can approximate how much someone may exaggerate their knowledge.
However as someone who answered as honestly as I could (without spending the time to verify my definition of each word) it is cool to know what my personal vocabulary is.
Also this article does claim 24k-30k is the average for native english speakers: http://www.independent.co.uk/news/world/americas/english-lan...
It certainly seems unlikely that someone who can produce an excellent 500,000 word work of fiction (aside: thanks very much by the way,) in addition to reams of technical writing, has a vocabulary not in the 95th percentile of the population. OTOH, HP:MOR has fewer words in it than I expected, and even the upper bound of 14795 seems low. Maybe the working is wrong; it's shown below.
[1] http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_M... — Eliezer's amazing Harry Potter fanfiction in case you're missing out.Harry also presumably does some practical limiting of his vocabulary in conversation, because only shared vocabulary is useful if you're trying to actually communicate and don't want to stop to give definitions all the time.
It might be more interesting to compare the unique word count of HP:MoR against some of the "real" Harry Potter books, if you can get your hands on the text.
Total word count is 1,122,131 which is longer than HP:MoR by a factor of three. Plotting mean unique word count for the whole, halves and quarters of MoR gives a fit of uniques=168*length^0.3357, which makes sense given Zipf's law. That formula predicts about 18,050 words for a work of the same length as the original HP.
(Edit to add obvious test in the other direction.) The first 386,829 words of the original HP contain 12,255 unique words. The last 386,829 words contain 13,635 uniques. So, its comparable but perhaps slightly more varied (MoR had 12,685).
In light of those figures, is it possible Eliezer's vocabulary is less good than he thinks (Dunning-Kruger)? Especially as the Harry Potter book were written for children and presumably edited as such.
On the other hand, the fact that Eliezer seems to have used fewer words in his writing than you'd expect if his vocab was excellent doesn't mean that his known vocab is poor — he might just not use all the words he knows in writing.
Additionally, given the success of J K Rowling as an author, you might expect her vocabulary to be excellent, so it is conceivable that he's good and she's better.
† I have all the Harry Potter books on a shelf at home. Is torrenting the pdfs at work so I can word count them infringing copyright? I could have done it manually, it just would have taken longer.
Excellent work, by the way; thanks for the analysis.
And yet I look at the vocabulary used in the comment posts of those claiming high scores, and wonder how they ever scored so highly (accepting that comments are not neccessarily reflective of ones general writing or vocabulary). I believe that as this test is so open to cheating, that using responses to it as the corpus for determining median scores renders the entire exercise completely meaningless.
http://www.reddit.com/r/linguistics/comments/edlnv/reddit_i_...
I never really considered Hacker News to be full of overconfident people. If anything, to me being part of HN is a humbling experience. It reminds me that there are so many people out there that are smarter or more experienced than me, and I've seen other people say similar things.
I think one contributing factor is the structure of the questions. If you ask me if I KNOW at least one definition for the word mawkish for example, I'll choose no I don't know a definition of it. I do however have a good enough feel for the word, that I can almost guarantee I can get a SAT style analogy with it correct, or if you gave me a multiple choice selection of definitions I can probably pick out the right one. I don't consider either of those skills equivalent to actually knowing a definition of a word.
During a discussion of 'whether open source contributions were being overly important to job seekers' a while back, a surprising number of HN commenters automatically put themselves in the role of employer, peering dubiously over their glasses at me. Many of these commenters were hilariously under-qualified to be taking on that kind of role with respect to anyone.
I don't think SAT verbal is nearly as intensely focused on obscure vocab. I didn't do it, but did do a GRE verbal back in 1994 to get into grad school. For what it's worth, my score on this and my GRE verbal were quite consistent.
You inferred this based on user profiles?
Heh.
anyway, what i'm saying is that i suspect sat verbal is testing something quite different - isn't it much more aimed at ability to use the language rather than whether you can recognise obscure words?
[what surprised me the most was how graded it was - on both tests there was a pretty clear cut-off point where the words became unknown. i didn't expect it to be that ordered.]
Maybe you're thinking of the Dunning-Kruger effect? (http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)
End result: substantial inflation of scores relative to what you'd see if you gave the same survey to the general population (and assuming everyone is relatively honest).
Edit: And, also to add, I followed 2 criteria for whether I know the word or not.
1. What's the absolute definition?
2. And can I find the equivalent or meaning of it in my native language? (which is Tamil, an Indian language, if anyone cares.)
Clearly my American public school education has served me well. >_<
I got a high score and went to public schools in the US, and didn't finish college either.
It's likely that education improves one's vocabulary, however there are a lot of variables there. People who go to good schools likely come from families where learning is important in any case, are wealthier, etc...
And at the upper end, it's more about personal predilections than anything else. Are rare words like little gems that you save and collect?
So, I beg to differ. Education can definitely help. Bravo to you, though, for building your vocabulary on your own.
Though I suppose, also -- IQ doesn't say much at all about what your vocabulary will be. If you don't read much, or mostly read popular literature, it doesn't matter how clever your are; your vocabulary won't be any bigger than the words you've encountered enough to form (or look up) a definition.
Though it's also worth noting that beyond a certain point, building a broader vocabulary isn't very useful; when you communicate, you generally need to limit yourself to vocabulary your audience knows. Likewise, when you're reading, it's pretty common that the authors using particularly obscure vocabulary are using it to muddy their meaning, not clarify it.