> Dr. Octagon is a persona created and used by American rapper Keith Matthew Thornton, better known as Kool Keith...Dr. Octagon is an extraterrestrial time traveling gynecologist and surgeon from the planet Jupiter. [1]
I checked for him immediately once I saw this chart. Between Doc Oc, Dr. Dooom and Black Elvis, he has some insanely weird lyrics.
We stuck together when one of my parakeets died
You broke down and cried, for the love of animals
I used to always cut the legs off a roach
See if he'll stay there on a piece of tissue
And give him a piece of toast
That morning, he would wake up and be gone
What, the insect had a ambulance?"
I had never heard this particular Kool Keith rap before but as soon as I read the lyrics I could hear his very distinctive delivery pattern. I listened to the track on YouTube and was not surprised it was just as I have imagined. How distinctive.
" First patient, pull out the skull, remove the cancer
Breaking his back, chisel necks for the answer
Supersonic bionic robot voodoo power
Equator ex my chance to flex skills on Ampex
With power meters and heaters gauge anti-freeze
Octagon oxygen, aluminum intoxicants
More ways to blow blood cells in your face
React with four bombs and six fire missiles
Armed with seven rounds of space doo-doo pistols
You may not believe, living on the Earth planet
My skin is green and silver, warhead looking mean
Astronauts get played, tough like the ukelele
As I move in rockets, overriding, levels
Nothing's aware, same data, same system"
[...]
Hook x4
Earth People, New York and California
Earth People, I was born on Jupiter.
This looks at the first so many lyrics in each rapper's career. Aesop Rock came out with some weird stuff right off the bat. I wonder if some of these other rappers became more sophisticated over time. Maybe an average per song would be better, or average uniques per word, would be better.
The problem with average per song is that you "use up" words in every new song, so all things being equal each marginal song has progressively fewer new words.
If a rapper released one song using n distinct words their score would be n/1, and if they released a second song using the same set of words their score would halve, to n/2, despite the fact their demonstrated vocabulary is still n words.
In fact, if their first song used n distinct words and their second used a completely distinct set of words, but the second song was shorter than the first, their score would drop.
That would be unusual behaviour for a measure of vocabulary.
I don't think that's what the poster meant. By "average unique words per song" I take it to mean, within each song words are only counted once, but across songs, words can be counted multiple times. So if song A had the words "I like cats" and song B had the words "I like dogs", then the average unique word count would be ((3 + 3) / 2) = 3, not ((3 + 1)/2) = 2.
That's definitely one solution, but it still wouldn't quite capture it. As an extreme example, if rapper A produced 100 songs, each with exactly the same lyrics, they should surely be penalized compared with rapper B producing 100 songs with no shared words— even if rapper A's average unique-words-per-song is higher than rapper B's.
I bet you could get something insightful from plotting "unique words" versus "total words" - That might give a good idea of the amount of repetition over time, the length or quantity of output, and the total vocabulary.
I haven't read all his work, but I guess Shakespeare didn't use "hangn" and "hangin" as an alternative to "hanging".
The author could validate the words against a dictionary, but it would still be flawed due to conjugations being counted as different words.
I would have been rather surprised not to see Aesop Rock fairly high up the list. I was reading the Rap Genius pages for a few of his tracks the other week and the sheer density of wordplay was fairly overwhelming.
I'd really like to see this broken down by established vocabulary and made up vocabulary. I think that would really start to show who were the best lyricists on both ends. Rappers with a lot of made up words might be on the far left, and rappers with a lot of unique words that aren't made up would be on the far right. Both sides of the scale would show rapping talent on different dimensions. Influential rappers like E-40 who add new words to the vocabulary, and wordy rappers like Aes on the right who use a really dense and descriptive vocabulary.
At what point do 'made up' words become 'established'. After they've been published? If so, every made up word in a song should be considered 'established'.
I didn't mean to imply that being on the far left or being made up meant 'worse' in any way. Quite the opposite in fact - I think that if someone can create a word or phrase that becomes normal in everyday usages, it's a sign of... something. Maybe influence, maybe genius. Like you said, Shakespeare first used words that we consider common.
I have no idea at what point a word becomes 'established' though.
EVERY word is a "made-up word". Some words were just made up longer ago than others. If you held everyone to this criteria you would severely impoverish language.
This is fascinating. I'm only a recent listener of hip-hop (primarily because of Earl Sweatshirt and Odd Future) and I'm in awe of the vernacular.
And similarly, as a boredom exercise a few weeks ago I did some lexical analysis of the song Timber (the monstrosity was being constantly played on the radio at the time) and here's what I came out with:
"83.1% of the words in the lyrics are five letters or less, 58.9% are four letters or less. The lexical density (the number of unique words divided by the total number of words, multiplied by one-hundred) is 29.1%. There is only one word in the song which has three or more syllables. Eleven people were involved with the writing of the song, each of them capable of producing just nine unique words each."
You must be using big words in case someone does the analysis on HN comments.
Definitions:
Vernacular: ??
Lexical analysis (in this case): ratio of unique words to non unique words
Lexical density: What persent of the words is unique?
The paragraph in quotes is copied and pasted from when I wrote that a few weeks ago, there's a definition in parentheses following lexical density. Analysis is a word that should not need a definition attached to it (you have used it yourself in your comment.)
Vernacular is commonly used in the United Kingdom. Google will provide you with a definition.
My last sentence was intended as a satire, lyrics are obviously not uniformly distributed between writers. I completely agree with you though, the song (despite my disdain for it) is incredibly catchy, and definitely not intended to be thoughtful or thought provoking in nature.
Is that with or without 'the' 'be' 'to' not that its any sort of literary accomplishment regardless but English is a terrible language for lexical density.
Its a nice touch including portmanteaus and 'incorrect' ebonics on the list (like "ery'day"), since authors like shakespeare, joyce and others took the same liberties with language. Arguably, that's how language develops and makes it interesting to study and think about. The OP could have easily stuck to words in the OED, kudos.
I'm a big fan of the project and the way it is presented. Not sure why Wu-Tang features so prominently but I guess I'm okay with that. Kool Keith should be broken down further into his constituent parts. I also would have thought the Beastie Boys would have run higher.
I think I remember reading Aesop say Del influenced him a great deal. If you like Aesop, check out El-P, Cannibal Ox, Illogic, and basically anyone who was associated with Def-Jux.
I'm not really sure that the other acts totally lack artistic integrity, but I'm pretty sure that ytcracker isn't a novelty act.
Some of them aren't serious and have the overdeveloped sense of irony that you expect from people kidding around, but I'm not sure that that makes them novelty music any more than, say, The Electric Six are.
> Shakespeare’s vocabulary: across his entire corpus, he uses 28,829 words, suggesting he knew over 100,000 words
Why does that suggest he knew over 100k words? Maybe it means he knew 28,829 and used all of them? Would he really know over 70,000 words he never used in his works? What would those 70,000 words be? Probably very obscure ones. How can you know that many obscure ones?
Vocabulary has for a while been considered in terms of 'receptive' and 'productive' capacity, with the assumption being that ones 'receptive' vocabulary can be larger, since it is easier to hear/read/understand a word than it is to use it correctly in reading/writing (this is not necessarily the popular opinion anymore [http://www.readingconnect.net/web/FILES/english-language-and...] but may provide the context for the claim about Shakespeare). The notion is that you are able to understand more words than you commonly use in your speech/writing, which is on some level intuitive, although of course it is an empirical question.
The 28K figure is acheived by counting multiple spellings of the same word. Shakespeare lived before dictionaries, so there was never a single standard way to spell a word.
I'm curious if words that Shakespeare invented count. There are many words that we see first used by Shakespeare, though some of them were probably words invented during his time by others with him merely being the first to record (in documents that survived until today).
The latter is probably more the case. The OED, for instance, has had a bias in favor of using Shakespeare as a word's origin since that dictionary's first edition. The number of words attributed to Shakespeare in the OED has dwindled over time.
Just like you or I know tens of thousands of words, but only use some small subset of them in any given work, you wouldn't expect that Shakespeare would use his entire productive vernacular in producing the limited corpus of his literary works.
I imagine that it would be something similar to the German Tank Problem. (http://en.wikipedia.org/wiki/German_tank_problem ) Taking each writing as a sample of the words that are known then would allow for an estimation of total words known. I imagine that this would need to be modified to account for the non-uniform distribution of word use, but the principle would be the same.
Just a note, those artists don't necessarily use all their vocabulary. Eminem for example clearly holds back on his vocabulary. Rap is as much an art as anything can be so there are all sorts of factors. Be careful what you might want to draw here other than curiousity.
Gibes regarding racism aside, some people seem more articulate due to the fact that they carry themselves differently during interviews.
Of course, many musicians will keep a persona going during interviews as well, so it's still not a very reliable metric.
The most extreme example I've seen was Marilyn Manson, but there are plenty of musicians who rap / sing about really inane stuff and then show that they're way smarter than the way they present themselves with their music.
there are plenty of musicians who rap / sing about really inane stuff and then show that they're way smarter than the way they present themselves with their music.
As mentioned in the article, Jay-Z even raps about doing just that.
Jay Z's stats are there: he's at 4,506. Pretty middle of the pack.
About Biggie:"35,000 words covers 3-5 studio albums and EPs. I included mixtapes if the artist was just short of the 35,000 words. Quite a few rappers don’t have enough official material to be included (e.g., Biggie, Kendrick Lamar)"
And to respond to your child comment, I'm sure the same problem (not enough material) applied to Big L.
an artist that meets the requirements for the data, AZ. He's got 5+ albums, some of which were gold and it'd be interesting as I believe he may also be the highest selling solo artist in the top 10 aside from Ghostface
Before making this study, what were your predictions? Would have have expected Wu-Tang and GZA to be near the top?
What did you expect the average to be?
It would be very interesting to do something similar for rockers too.
Any chance you would release your code for this? I'd love to run an analysis on some lesser known rappers and play around with some of the filters. Awesome project btw.
EDIT: The reason I ask is that I assume you don't have the time or desire to add every rapper every person asks you for.
Weird Al's songs are not articulate masterpieces, but cheap parodies of other rap songs and rappers. He's probably somewhere around the 5,000 mark with the other artists.
A cursory google on the size of the average vocabulary [1] yields an interesting fact. I'm not sure how watertight it is. I realise it's probably unfair to compare the size of the average vocabulary to that of a series of songs. Songs being shorter for one. Still, it's interesting.
Not sure if that's fair to Weird Al. The people he parodies wouldn't really agree either [1]. It's not like he's doing the cheap morning show tactic of swapping clean words for dirty words or bad puns but leaving the rest of the song intact. They maintain a consistent theme which is really tough.
Yeah, perhaps I was a little harsh. Don't get me wrong, I like Weird Al. I probably could have phrased my comment a little better. I should have said something along the lines of, "In my experience listening to Weird Al, it doesn't feel like he explores a lot of the English language."
Really interesting, but not as representative as it should be. It's not clear why some have larger vocabulary than others. It could be using words like "zeitgeist" (in case of Aesop Rock) or some clever wordplay (I don't know much about hip-hop, so I can't find example for some artist from the list right off the bat, but I remember Marilyn Manson using word "gloominati" for instance) or pretty meaningless made up words like "schizzle" (in case of Snoop Dogg) or usual derivatives like "fuckedy fuck". Moreover, in many transcripts for hip-hop people write down words as they are pronounced, which can be pretty much distorted for some artists (which of course ideally shouldn't count as a "new word", but that's complicated, yeah).
While Aeson Rock and DMX are clearly extreme and not surprising at all, it's not that clear for some guys in the middle.
So, first off, for every data project sources should be provided, or at least more specific definition, how text was processed, tokenized, analyzed. Second, several more "data slices" should be provided, for instance 100 most used words which are unique for that artist compared to other artist in the list.
The example you used for clever wordplay, "gloominati," is actually considered a portmanteau word. It's the result of combining multiple words to create a new word. (I say this not to be a pedant, but because I learned the term recently and was amused that we actually have a word for it.)
Yes, portmanteaus are exactly what I was meaning by "clever wordplay". English isn't my primary language so it's hard to remember the right word sometimes, sorry for that. :)
OP here. Do I really need to provide all of this to satisfy the reader's ability to grasp the basic premise of the site? this isn't a thesis or academic pursuit, just comparing some rappers for fun.
I used plain NLTK token analysis on rap genius lyrics. in terms of several more data slices...I agree that there should be more cuts of the data, but you must understand the amount of time that it took me to put this together.
Of course it's entirely up to you what to provide. It would be silly of me to question that. I'm not paying you for that, so how can I demand anything? I'm just saying what I think should've been done. That's how I would do it, at least.
You see, I have a strong opinion on that any data-analytic work is pretty close to being useless if it's not reproducible. And I mean really close. I already mentioned few questions that naturally arise reading your article which are crucial to understand your results and are not addressed by the article. So, ideally, any data analysis done for the open community should provide both full dataset and full sources. Unfortunately, it's not always possible: dataset may be several terabytes long, there might be legal issues about disclosure of the data (or letting everyone know how exactly author acquired them), sources might be under NDA, whatever. In this case description should be as pedantic as possible, because some little details can change the whole meaning of statistics entirely.
By the way, NLTK allows you to do all kinds of processing, so it isn't the answer.
So what would I do? The usual course, actually: would do the work in iPython Notebook, cleaning it up afterwards, would've drawn graphs in place and printed some few slices I've said already while processing, so it would be easier to understand what actually that unique words counted might be like. While fancy d3 graphs are cool for sure, but not nearly as useful.
275 comments
[ 2.9 ms ] story [ 264 ms ] threadI love the fact that E-40 is about on par with Shakespeare. I'm sure he would take it as a compliment to be called the modern day Shakespeare.
> Dr. Octagon is a persona created and used by American rapper Keith Matthew Thornton, better known as Kool Keith...Dr. Octagon is an extraterrestrial time traveling gynecologist and surgeon from the planet Jupiter. [1]
1. http://en.wikipedia.org/wiki/Dr._Octagon
" First patient, pull out the skull, remove the cancer Breaking his back, chisel necks for the answer Supersonic bionic robot voodoo power Equator ex my chance to flex skills on Ampex With power meters and heaters gauge anti-freeze
Octagon oxygen, aluminum intoxicants More ways to blow blood cells in your face React with four bombs and six fire missiles Armed with seven rounds of space doo-doo pistols
You may not believe, living on the Earth planet My skin is green and silver, warhead looking mean
Astronauts get played, tough like the ukelele As I move in rockets, overriding, levels Nothing's aware, same data, same system"
[...] Hook x4
Earth People, New York and California Earth People, I was born on Jupiter.
yup, Jupiter indeed. God I love this song.
In fact, if their first song used n distinct words and their second used a completely distinct set of words, but the second song was shorter than the first, their score would drop.
That would be unusual behaviour for a measure of vocabulary.
http://www.mdaniels.com/vocab/scatter.png
love your other ideas – hopefully can do them later.
So much for the modern Shakespeares on the list.
http://poetry.rapgenius.com/William-shakespeare-hamlet-act-2...
I've now got empirical evidence of what I always thought.
I think DMX rhymes words with themselves more than any rapper I've ever heard.
It is rap for geeks though ;)
It's interesting that you think that. I'd recommend you look at this: http://www.shakespeare-online.com/biography/wordsinvented.ht...
At what point do 'made up' words become 'established'. After they've been published? If so, every made up word in a song should be considered 'established'.
I have no idea at what point a word becomes 'established' though.
Claim is that some 1700 or so of his 30,000 word vocabulary was invented by him. Including terms such as "assasination" and "gnarled".
And similarly, as a boredom exercise a few weeks ago I did some lexical analysis of the song Timber (the monstrosity was being constantly played on the radio at the time) and here's what I came out with:
"83.1% of the words in the lyrics are five letters or less, 58.9% are four letters or less. The lexical density (the number of unique words divided by the total number of words, multiplied by one-hundred) is 29.1%. There is only one word in the song which has three or more syllables. Eleven people were involved with the writing of the song, each of them capable of producing just nine unique words each."
Definitions: Vernacular: ?? Lexical analysis (in this case): ratio of unique words to non unique words Lexical density: What persent of the words is unique?
Vernacular is commonly used in the United Kingdom. Google will provide you with a definition.
I'm not sure why this is notable when you consider that lyrics are probably the least important aspect of a song intended for the top 40.
If you take a moment to listen to the melody and production, you'd probably see why it's credited to 10+ people. That song is a well-oiled machine.
http://www.koolkeith.co.uk/personas.html
If you're looking for expansive vocabularies you should consider exploring other dorky rappers like Scroobius Pip.
http://www.reddit.com/r/Music/comments/24omhw/rappers_sorted...
I'm interested to see where the likes of MC Frontalot, Wordburglar, YTCracker, etc. rank on that scale...
http://www.reddit.com/r/hiphopheads/comments/24neym/rappers_...
http://www.reddit.com/r/dataisbeautiful/comments/24nw9p/rapp...
My gut feeling is that the nerdcore guys would be decidedly middle-of-the-road, but I'd love to see the result!
Some of them aren't serious and have the overdeveloped sense of irony that you expect from people kidding around, but I'm not sure that that makes them novelty music any more than, say, The Electric Six are.
Why does that suggest he knew over 100k words? Maybe it means he knew 28,829 and used all of them? Would he really know over 70,000 words he never used in his works? What would those 70,000 words be? Probably very obscure ones. How can you know that many obscure ones?
E.g. Shakespeare actually invented a lot of the obscure words he used.
Kate Tempest - My Shakespeare: https://www.youtube.com/watch?v=i_auc2Z67OM
;-)
What makes you say this?
Of course, many musicians will keep a persona going during interviews as well, so it's still not a very reliable metric.
The most extreme example I've seen was Marilyn Manson, but there are plenty of musicians who rap / sing about really inane stuff and then show that they're way smarter than the way they present themselves with their music.
As mentioned in the article, Jay-Z even raps about doing just that.
It'd also be cool to add the members of AOTP to the analysis.
[Edit] also Notorious B.I.G. :)
About Biggie:"35,000 words covers 3-5 studio albums and EPs. I included mixtapes if the artist was just short of the 35,000 words. Quite a few rappers don’t have enough official material to be included (e.g., Biggie, Kendrick Lamar)"
And to respond to your child comment, I'm sure the same problem (not enough material) applied to Big L.
It would be very interesting to do something similar for rockers too.
I didn't expect wu-tang to be on the top since it would go across 11 different vocabs. I felt like the law of averages would basically play out.
EDIT: The reason I ask is that I assume you don't have the time or desire to add every rapper every person asks you for.
nltk.org/book
A cursory google on the size of the average vocabulary [1] yields an interesting fact. I'm not sure how watertight it is. I realise it's probably unfair to compare the size of the average vocabulary to that of a series of songs. Songs being shorter for one. Still, it's interesting.
[1] http://www.weirdalforum.com/viewtopic.php?t=5673
Your link is cool, thanks for sharing that.
Whack this through your Bowers and Wilkins:
https://www.youtube.com/watch?v=p_SQEUZomug
While Aeson Rock and DMX are clearly extreme and not surprising at all, it's not that clear for some guys in the middle.
So, first off, for every data project sources should be provided, or at least more specific definition, how text was processed, tokenized, analyzed. Second, several more "data slices" should be provided, for instance 100 most used words which are unique for that artist compared to other artist in the list.
I used plain NLTK token analysis on rap genius lyrics. in terms of several more data slices...I agree that there should be more cuts of the data, but you must understand the amount of time that it took me to put this together.
You see, I have a strong opinion on that any data-analytic work is pretty close to being useless if it's not reproducible. And I mean really close. I already mentioned few questions that naturally arise reading your article which are crucial to understand your results and are not addressed by the article. So, ideally, any data analysis done for the open community should provide both full dataset and full sources. Unfortunately, it's not always possible: dataset may be several terabytes long, there might be legal issues about disclosure of the data (or letting everyone know how exactly author acquired them), sources might be under NDA, whatever. In this case description should be as pedantic as possible, because some little details can change the whole meaning of statistics entirely.
By the way, NLTK allows you to do all kinds of processing, so it isn't the answer.
So what would I do? The usual course, actually: would do the work in iPython Notebook, cleaning it up afterwards, would've drawn graphs in place and printed some few slices I've said already while processing, so it would be easier to understand what actually that unique words counted might be like. While fancy d3 graphs are cool for sure, but not nearly as useful.