Show HN: News headline analysis of over 140,000 headlines over several years (headlines-and-data.herokuapp.com)
This is the result of an 8 day final project for DBC Chicago. Our team scraped over 140,000 headlines of several news agencies stretching back several years. We then took those headlines and fed them through the AlchemyAPI sentiment analysis engine to assign each one a score. They were then plotted in a couple different ways using D3.
This is far from perfect and even farther from scientific. It was done in 8 days by some passionate amateur developers. It was however a lot of fun and very interesting.
You can read about it and the team in more detail on the repo page here:
https://github.com/kelmerp/headline_sentiment_rating
and see some slightly more technical slides here:
https://speakerdeck.com/luizneves77/sentimental-headlines
This was written in RoR, Postgres (Memcached), and javascript + D3.
I'm also the creator of:
onionornot.com reddesigned.com http://luiz-n.github.io/route-search/
and am interviewing for web dev (and data visualization) positions in the chicago area if you would like to reach out to me. @hey_luiz
5 comments
[ 3.3 ms ] story [ 16.6 ms ] threadThis is far from perfect and even farther from scientific. It was done in 8 days by some passionate amateur developers. It was however a lot of fun and very interesting.
You can read about it and the team in more detail on the repo page here:
https://github.com/kelmerp/headline_sentiment_rating
and see some slightly more technical slides here:
https://speakerdeck.com/luizneves77/sentimental-headlines
This was written in RoR, Postgres (Memcached), and javascript + D3.
I'm also the creator of:
onionornot.com reddesigned.com http://luiz-n.github.io/route-search/
and am interviewing for web dev (and data visualization) positions in the chicago area if you would like to reach out to me. @hey_luiz
A few things immediately jumped out at me:
1. On an average, Fox, seems to be more positive than CNN. Hmm.
2. The headlines were more positive than most {sources/days} on CNN Politics in the second half of 2009. Obama's election is the only story I can think of that could have affected all the sources, but I may be wrong.
3. Around the same time, (shortly after the election results, in fact) HuffPost saw a sharp decline in the positivity of their headlines and dipped into negative. Hmm...
4. The overall trend of headlines is more positive than negative, leading me to wonder why is it that, as a people, we are so cynical.
I'd love to see a graph that plots the median instead of the average of the daily scores. Also, I dunno much about the Alchemy scoring, but quite a few of the headlines (~30-50% per day) seem to have scored 0.0 - is that bad detection or consistently neutral reporting? If so, I suspect the median graph will look a lot different.
I'd also love to see a similar analysis for non-mainstream media - especially, aggregators like reddit, HN and slashdot. Maybe, you could add the post ranking/comments/karma as other variables and attempt to refine the analysis further?
Just bouncing a few ideas, that all. All in all, good job. :)
In a more automated way, you can extract all keywords from your titles, and then auto-extract the top keywords that are the most dividing.
(Source: former Google News TL, we used to have way too much fun doing stuff like this)