HN Explorer

A dataset of every Reddit comment

294 points by avinassh ↗ HN
TLDR; magnet link:

    magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80
Someone has already put it on Google Big Query - https://bigquery.cloud.google.com/table/fh-bigquery:reddit_comments.2015_05

Link to original Reddit thread - https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/

95 comments

[ 6.0 ms ] story [ 183 ms ] thread
[deleted]
(comment deleted)
akhilcacharya
This should be pretty great for research - the Reddit API is actually fairly limited in my experience, I wasn't even able to get my full comment history last time I tried.
tspike
This could be amazing as input to a question answering engine.
aluhut
I would love to feed my megahal with that.
douche
I think Reddit was one of the places they had to wall off Watson from data-mining, because it devolved into foul-mouthed memes. I wish they hadn't, nothing would be better than a billion dollar piece of technology deciding to reenact Sean Connery Jeopardy skits, on Jeopardy.

[2001 HAL voice]: I have a question about the penis mightier. Does it work?

ilyaeck
It's not the hard to filter put the foul-mouthed content
chiph
But what about phrases with double meanings (aka: Fun With English):

Children make nutritious snacks

ilyaeck
Not necessarily an issue in the grand scheme of things.
Houshalter
I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general.

I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching on what is possible with that data.

The scope of the discussions on reddit is huge. Despite a lot of jokes in the comments here, the quality of the comments on average, is pretty decent. And the metadata like subreddit and score are extremely useful for filtering it down more.

ilyaeck
I'd be very interested to learn more about your project and findings. Is that bot still alive?
bl4ckdu5t
If it's open source I'll love to take a look
Houshalter
I pastebined the code. It's terrible. I should rewrite it. But here it is: http://pastebin.com/CM9u17jq
jetpm
Nice, I see you use neural networks, can you explain a bit how you are training them?
Houshalter
It searches reddit for whatever the user queries. Reddit returns up to 100 threads. I then choose a thread and take the top comment.

The neural network predicts which thread is most likely to produce a satisfying answer. The main features are number of n-gram matches with the question, the score, the number of comments, and some other metadata.

It's far from optimal but it does improve it a bit.

Houshalter
Sample of conversation with it. Not cherry picked, just showing some of the good and the bad:

https://i.imgur.com/LDD9isL.png?1

mcintyre1994
This is awesome, I love the "current drama on reddit" answer..
avinassh
awesome! how does one get started programming 'intelligent' things? I know how to program, haven't coded anything that is 'intelligent'. I would really love to know what all the things/concepts I should know to build a bot like yours. Thanks!
fit2rule
Anyone know how big it is yet?
minimaxir
Dataset on BigQuery is 265GB.
fit2rule
I wasn't able to get access to it on BigQuery .. the progress bar just sat and sat .. can you confirm that its usable on BigQuery - I'd sure love to play with this data.
ihsw
160GB.

One bz2 file of comments per month.

iaw
This is an awesome dump, what types of analysis could we use this for?

tspike mentioned a QA engine which would be awesome, what else can people think of?

roflchoppa
If anyone is working with the data, we should see how many users actually hivemind comment on things vs OC
thomasfromcdnjs
Someone should try make a sentient twitter bot that learns from the Reddit data
avinassh
How does one go about doing that? I know how to program, but I have written anything which do 'intelligent' stuff.
thomasfromcdnjs
I have no idea either, I would also love for someone to chime in with some first steps.
Ianvdl
Markov chains would probably be a good start.

See reddit.com/r/subredditsimulator (not always safe for work)

[deleted]
(comment deleted)
sneak
People have been trying to make sentient bots of any kind for some time now.
shiggerino
I'd be extremely impressed with a somewhat general AI at the insect or nematode level, let alone human.
minimaxir
As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel...

BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.)

I can answer any other questions that people have.

daave
> Since the full dataset is ~285GB, you only get 4 queries per month.

That's only true if your 4 queries need to read every single column.

One of the big advantages of BigQuery's column-oriented storage is that you only pay to read the columns that are actually needed to answer your query.

For example, this query to extract the top 10 authors only cost me 19GB to run (and took 7.0s):

  SELECT
    author,
    COUNT(*) AS COUNT
  FROM
    TABLE_QUERY([fh-bigquery:reddit_comments], "table_id CONTAINS '20' AND LENGTH(table_id)<8")
  GROUP EACH BY
    author
  ORDER BY
    COUNT DESC
  LIMIT
    10;

  author,COUNT
  [deleted],228425822
  AutoModerator,3677774
  conspirobot,575576
  ModerationLog,547671
  autowikibot,402076
  PoliticBot,388395
  imgurtranscriber,360248
  dogetipbot,358093
  qkme_transcriber,301968
  TweetPoster,293309
minimaxir
Good point.

Although if you're doing analysis on the body column itself, it'll use the majority proportion of the data set, of course.

ilyaeck
Is this data in a format that allows to recover threads? I.e., comment X was a response to comment Y ?
minimaxir
parent_id is a column, so you could recreate hierarchy by JOINING a table on itself.
[deleted]
(comment deleted)
rcpt
I want all the url submissions in a given subreddit, but all I can find in the tables is "link_id". How do I map link_ids to urls?
minimaxir
There is no submission data yet.

/u/Stuck_in_the_Matrix commented on another thread that he's working on it.

nathan_f77
I think it's fascinating that if anyone uses this dataset to train an AI, some tiny piece of my personality might make a contribution.
kristopolous
It's going in my /raid/datasets with about 1.4TB of other comment data. So the answer is going to be "yes" ... well probably by Tuesday morning it will be.

I've used older reddit dumps before ... it's not great data for most things, but there's an awful lot of it!

danso
Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active user-run bot ecosystem that helps manage and monitor Reddit's discussions.
hackcasual
I believe reddit comments are made under a CC license
mattrepl
Closest thing to an answer I've found is from the licensing page: https://www.reddit.com/wiki/licensing

But it doesn't specifically address collection and distribution of API results as a dataset.

(From the page:)

A licensing agreement is required in order to:

* use the reddit API for commercial purposes. Use of the API is considered "commercial" if you are earning money from it, including via in-app advertising or in-app purchases. Open source use is generally considered non-commercial.

* use the reddit alien logo ("snoo") in your app or for its thumbnail. Any new apps you create must be approved as well before usage. The circular "r" logo is reserved solely for use by reddit, Inc.

* allow users to subscribe to reddit gold via in-app purchases. If your platform allows for it, we encourage you to work with us to make this happen. We see gold purchases as a way for you to help reddit and to give back to the reddit community.

danellis
> use the reddit alien logo ("snoo") in your app or for its thumbnail

Do any of the apps actually do this, though? (Especially the free ones.)

anastasds
UReddit does, but may not be a great example because I received permission to use the logo in a private conversantion with hueypriest years ago (as long as I didn't try to monetize the project).
jchomali
Not sure if the comments are under any license.
kbullaughey
According to Reddit's User Agreement user content:

"You retain the rights to your copyrighted content or information that you submit to reddit ('user content') except as described below."

And the exceptions just state that Reddit has a perpetual irrevocable worldwide license.

So it seems like there's no default license and others don't have any automatic rights to use the content. Does this assessment seem correct? In practice, it may not be a big problem, particularly for academic research and such, but I'm guessing there are some uses that might cause problems.

hackcasual
Good find, I think I got reddit confused with stackoverflow or wikipedia. If there's no clear assignment of copyright to api users, I would imagine that would be problematic for 3rd party app makers.
JeremyBanks
Nit: I don't think copyright assignment is the correct term here. That refers to transferring the ownership of the root rights to the material, but you're just referring to granting specific rights under a license.
hmsimha
Isn't this true only if the 3rd party apps are saving the data and/or redistributing it? Apps should be able to request data from reddit just as a normal user would through their browser, since reddit has permission to use (and serve) that content.
curiousjorge
how to access the BigQuery data?

I get this screen

Welcome to BigQuery! What is BigQuery?

but it doesn't show the data.

minimaxir
You'll need to set up a project in BigQuery first. (you don't have to give billing information to use the free quota)
curiousjorge
got it running!

I just did a query for my reddit handle and it took 6.5 seconds to retrieve all of my comments. Kind of a snowden moment but this is super interesting, first time I played with Big Query.

Would love to run some google API for sentiment analysis.

TTPrograms
This would be a really amazing way to make factually backed statements about the nature of Reddit for news sources given the recent publicity.

I.e. the frequency of comments of a certain nature, typical karma scores for those comments, breakdown by subreddit etc.

Lerc
You have to be careful determining the nature of a certain comment algorithmically.

I saw one analysis of subreddits that checked for negativity. /r/PathOfExile came in as one of the most negative subreddits, which could easily be turned into a narrative about gaming culture.

To a person familiar with the context it seems far more likely that game concepts were skewing the data. Discussions involved killing, physical damage, life leech, killing, dying etc. Not to mention creature names themselves Devourers, Plummeting Ursa, Miscreations,

mbell
Wow, that is much, much, smaller than I thought it would be.

Anyone have a recent link on Reddit's infrastructure? Given the small size I would think it easily fits in memory so I'm a bit curious how they handle it.

Houshalter
I really want to get my full commenting history from this, but the download is enormous. Anyone know how I could go about this with cloud services?
stuck_in_the_ma
Hey there! I'm the one that uploaded the original dataset. I'm creating API endpoints that will easily allow you to do this. It should be completed in 1-2 weeks.
x0
Beautiful!
ytdht
here it is, only selected the body of your comments though, out of 4005 comments... http://s000.tinyupload.com/?file_id=09238386475544637092

here is the query that I used:

  SELECT
    author,
    body
    FROM
       [fh-bigquery:reddit_comments.2007],
       [fh-bigquery:reddit_comments.2008],
       [fh-bigquery:reddit_comments.2009],
       [fh-bigquery:reddit_comments.2010],
       [fh-bigquery:reddit_comments.2011],
       [fh-bigquery:reddit_comments.2012],
       [fh-bigquery:reddit_comments.2013],
       [fh-bigquery:reddit_comments.2014],
       [fh-bigquery:reddit_comments.2015_01],
       [fh-bigquery:reddit_comments.2015_02],
       [fh-bigquery:reddit_comments.2015_03],
       [fh-bigquery:reddit_comments.2015_04],
       [fh-bigquery:reddit_comments.2015_05]
   WHERE
       author = 'Houshalter'
this is where you can run another query: https://bigquery.cloud.google.com/table/fh-bigquery:reddit_c...

I did the same for my account and some comments are definitely missing... The first one that I cross-referenced is missing (5 month old and 1 point comment). But I have other comments with only 1 point and also 5 months old and those are there.

Houshalter
Thank you so much! I'd like to try it myself, but your link redirects me to some getting started page.

Any idea why some comments are missing? I haven't checked to see if they are all there, as far as I know they are.

I know that any in closed subreddits or comments that were removed by mods might be missing. But if you can access the comment's permalink without being signed in, then it should be in your data.

ytdht
No problem. I believe that you need to create a project first, for the page to work.

You were right and that comment has been deleted for some reason.

alexggordon
I used this[0] query to find the top ten[1] most downvoted comments of all time on Reddit.

The most downvoted comment[2] is ironically in iAMA, by a mod of iAMA (ironic because of the recent drama).

I'd find the top ten most upvoted, but I ran out of free bandwidth on BigQuery :(.

[0] https://gist.github.com/alexggordon/7b56353dcf8044a7a5f9

[1] https://drive.google.com/file/d/0Bzxo-UKxFmN-eWticy1BR2tCRDQ...

[2] https://www.reddit.com/r/IAmA/comments/s5guk/iam_bad_luck_br...

etblg
Here's the top ten most upvoted using that query with desc instead of asc:

http://pastebin.com/Zvg6mdjZ

sova
I don't even want to spoil any bit of the surprises that await you, dear readers. Thank you for BigQuerying this
whoopdedo
Could this be used to model the lifecycle of a Reddit post? What makes one thread get more votes and be replied to more often versus another similar thread posted at the same time. Is there a time when a comment is more likely to be voted for, based on wall clock or relative to the start of the thread. What topics are more likely to receive votes than others. Is there a minimum time between when reposts can be posted and still get upvotes. What are the most reposted things on Reddit. Why do people keep voting for reposts.
linkmotif
Does anyone know if this includes the spam? This dataset is obviously very interesting in its own right without spam, but far more useful for me would be all of it, including the spam.

EDIT: regardless, does anyone know if similar datasets with spam?

jedberg
If they grabbed it via the API then no, it doesn't. It also won't include a lot of older stuff which is no longer accessible through the API.
colordrops
Does this data include private subreddits?
visarga
I grep-ed my own comments in it and only shows 39 out of thousands. Is it really complete?
justwannasing
Since 80% of all Reddit comments are vitriol, nonsense by 15-year old kids, why would anyone do this?
ska
How does complaining about the SNR on reddit in a way that decreases the SNR here help anyone?
joelthelion
Does someone have the equivalent for reddit links (with associated title)?