Ask HN: Is anyone working on a Reddit archive?

22 points by nidnogg 3y ago ↗ HN

Although I disagree with most of the criticisms leveraged towards the platform, I still readily depend on it for a lot of day-to-day resources and general questions, many times less technically oriented.

While I wouldn't mind losing the system there in the long run, I think the state of posts before this upheaval was very valuable as a reference.

Like the title says - has anyone done anything like a Reddit "takeout" yet?

6 comments

[ 3.9 ms ] story [ 25.4 ms ] thread

minimaxir 3y ago ↗

Pushshift was the Reddit archive but apparently recent agreements with Reddit may have changed that.

Anyone else creating a Reddit archive will likely get a C&D.

speedgoose 3y ago ↗

Pushift is supposed to come back with a new agreement with Reddit. No precise date was given though, and it’s likely not going to be accessible to everyone but only moderators, researchers, and perhaps a few businesses.

uniqueuid 3y ago ↗

Yes.

There is the pushift dataset covering posts and comments through 2022 [1].

And the ArchiveTeam has begun crawling reddit as well some time ago [2]

[1] https://old.reddit.com/r/pushshift/comments/10bwxke/updated_...

[2] https://news.ycombinator.com/item?id=36254172

cookiengineer 3y ago ↗

I was focussing mostly on cyber security related subreddits because the vulnerability and exploit discussions were of great value to me.

I built a little scraper in golang that stores the JSON data (instead of the HTML which the archive warrior stores) to save hdd storage. [1]

The problem with reddit's API is that it only shows 1000 entries over 10 pages in every api. Meaning hot/top/new, and search results are limited. If you have more links related to the keyword, you won't discover more.

So you need a very specific keyword list to be able to discover more posts, and search each subreddit for each entry in the keyword list.

[1] https://github.com/cookiengineer/reddit-archivar

simonblack 3y ago ↗

It's next on my list after I finish the MySpace archive.

Seriously, why would anybody do this? Reddit has such a high noise-to-signal ratio that it would be a waste of resources. There may be value in keeping an archive of some individual subreddits, but not the main bulk of Reddit itself.

eimrine 3y ago ↗

Probably the most of the noise is inside obvious places like r/fun or r/music. And most of the signal is in some micro subreddits with few hundreds of subscribers.