HN Explorer

Ask HN: Legality of archiving/re-hosting Reddit content

25 points by dom96 ↗ HN
For content hosted publicly on reddit.com, what is the legality of downloading/scraping that content and re-hosting it on a separate website?

I am aware that the Archive Team is currently archiving Reddit[1]. As far as I understand what they do is legal. But I would like some reassurance.

Are there any good articles on this topic? Contact details for lawyers specialising in this area of the law also welcome.

1 - https://news.ycombinator.com/item?id=36254172

54 comments

[ 7.1 ms ] story [ 118 ms ] thread
neovialogistics
It depends entirely on what nation you're operating in. (And additionally state/territory, if the answer is the United States)
dom96
Personally I'm most interested in the United Kingdom (since that is where I reside). But I would guess HN would be mostly interested in California (and I would be too since that is where Reddit is based).
pyeri
Isn't it true that most implementations of copyright laws (irrespective of country/region) will likely have fair use exceptions for things like parody, journalistic reporting and archiving?
codingdave
Well... no.

Some jurisdictions will have fair use exceptions for some scenarios. I don't think we can assume "most", "likely", or "irrespective of country/region", although it can feel like that if you happen to live in an area where such things are commonly allowed.

anonu
This is probably bad legal advice (above and following...). The EULA clearly states reddit content is owned by reddit. If you break their agreement you break it - and the Governing Law section says you would be put in Federal or state court in California, even if you are a foreign national.

Does a California court have jurisdiction over a foreign national. Probably not - which is entirely possible that Reddit would take a lawsuit to whatever court is closest to the person.

As I stated above, this isn't legal advice either, just my layman's view.

ben_w
I'm also not a lawyer.

My reading of the user agreement[0] is that it distinguishes between "content that belongs to Reddit Inc." (things like the CSS and up/down vote icons) and "content licensed by Reddit" (what users post).

But yeah, lawyering is hard, don't just trust random internet comments like this :)

[0] https://www.redditinc.com/policies/user-agreement

kaba0
Are EULA even enforcible? Also, AFAIK you can never completely revoke your rights to content you yourself created in the EU.
BunnyOSteele
From https://www.redditinc.com/policies/user-agreement

> Except and solely to the extent such a restriction is impermissible under applicable law, you may not, without our written agreement:

> - license, sell, transfer, assign, distribute, host, or otherwise commercially exploit the Services or Content;

Even without being a lawyer that seems pretty clear that it is not legal. Unless you have Reddits written permission, which I guess the Archive Team has

dark-star
on the contrary, I read this as "it is allowed, unless your local laws allow us to restrict you from doing it" ;-)
3np
That "except and solely" is what makes it very much not clear at all. They're basically saying "we make these demands iff applicable law allows us to do so".
TazeTSchnitzel
I would guess that Archive Team don't ask for permission before acting, and certainly don't wait. The sad truth is that copyright law is not on their side. They preserve cultural memory because they believe in it, and they get away with it only because they're a disorganised collective of volunteers, so there's little to gain from suing them.
[deleted]
(comment deleted)
Cloudef
What's with the bunch of reddit crap on the front page recently?
caturopath
https://old.reddit.com/r/OutOfTheLoop/comments/147fcdf/whats...
raldi
Why are we still posting links to claims they are "raising the price of access to their API from being free to a level that will kill every third party app" when they've made completely clear that over 90% of reddit apps won't have to pay a penny, and neither will the remaining 10% if they fix their inefficiency problems?

https://www.reddit.com/r/reddit/comments/145bram/addressing_...

dewey
> when they've made completely clear that over 90% of reddit apps won't have to pay a penny

Because the distribution of Reddit apps is not linear and if 90% of the long tail of Reddit apps (that nobody uses) are not affected that doesn't really change things.

raldi
Besides the difference between "all" and "10%", there's also the difference between "kill" and "require to fix gross inefficiencies".
vlakreeh
These apps aren't grossly inefficient though, they've been substantially lower than the 84k per user per day API limit that reddit has held in the past for these applications, typically being below 500 requests per user per day. Even reddit's official app uses more API requests per day.
raldi
86,400 requests per day is Reddit’s historic per-user limit, but the number was picked as a maximum, not an average or a number any significant fraction of users would be hitting any significant fraction of the time. An app aggregating many users, if reasonably efficient, would be expected to have an average per-user rate over time much much lower than the single-user, single-moment absolute limit.
vlakreeh
> not an average or a number any significant fraction of users would be hitting any significant fraction of the time

Mhm and no major third party reddit app is going anywhere near that with their 500 requests per day per user.

> An app aggregating many users, if reasonably efficient, would be expected to have an average per-user rate over time much much lower than the single-user, single-moment absolute limit.

The per user rate of these apps is 500 a day, which is reasonably efficient. Your point of them being "grossly inefficient" is just false.

BaseballPhysics
Do you have a source for this claim other than spez himself, who has proven to be willing to, uh, let's say bend the truth?
raldi
Who would you consider a sufficiently authoritative source?
BaseballPhysics
The developers themselves are every bit as authoritative on this topic while also not having been caught out lying. Given that, you'll forgive me if I choose not to put my chips with Reddit management.
thebooktocome
Those two claims aren’t mutually exclusive.
raldi
Could you elaborate?
ben_w
I'm not going to claim anything either way about the efficiency of these apps nor the potential (or lack thereof) to improve; but if they're shutting down rather than complying, this suggests that any efficiency improvements are not low hanging fruit.

Of course, now I'm thinking of that time a coworker insisted that $feature couldn't possibly be made more efficient and it was absolutely impossible to optimise it to take less than 20 minutes to complete, and I spent a few hours looking at what it was doing and got it down to 200 milliseconds…

…but I can't project random success anecdotes like that onto projects I'm not being paid to work on.

angoragoats
I’m interested specifically in Apollo, one of the most popular iOS Reddit apps, and the one that I use. Conveniently, its backend was recently open sourced [1] so how it uses the API is public.

Please cite one example of an “inefficiency problem” that Apollo could fix to put it on the path toward not “pay[ing] a penny.”

[1] https://github.com/christianselig/apollo-backend

raldi
Just at a glance, and maybe I'm reading it wrong, but this appears to be issuing six requests every five seconds:

https://github.com/christianselig/apollo-backend/blob/main/i...

See also:

> On March 14th, Apollo made nearly 1 billion requests against our API in a single day, triggered in part by our system outage. After the outage, Apollo started making 53% fewer calls per day. If the app can operate with half the daily request volume, can it operate with fewer?

https://www.reddit.com/r/redditdev/comments/13wsiks/api_upda...

angoragoats
> Just at a glance, and maybe I'm reading it wrong, but this appears to be issuing six requests every five seconds:

I don't believe any of these six operations make any calls to the Reddit API whatsoever. If that's correct, the fact that you raised this as a possible area to optimize indicates to me that you aren't qualified to make any judgments at all about what Apollo could or could not do to reduce its API usage.

> On March 14th, Apollo made nearly 1 billion requests against our API in a single day, triggered in part by our system outage. After the outage, Apollo started making 53% fewer calls per day. If the app can operate with half the daily request volume, can it operate with fewer?

Maybe I'm stupid but I'm not really following how this open-ended speculation indicates that it would be possible for them to operate under the new oppressive API pricing scheme without paying a single cent. Can you please clarify?

raldi
Do you consider yourself “qualified to make any judgments at all” about Apollo’s efficiency?
angoragoats
Not really, which is exactly why I qualified the statement in my reply with “if that’s correct.” But I wasn’t the one making unsubstantiated claims about Apollo’s performance. Still waiting for your examples/clarification.
psychphysic
You're going to be downvoted to oblivion because the only thing HN hates more than Reddit is people pointing out that HN is infatuated with Reddit.
malermeister
Which is funny because HN is, let's be real, basically a subreddit.
InvertedRhodium
In the same way that any forum hosting user-generated content around a specific topic is a subreddit, sure. /b/, for example.

And when you think about it, a subreddit is pretty analogous to a Usenet usergroup.

malermeister
I think the up/downvote mechanism is very reddit-like and not a feature of most forums or usergroups.
ben_w
1. New Reddit API access fees announced

2. Third-party app developers said this is too high, will shut down rather than pay

3. Reddit official response had zero chill

3. a. Repeatedly

4. Significant fraction of subreddits organised a "go private" protest

5. This broke Reddit for everyone else

raldi
For what it’s worth, Reddit’s TOS reads:

You may not: Access, search, or collect data from the Services by any means (automated or otherwise) except as permitted in these Terms or in a separate agreement with Reddit (we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit’s prior written consent is prohibited)

dzek69
TOS is not a law
chrisshroba
But in some (not all!) cases it is a lawful contract, so it's worth consideration.
super256
What if I crawl reddit comments without ever clicking on „agree“/signing up?
raldi
Then the same law would apply as if you recorded a movie in a theater and distributed it without signing anything.
gremlinsinc
How do you figure? In a movie you bought a ticket, that ticket is your contract with the theater, whereas scraping content while logged out is more like filming a police beating.
[deleted]
(comment deleted)
raldi
If you’re record a sporting event off broadcast TV, is it your impression that the law allows you to redistribute the recording freely?
YetAnotherNick
Only if you agree to it.
raldi
If you don't agree to the licensing terms, then you have no legal right to access, archive, or redistribute the content.
neovialogistics
Is it true that people have no right to access any of the content on a website based in the USA without agreeing to that website's TOS if:

* the majority of the content is publically available to users who have not agreed to the TOS, such as when arriving on the website for the first time from a search engine result?

YetAnotherNick
Linkedin lost the case on this AFAIK. It is allowed to scrape linkedin if you haven't accepted TOS.

They could TOS-wall the site, but if they allow to access without TOS then you could access it without agreeing. Also Google definitely access every site and even caches it without accepting any TOS.

linuxftw
There are separate terms for the API, which seem to indicate to me it's legal to use their API to download user content: https://www.reddit.com/wiki/api-terms

According to these terms, the content is owned by the users, and you're not to modify the content. However, if the content is owned by the users, then IMO Reddit cannot really say what you do or don't do with the content, as long as you're not building an application that acts as a proxy to Reddit.

The license is revocable to accessing their API, but they're not licensing you the user content, only the ability to download it. What you can do with that content is likely up to the laws in your jurisdiction. I'd say most content would qualify as public domain, though obviously some content will have copyright protection.

I would do the downloading now before they start charging for the API if you're serious about the project.

dom96
> as long as you're not building an application that acts as a proxy to Reddit.

Presumably you mean an app which displays Reddit.com as-is (i.e. with their logos/icons/UX). But what about a service which acts as a proxy to the Reddit API?

If content is not owned by Reddit and API design cannot be copyrighted then this seems legal, right?

linuxftw
IMO, the API terms look like they're written by a teenager, and you can do just about whatever you want. They admit the content is owned by the users, so not sure what they could actually enforce other than disallowing you to use their API. Once you have the content, it's none of their business at that point.
tssva
When someone posts content to Reddit under the Reddit terms of use they grant Reddit a license to use and distribute the content but ownership rights of the content remains with the poster. If you scrape Reddit and post the content in theory you open yourself up to copyright violation claims from the original poster of the content. The odds that someone is going to sue you for redistributing without permission content they posted to Reddit is likely extremely small but it is not zero.
CM30
Given the users own the content, it'd presumably be up to them whether it can be rehosted or not. Reddit gets a license to display the content, but they don't really have any control over what third parties can do with it.

Personally I don't care if anyone reuses stuff I've posted on Reddit or other forms of social media (forums like Hacker News included), but there's always the possibility that someone might. And if you remove their posts when asked, I doubt most of them will take it any further than that.