Ask HN: Why Reddit blocks all automated access but has .json for all URLs?

5 points by ksajadi 2mo ago ↗ HN

Reddit is aggressively blocking all automated access (see their robots.txt) and uses a lot of heuristics to block crawlers that do not honor it.

However, at the same time all Reddit URLs can be made machine readable by adding a .json to the end.

Can anyone explain what's the point of that?

7 comments

[ 2.8 ms ] story [ 22.0 ms ] thread

PaulHoule 2mo ago ↗

Circa 2009 or so I was interested in automated link building systems, there were some sites that had no defenses, but I saw enough going on around Reddit that I just didn't want to mess with it.

downbad_ 2mo ago ↗

I just want to know why Reddit keeps banning people for no good reason,while letting bots roam free.

maheenaslam 2mo ago ↗

I guess it's Reddit saying we don’t want just any bot crawling around, but if you need the data in a simple format, here’s an easy way to get it.

brudgers 2mo ago ↗

My guess is they see correlation between patterns of automated access and problematic behaviors.

In other words, the current state is the result of hard won experience not syllogistic reasoning.

ksajadi 2mo ago ↗

That is a good hypothesis but then you’d imagine they could combine it with registered clients to keep crawlers accountable.

jerlam 2mo ago ↗

Reddit locked down their API three years ago, purportedly to monetize their content for AI. This also killed all the free third party apps.

https://arstechnica.com/gadgets/2023/10/reddit-may-block-sea...

dlcarrier 2mo ago ↗

JSON is a popular way to send data around, and the site was probably built around it long ago, using third-party libraries that are difficult to customize.

It does make one wonder for a site so easy to scrape, why there aren't any popular third-party clients that use scraped data, like FreeTube and NewPipe do with YouTube.