Every search engine other than Google has stopped indexing pages from Reddit.
Google has not commented on whether they plan to respect it. Rich Results[0] say they're using a version from June 25. The new version was last modified July 1.
Google is paying Reddit $60 million to not have to respect it. The media reports it's for training data for AI, but none of us have read that contract.
Yes, it makes sense from a 600 million monthly visitors point of view, but that would still be a pretty interesting development if they don't respect it.
any other company, specifically Microsoft/Bing and OpenAI, are free to pay reddit for the same privilege, so I don't see how it could be considered that.
They can and will probably respect it and still be able to have reddit results.
“User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it..
But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages.
But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this.
Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$
Yup, this is what I thought would happen. It wouldn't surprise me if reddit goes a step further and requires login to view pages.
AI is the death-knell of the web as we've known it for the past three decades. Once freely available information will retreat behind login walls and charge bigco's for access to train their models.
I wonder if some standardised data API will be settled upon, perhaps it already exists?
that's not ai but the result of the linked in scraper lawsuit. if it's not behind a login, it's fair game to be scraped. Behind a login, the site operator doesn't have to let you scrape them.
The rugpull is the community giving content for free to these corporations. Perhaps for the next big public forum we should require better licensing. Or just avoid these VC-backed sites altogether and go back to something like Usenet.
Interesting. There have been all sorts of "Reddit's content is key to search results, adding 'reddit' to search results makes them good" stories. And there's been a lot of talk about how some of the big ML makers, notably Google, depend on Reddit's content to train their AI. And Google has that recent $60 million deal for content. So clearly Reddit's execs have been talking about how their content is valuable and they shouldn't give it away for free.
But at the same time, blocking search engines from indexing your social media site is a dangerous game. Any search engine that respects this is gonna effectively de-list Reddit. That's no good for views, and views is what makes Reddit money. Presumably they have negotiated private deals with Google and probably Microsoft for this and are trying to sell their data to ML companies, because otherwise this would seem suicidal.
Kind of a shame. The information is still going to get shared around to all the giant corporations, but Reddit will presumably make it harder to access for all the little guys. And the more they tie the content to dollars, the more managers on the inside will start doing stupid things to try and generate more of whatever the most valuable kinds of content are.
In my personal experience, Reddit has become mainstream in the past few months/years. Even my completely non-tech friends use it, having not even heard about it 1-2 years ago.
So maybe they can afford to be delisted from search engines, because the site is addictive enough to bring back its users – and new ones because it's mentioned everywhere.
Like Discord or Facebook, where most content is also unavailable to search engines (although they probably rely much more on the social aspects).
> In my personal experience, Reddit has become mainstream in the past few months/years. Even my completely non-tech friends use it, having not even heard about it 1-2 years ago.
Since the moderator revolt, I've noticed an increase in claims that there is an overwhelming volume of AI-generated content on Reddit coupled with boosting. Either that started with reddit's strike-busting efforts or was made more obvious by it.
Since then, this is becoming far worse. Reddit's front page is dominated by random reposts by random low-karma user accounts that all of a sudden create the top post in the whole Reddit universe. Then there are reposts by other random low-karma user accounts on other subreddit that benefit from the same exact treatment.
I'm convinced that the reddit revolt motivated reddit to take a more direct and involved approach to generating and boosting content, and now the effort targets specific segments. This also includes the creation and boosting of subreddits dedicated to mainstream topics. Regardless of this coming from reddit itself or marketing/PR companies, I would not be surprised if the bulk of this content was machine-generated fluff.
They are for sure fucking around with peoples personal feeds. I am forever bombarded with "new" posts from the subreddits I sub to in my main feed.
For any sub where people ask a lot of questions it's very annoying. I've already unsubbed from a bunch of subs because I didn't want to see the inane/clueless questions being asked.
I have to assume they're promoting these dumbass questions into peoples feeds because they're precisely the type of content AI will love. People asking questions and humans giving answers, but boy it sure does make the experience as a user worse.
Reddit has never been organic. Theres a video by an ex karma farmer explaining his strategies for creating the most engagement. Surprise, 2016 was a huge boost to his metrics for obvious reasons.
> "Reddit's content is key to search results, adding 'reddit' to search results makes them good" stories.
The biggest issue for me personally is that Reddit's built-in search is about as good as it's built-in mobile video player.
Which is to say it is one of the worst pieces of software ever foisted upon humanity despite having been an objectively solved problem with off-the-shelf solutions available for decades.
This wouldn't really bother me if Google weren't the only way to effectively search Reddit.
I’m not convinced Reddit wants web traffic at all.
Every single time I have the misfortune of trying a Reddit link on mobile, I have to fight prompt after prompt after prompt telling me, no begging me, to use the app. It’s like they hate me so much for having the audacity to not install their app, it’s insane.
I guess they only bother keeping the web site around at all is for the faint hope that it’ll act as a funnel to their app? They certainly don’t want anyone actually using the website, they’ve made that perfectly clear.
If anyone is curious how deeply destructive, or how deeply approved by the NSA, or how deeply self-sabotaging for society the modern AI training data pipeline is becoming I’d refer them to SB 1047.
OpenAI is openly collaborating with the NSA, Google is manipulating the definition of a web crawl, Anthropic has installed a bunch of humanitarians from Jump Trading as the leading mech interp group that makes strident claims about how all this stuff works based on weights you do not and never will have access to.
They’re telling you: “And you will do nothing, because you can do nothing.”
I invite you to join me in proving that we can in fact do something.
The public is distressingly indifferent to both the handling of their data and their pricing power relative to the modern cartel.
The enterprise is keenly aware that they will be decades recovering from how badly they got put over a barrel on cloud. The enterprise cares deeply about where their privileged and in many cases legally regulated data goes and is stored.
Big business needs an ally in the tech game, and for once that is looking likely to trickle down to John Q Taxpayer.
Agreed. None of the modern AI shops has anything like the level of elite technology expertise that the serious Unix companies did in the 80s or 90s: the Sun Microsystems people would eat OpenAI and still be hungry.
And they got their ass kicked by Linux. Not even a close, hard-fought match. Brutally humiliated and driven out of business and sold off for parts.
The reckless contempt that the modern AI cartel holds open models in, and GGUF in, and even Apple in on this stuff is staggering and makes me wonder if any of the leadership even knows the history.
No one has ever tried to tell the technology community what they can and can’t do on their own machine and lived to tell the story. The people pushing DRM over the years need a restraining order against their own employees.
Bill Gates is like, a thousand times smarter and more ruthless than any of these people and he backed off in his prime.
But the next 20 hits are just more ways Google is pushing the envelope on creative interpretations of robots.txt
Yesterday I asked Sonnet 3.5 how to compete with Anthropic and it was like, point out that no serious science can be done without the ability to verify it. These things are pretty reliable at pointing out why they are bad news.
The source you picked was 2,000 words and did not contain define or any related word. Respect other people's time please.
The only creative interpretation of robots.txt I saw was allowing people to opt out of AI uses by disallowing Google-Extended though there is no Google-Extended crawler. This did not manipulate the definition of a web crawl.
Technically this title violates HNs title policy as it should just be "Reddits robot.txt" or something, but "Reddits robot.txt changed" is more useful. I'm curious to see if mods change it.
It's even worse. Many subreddits were taken over by bad actors. When they saw a popular community, they found at what times the mods were inactive (e.g. late night or weekends) and posted evil stuff. Then went to cry to the admins about the evil stuff staying up for hours and got their friends to become mods of the community. In the case of reddit, many people suspect the admins and 3 letter agencies were in the know, or at least played along to their advantage.
And if you self-host to avoid the risk of handing over your community to those sites under control, you get flooded with DDoS and hacking attacks (including state-sponsored actors). Sad times.
80% of my Google searches for other people's opinions now end with "site:reddit.com", and there is surprisingly quite a few of them. The alternative is Reddit's own search and it tends to produce less relevant results.
# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
User-agent: *
Disallow: /
lol what? Just 20-30 minutes ago I saw this at #52, and tried to find it again now and see it #468 https://archive.ph/ReTR5 but I wonder if that is algorithmically natural or whatnot, lol
60 comments
[ 0.14 ms ] story [ 173 ms ] threadGoogle has not commented on whether they plan to respect it. Rich Results[0] say they're using a version from June 25. The new version was last modified July 1.
[0]: https://search.google.com/test/rich-results
“User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it..
But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages.
But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this.
Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$
AI is the death-knell of the web as we've known it for the past three decades. Once freely available information will retreat behind login walls and charge bigco's for access to train their models.
I wonder if some standardised data API will be settled upon, perhaps it already exists?
But at the same time, blocking search engines from indexing your social media site is a dangerous game. Any search engine that respects this is gonna effectively de-list Reddit. That's no good for views, and views is what makes Reddit money. Presumably they have negotiated private deals with Google and probably Microsoft for this and are trying to sell their data to ML companies, because otherwise this would seem suicidal.
Kind of a shame. The information is still going to get shared around to all the giant corporations, but Reddit will presumably make it harder to access for all the little guys. And the more they tie the content to dollars, the more managers on the inside will start doing stupid things to try and generate more of whatever the most valuable kinds of content are.
Yeah see the API changes from last year and their pre-IPO investor roadshow. This ground is well trodden.
So maybe they can afford to be delisted from search engines, because the site is addictive enough to bring back its users – and new ones because it's mentioned everywhere. Like Discord or Facebook, where most content is also unavailable to search engines (although they probably rely much more on the social aspects).
Since the moderator revolt, I've noticed an increase in claims that there is an overwhelming volume of AI-generated content on Reddit coupled with boosting. Either that started with reddit's strike-busting efforts or was made more obvious by it.
Since then, this is becoming far worse. Reddit's front page is dominated by random reposts by random low-karma user accounts that all of a sudden create the top post in the whole Reddit universe. Then there are reposts by other random low-karma user accounts on other subreddit that benefit from the same exact treatment.
I'm convinced that the reddit revolt motivated reddit to take a more direct and involved approach to generating and boosting content, and now the effort targets specific segments. This also includes the creation and boosting of subreddits dedicated to mainstream topics. Regardless of this coming from reddit itself or marketing/PR companies, I would not be surprised if the bulk of this content was machine-generated fluff.
For any sub where people ask a lot of questions it's very annoying. I've already unsubbed from a bunch of subs because I didn't want to see the inane/clueless questions being asked.
I have to assume they're promoting these dumbass questions into peoples feeds because they're precisely the type of content AI will love. People asking questions and humans giving answers, but boy it sure does make the experience as a user worse.
The biggest issue for me personally is that Reddit's built-in search is about as good as it's built-in mobile video player.
Which is to say it is one of the worst pieces of software ever foisted upon humanity despite having been an objectively solved problem with off-the-shelf solutions available for decades.
This wouldn't really bother me if Google weren't the only way to effectively search Reddit.
Every single time I have the misfortune of trying a Reddit link on mobile, I have to fight prompt after prompt after prompt telling me, no begging me, to use the app. It’s like they hate me so much for having the audacity to not install their app, it’s insane.
I guess they only bother keeping the web site around at all is for the faint hope that it’ll act as a funnel to their app? They certainly don’t want anyone actually using the website, they’ve made that perfectly clear.
OpenAI is openly collaborating with the NSA, Google is manipulating the definition of a web crawl, Anthropic has installed a bunch of humanitarians from Jump Trading as the leading mech interp group that makes strident claims about how all this stuff works based on weights you do not and never will have access to.
They’re telling you: “And you will do nothing, because you can do nothing.”
I invite you to join me in proving that we can in fact do something.
The enterprise is keenly aware that they will be decades recovering from how badly they got put over a barrel on cloud. The enterprise cares deeply about where their privileged and in many cases legally regulated data goes and is stored.
Big business needs an ally in the tech game, and for once that is looking likely to trickle down to John Q Taxpayer.
And they got their ass kicked by Linux. Not even a close, hard-fought match. Brutally humiliated and driven out of business and sold off for parts.
The reckless contempt that the modern AI cartel holds open models in, and GGUF in, and even Apple in on this stuff is staggering and makes me wonder if any of the leadership even knows the history.
No one has ever tried to tell the technology community what they can and can’t do on their own machine and lived to tell the story. The people pushing DRM over the years need a restraining order against their own employees.
Bill Gates is like, a thousand times smarter and more ruthless than any of these people and he backed off in his prime.
Explain please.
https://www.theverge.com/24067997/robots-txt-ai-text-file-we...
But the next 20 hits are just more ways Google is pushing the envelope on creative interpretations of robots.txt
Yesterday I asked Sonnet 3.5 how to compete with Anthropic and it was like, point out that no serious science can be done without the ability to verify it. These things are pretty reliable at pointing out why they are bad news.
The only creative interpretation of robots.txt I saw was allowing people to opt out of AI uses by disallowing Google-Extended though there is no Google-Extended crawler. This did not manipulate the definition of a web crawl.
The question was not why are LLMs bad.
What are they thinking?
Side note: They seem to serve other robots.txt for different User-Agents & IPs: https://merj.com/blog/investigating-reddits-robots-txt-cloak...
https://www.reddit.com/r/RedditAlternatives/wiki/index/
I think a moderated mirror of certain subs would be a better solution. Because people like freedom but love their cats and memes much more.
Ironically the only off-site that ever really took off is the snuff film site which has 2.3M users now.
Because that is Reddit's magic ingredient and you will have to do it 24/7 from day one.
And if you self-host to avoid the risk of handing over your community to those sites under control, you get flooded with DDoS and hacking attacks (including state-sponsored actors). Sad times.
80% of my Google searches for other people's opinions now end with "site:reddit.com", and there is surprisingly quite a few of them. The alternative is Reddit's own search and it tends to produce less relevant results.