> LLMs are the next generation’s search layer. They’re already generating massive amounts of pipeline for the companies and websites that have gotten good at getting their content displayed in LLMs
It’s not dumb because Googlebot follows the robots.txt rules. That is the sincere crux of it all. No one is going to casually open their site up to Llms that are blatantly scraping their site to then use their information to displace them.
Not blocking violent, bad-actor scrapers is dumb. Letting through bad-actor scrapers because a bunch of rich people want to make it the norm is dumb.
Llms are not directing traffic to the sites and that is the tradeoff that site owners allow with Googlebot. Even if Perplexity or Claude will provide a source, the Llm user is most likely not asking/clicking for it 99% of the time.
That is basically several paragraphs to just say "well you should just adapt to the new world instead of pushing against bad practices". There is barely any "why" actually said here.
We just had the article about how AI search is leading to less clicks, so where is that supposed "pipeline"?
Also completely ignores how you may not want your information to be misconstrued (lied basically) to the user with a helpful link telling them where the source is, but they may never click through. And worse if they know that the information being told to them is wrong, they may then think it was because your site was wrong and trust you less, all without ever clicking that link.
“Providing high quality content that LLMs will actually cite is the new game in town.”
That is not my job nor is it my goal. These companies are taking my work, repurposing it, and selling it under the assumption that because they can access it they can sell it.
Maybe the OP should leave their house door open so people can come in and use his couch. The new game in town is to let other people use your couch.
The mental gymnastics in this post qualify for the Special Olympics.
This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. People are blocking LLMs with the precise intent of trying to preserve the flow to their content, be it commercially, reputationally, whatever.
> how many of you wouldn’t hook up your website to Google?
If there was a paid-only search engine with dubious ethics practices that was overwhelming my site with traffic in order resell search trained off of (among other things) my personally generated content, I would absolute block it.
LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.
> it also cuts you off from the fastest-growing distribution channel on the web.
I haven't seen the needle tip at all in my acquisition channels from LLMs. Unless you're a household name or very large, LLMs aren't going to shill for your business.
> most LLMs have an agentic web-search component that will actively generate links
Totally. Which is why I don't care if the LLMs index it. Let web content search be good, and lead LLMs to good content; product placement in LLM weights ain't what I'm gonna optimize for, or even permit, if it comes at a cost to me and my infra.
Even if we take the argument at face value, we should allow LLMs to train their models for free, on the backs of real people's work, just so that there's a chance that they actually improved well enough to replace humans, all that just to have a temporary boost on search discovery of our content.
Not to mention LLMs still spew a lot of badly wrong results (no I will not anthropomorphize the models, they're not ready yet).
This is one heck of a poison chalice. 王先生，你願意喝這杯鶴酒嗎？
Your computer doesn't have the right to scrape what I say or do anything with it.
I know one of the primary reasons that I do anything online is to provide an outlet for someone else to see it. If I didn’t want someone else to see it, I’d write it down on my notebook, not on the public web.
Sounds like the same schpiel from the anti-privacy advocates who think that we should all expose everything we're doing because "you should have nothing to hide".
This article was written for Wired by Moxie Marlinspike in 2013, who went on to later develop the Signal protocol.
I don't want my thoughts or ideas spread across the web promiscuously. The things I say publicly are curated and full of context. That's why I have my own website, and don't post elsewhere.
I'm not playing the same game you are, which appears to be to post liberally and have loose thoughts to maximize "reach".
Nonsensical article. Even if your goal is to create something on the web “for others” (as the article asserts), when 99.9% of your costs go to serving LLM crawlers, it puts that very objective at risk.
The whole thing about LLM is training on content other people created, redirecting that traffic to you and ultimately earn money on it. The whole thing about LLM being pushed everywhere is to get free training data too.
As an amateur blogger, I would not like LLMs to "steal" my content, display the users the needed pieces they are looking for, while leaving me with zero visitors. The reason I write is to convey a particular message, which the meaning of gets lost, or worse communicated wrongly, due to LLMs.
As an online business owner, I do see both ChatGPT and Perplexity as referrers to my business, meaning that potential customers ask LLM a question/service recommendation, and LLM is directing them to my service, and I would not like to lose this vertical of organic customer acquisition.
---
On a completely different note, medium should die as a platform, together with substack. The amount of intrusive popups, "install our app" bars, and paywalls is just insane. Bloggers, especially technically savvy ones, should be able to host their own blog.
This article is based on the false assumption that use of a site by an LLM directs any user traffic to it whatsoever.
Why would anyone choose to anonymously and freely provide content to LLMs? Actually the only use case for that is deliberately seeding misinformation. Which is likely already happening and will soon be the majority of the content accessible to LLMs regardless of what blocking legitimate content providers choose to use.
The first sentence of the article is literally wrong as it conflates LLM and the search part of a RAG (retrieval augmented generation, when you mix a web search and an LLM).
Blocking bots cuts you off from the next-generation search, because it cuts you off from search at all. So far, blocking LLM simply prevents you from being part of the training dataset, which is not the same thing.
Please stop upvoting such bad content it really makes Hackernews a terrible place for staying informed on LLMs.
Once again I see someone mistaking an LLM regurgitating (in a right, wrong, misleading way, who knows) your content for people "accessing" your content. If the LLM sits between me and my reader and acts as a filter (because the information is rehashed, because maybe sometimes doesn't reference me), basically my goal is to provide information to tools for other companies to make money?
I don't write for money, but if you remove also the basic human interaction reader-writer, I might as well really write in my notes.
35 comments
[ 1.1 ms ] story [ 44.7 ms ] thread[citation needed]
Not blocking violent, bad-actor scrapers is dumb. Letting through bad-actor scrapers because a bunch of rich people want to make it the norm is dumb.
Llms are not directing traffic to the sites and that is the tradeoff that site owners allow with Googlebot. Even if Perplexity or Claude will provide a source, the Llm user is most likely not asking/clicking for it 99% of the time.
We just had the article about how AI search is leading to less clicks, so where is that supposed "pipeline"?
Also completely ignores how you may not want your information to be misconstrued (lied basically) to the user with a helpful link telling them where the source is, but they may never click through. And worse if they know that the information being told to them is wrong, they may then think it was because your site was wrong and trust you less, all without ever clicking that link.
That is not my job nor is it my goal. These companies are taking my work, repurposing it, and selling it under the assumption that because they can access it they can sell it.
Maybe the OP should leave their house door open so people can come in and use his couch. The new game in town is to let other people use your couch.
The mental gymnastics in this post qualify for the Special Olympics.
[1]: https://www.pewresearch.org/short-reads/2025/07/22/google-us...
If there was a paid-only search engine with dubious ethics practices that was overwhelming my site with traffic in order resell search trained off of (among other things) my personally generated content, I would absolute block it.
LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.
> it also cuts you off from the fastest-growing distribution channel on the web.
I haven't seen the needle tip at all in my acquisition channels from LLMs. Unless you're a household name or very large, LLMs aren't going to shill for your business.
> most LLMs have an agentic web-search component that will actively generate links
Totally. Which is why I don't care if the LLMs index it. Let web content search be good, and lead LLMs to good content; product placement in LLM weights ain't what I'm gonna optimize for, or even permit, if it comes at a cost to me and my infra.
How about the fact that Google (ideally) sends users to you rather than sharing your work unattributed?
Not to mention LLMs still spew a lot of badly wrong results (no I will not anthropomorphize the models, they're not ready yet).
This is one heck of a poison chalice. 王先生，你願意喝這杯鶴酒嗎？
Your computer doesn't have the right to scrape what I say or do anything with it.
Sounds like the same schpiel from the anti-privacy advocates who think that we should all expose everything we're doing because "you should have nothing to hide".https://archive.is/WjbcU
This article was written for Wired by Moxie Marlinspike in 2013, who went on to later develop the Signal protocol.
I don't want my thoughts or ideas spread across the web promiscuously. The things I say publicly are curated and full of context. That's why I have my own website, and don't post elsewhere.
I'm not playing the same game you are, which appears to be to post liberally and have loose thoughts to maximize "reach".
I guess that’s the problem - search being only a component.
Is the possible search traffic worth having your content become part of an LLM’s training set and possibly used elsewhere?
I guess the answer depends on the content and the website’s business model.
As an amateur blogger, I would not like LLMs to "steal" my content, display the users the needed pieces they are looking for, while leaving me with zero visitors. The reason I write is to convey a particular message, which the meaning of gets lost, or worse communicated wrongly, due to LLMs.
As an online business owner, I do see both ChatGPT and Perplexity as referrers to my business, meaning that potential customers ask LLM a question/service recommendation, and LLM is directing them to my service, and I would not like to lose this vertical of organic customer acquisition.
---
On a completely different note, medium should die as a platform, together with substack. The amount of intrusive popups, "install our app" bars, and paywalls is just insane. Bloggers, especially technically savvy ones, should be able to host their own blog.
LLMs are being blocked by standard bot detection - and the use cases are very much the same. People want smarter bots for the same shitty use cases.
Why would anyone choose to anonymously and freely provide content to LLMs? Actually the only use case for that is deliberately seeding misinformation. Which is likely already happening and will soon be the majority of the content accessible to LLMs regardless of what blocking legitimate content providers choose to use.