That would require knowing all the user agents that would scrape content, assuming that you want to only exclude AI scrapers and not search engines in general.
OpenAI's user agent is GPTBot[1], I am not sure about the others.
How are you proposing to distinguish between "AI" and "search engines"? Most of the search engines now have a summarizer at the top which is presumably LLM output, and search engines operate on the basis of ML in general.
> Most of the search engines now have a summarizer at the top which is presumably LLM output, and search engines operate on the basis of ML in general.
There is still a difference between scraping content for the purpose of searching it and training on it.
Facebook's response is insightful. They only run ads through their own network.
# Facebook.com/ads.txt
#
# Facebook inventory can only be purchased from Facebook advertising
# systems, and cannot be purchased programmatically through open
# exchanges. Therefore there are no authorized sellers or resellers
# listed in this facebook.com/ads.txt file.
How is this supposed to work when anybody can resell anything?
Suppose someone wants to run a service that will put your ads on all the popular sites, one stop shopping. Facebook only sells ad space directly, but this company would simply take your money and use it to buy ads from Facebook and others, which is totally legitimate. And Facebook interfering with this would have obvious anti-trust implications.
But why would anybody care about that? You want to know if something is real, not if it's authorized. It's like buying gray market products. The important thing is to be able to tell if they're fraudulent/counterfeit, not to be able to tell if they were original sold into your region, which the customer doesn't give a lick about.
19 comments
[ 3.5 ms ] story [ 55.7 ms ] threadThat would require knowing all the user agents that would scrape content, assuming that you want to only exclude AI scrapers and not search engines in general.
OpenAI's user agent is GPTBot[1], I am not sure about the others.
[1] https://news.ycombinator.com/item?id=37030568
There is still a difference between scraping content for the purpose of searching it and training on it.
https://www.spotify.com/ads.txt
Suppose someone wants to run a service that will put your ads on all the popular sites, one stop shopping. Facebook only sells ad space directly, but this company would simply take your money and use it to buy ads from Facebook and others, which is totally legitimate. And Facebook interfering with this would have obvious anti-trust implications.