What is the rationale here? Why is robots.txt not sufficient? Most search engines are already using AI, what separation are you trying to carve out with this question?
(1) indexing public content with the purpose of referring to the source in a search result and
(2) ingesting content (no matter the license) to train models that can reproduce smaller or larger chunks later on.
Google and others might be doing this with data from their crawlers already, but that is besides the point. I think these two uses of publicly available data is fundamentally different and that a distinction should be made.
Interestingly Google have a big advantage here, almost everyone wants to be top-ranked on Google and will gladly allow Google to crawl and scrape even when they aggressively close the door on everyone else crawling and scraping.
Google won't separate these things out, you get ranked high in the search engine _and_ your content goes into the AI machine, or you get neither.
3 comments
[ 2.9 ms ] story [ 19.3 ms ] thread(1) indexing public content with the purpose of referring to the source in a search result and
(2) ingesting content (no matter the license) to train models that can reproduce smaller or larger chunks later on.
Google and others might be doing this with data from their crawlers already, but that is besides the point. I think these two uses of publicly available data is fundamentally different and that a distinction should be made.
Google won't separate these things out, you get ranked high in the search engine _and_ your content goes into the AI machine, or you get neither.