Why? Why is a random response code not good enough?
Rather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out.
Just a few days ago, an OSS project that I participate in had its wiki scraped, that looked like some random mac user agent, but suddenly from around the world, everyone looking at the wiki history, reading with crazy speed...
I know very well what an http response code is. But what is the reason for wanting to extend the standard? What exactly makes 450 better than 404? Who benefits from it, and in what form?
The idea behind introducing a new HTTP status code for blocking AI traffic should be to provide a clearer signal to both users and automated systems about the nature of the response. While a 404 indicates that a resource is not found, it doesn't convey the specific reason that the content is being restricted from AI access.
While User-Agents can easily be spoofed by users and automated traffic, as you mentioned in your earlier reply, the HTTP status code would be a clear message from the server.
If it’s a clear message, they will simply avoid your AI identification heuristics. Better to feed them 404s so they eventually give up (or you drop their requests with no response, assuming known IP blocks crawls originate from). The less signal provided, the better, broadly speaking.
7 comments
[ 3.1 ms ] story [ 23.9 ms ] threadRather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out.
Just a few days ago, an OSS project that I participate in had its wiki scraped, that looked like some random mac user agent, but suddenly from around the world, everyone looking at the wiki history, reading with crazy speed...