> Originality tallied which sites block GPTbot and other AI scrapers by surveying the robots.txt files that websites use to inform automated web crawlers which pages they are welcome to visit or barred from.
Amazing how much you can tell from someone's `robots.txt` file... What does the complete lack of such a file on my site mean? Am I welcoming the AI overlords or fighting the Nazi censors?
3 comments
[ 2.7 ms ] story [ 22.0 ms ] threadAmazing how much you can tell from someone's `robots.txt` file... What does the complete lack of such a file on my site mean? Am I welcoming the AI overlords or fighting the Nazi censors?