What AI means for content creators: how to use it and how to fight it (medium.com) 1 points by lucas_crocker 2y ago ↗ HN
[–] lucas_crocker 2y ago ↗ Also, does anyone have anything I missed? I know some people were suggesting not letting google index everything by updating your robot.txt file. [–] dazc 2y ago ↗ robots.txt isn't an ideal way of preventing pages being indexed.X-robots http headers are more reliable: https://developers.google.com/search/docs/crawling-indexing/...Regarding AI, it's a bit more tricky since it isn't going to abide by your rules. Cloudflare have tools: https://blog.cloudflare.com/ai-bots/How effective these are though, IDK? [–] lucas_crocker 2y ago ↗ Oh that's interesting. Yea, paywalls might end being the most effective way of preventing AI bots, but even those could be circumvented. It's a tricky problem for sure.
[–] dazc 2y ago ↗ robots.txt isn't an ideal way of preventing pages being indexed.X-robots http headers are more reliable: https://developers.google.com/search/docs/crawling-indexing/...Regarding AI, it's a bit more tricky since it isn't going to abide by your rules. Cloudflare have tools: https://blog.cloudflare.com/ai-bots/How effective these are though, IDK? [–] lucas_crocker 2y ago ↗ Oh that's interesting. Yea, paywalls might end being the most effective way of preventing AI bots, but even those could be circumvented. It's a tricky problem for sure.
[–] lucas_crocker 2y ago ↗ Oh that's interesting. Yea, paywalls might end being the most effective way of preventing AI bots, but even those could be circumvented. It's a tricky problem for sure.
4 comments
[ 4.1 ms ] story [ 29.2 ms ] threadX-robots http headers are more reliable: https://developers.google.com/search/docs/crawling-indexing/...
Regarding AI, it's a bit more tricky since it isn't going to abide by your rules. Cloudflare have tools: https://blog.cloudflare.com/ai-bots/
How effective these are though, IDK?