This is a longstanding UX issue with unlisted URL’s. I’ve encountered this in my own products. Folks don’t know that sharing a link means it definitely will get indexed by Google.
Unfortunately this can’t be fixed via robots.txt — Google won’t proactively crawl the URL, but it does still index pages it finds from external links.
A `noindex` meta tag, however, does prevent pages from being indexed, so that feels like the appropriate solution here.
3 comments of 9
[ 0.29 ms ] story [ 9.1 ms ] threadsite:claude.ai/public/artifacts
Unfortunately this can’t be fixed via robots.txt — Google won’t proactively crawl the URL, but it does still index pages it finds from external links.
A `noindex` meta tag, however, does prevent pages from being indexed, so that feels like the appropriate solution here.