Show HN: Fetchcheck – send a signed crawler to any site and see if it gets in (sitedex.dev)

If you were ever curious whether a site allows signed crawlers, you can use this tool. This tool respects the website's robots.txt directive.

You will be surprised how many sites allow all robots, but then block crawlers