zeppelin_7

↗ HN profile [ 84.2 ms ] full profile
Karma
0
Created
()
Submissions
0
  1. If you were ever curious whether a site allows signed crawlers, you can use this tool. This tool respects the website's robots.txt directive. You will be surprised how many sites allow all robots, but then block crawlers

  2. I run a crawler (SitedexBot) that signs every request using Web Bot Auth. I was trying to debug my HTTP message signatures directory, and ended up making the tool free/open. No data is stored after the keycheck. You can…