Show HN : I have developed a robots.txt full text search engine. (robots.dnsdigger.com)

1 points by Joyfield 12y ago ↗ HN

As a sub project of my pet project DNSDigger.com i have developed a search engine that index robots.txt (http://en.wikipedia.org/wiki/Robots_exclusion_standard). I actually don't know what poeple could use this for but i suspect poeple that are into SEO could use it and black/white-hat hackers(?). What i need help with is VPS/$$ for spidering. And as i sit on a considereble index of domainnames (though DNSDigger.com 300m hosts) i thought that if you want a TLD indexed sooner than later you could sponsor it with either VPS or Paypal. See it as a strange "kickstarter". And if you think the "design" suck (which it does because there is a lot of things i suck at and design is one of them) you are free to submit anything better. I can put up a banner for your service a limited time. I am adding new hosts/TLD all the time.

And if you have to ask what this is you will not have any use for it even if you did ;)

2 comments

[ 2.7 ms ] story [ 14.6 ms ] thread

Xeroday 12y ago ↗

Very cool project. I'm curious as to how you're getting all these robots - are you scraping them yourself?

Joyfield 12y ago ↗

Downloading them as we speak. I have a big list of hosts/domains i have collected through spidering for my DNSDigger.com. This is a hobby project that has grown a bit over my head hehe. And there is no scraping needed. Robots.txt is just simple textfiles. Download and parse, repeat a couple of million times and build an index :)