Tell HN: Nuuton crawler is active.

3 points by orangethirty ↗ HN
The second generation of the Nuuton crawling system (codename: Gravity) is in the wild. I will be testing a lot of different parameters. The speed of the crawl is not quite ideal, and I will be adjusting it to respect your bandwith. If you experience a sudden jolt of bandwith consumption, let me know. It is not on purpose, because this thing is a prototype. It should not happen, because I have included safety mechanism, but its not 100% perfect. At the moment, I'm only using the test server, so the probability is not very high. Still, let me know if anything. I dont want to damage your things.

Also, if you want Nuuton to return to your site (you have a blog), then include the following line on your robots.txt file:

nuuton: return(n)

where n is the amount of time between crawls in the unix timestamp format (epoch).

I make this post to alert the community of possible issues. I will be posting on twitter/orangethirty any issues that may arise.

Please report any issues to orangethirty@nuuton.com

:)

3 comments

[ 5.4 ms ] story [ 15.3 ms ] thread
What is Nuuton? Why invent a proprietary robots.txt feature instead of the (common but non-standard) "Crawl-delay:"?
1. Nuuton is a search engine.

2. Because this feature ties in with others that I cannot talk about. Its optional, and meant for optimizing for Nuuton.