Tell HN: Nuuton crawler is active.
The second generation of the Nuuton crawling system (codename: Gravity) is in the wild. I will be testing a lot of different parameters. The speed of the crawl is not quite ideal, and I will be adjusting it to respect your bandwith. If you experience a sudden jolt of bandwith consumption, let me know. It is not on purpose, because this thing is a prototype. It should not happen, because I have included safety mechanism, but its not 100% perfect.
At the moment, I'm only using the test server, so the probability is not very high. Still, let me know if anything. I dont want to damage your things.
Also, if you want Nuuton to return to your site (you have a blog), then include the following line on your robots.txt file:
nuuton: return(n)
where n is the amount of time between crawls in the unix timestamp format (epoch).
I make this post to alert the community of possible issues. I will be posting on twitter/orangethirty any issues that may arise.
Please report any issues to orangethirty@nuuton.com
:)
3 comments
[ 5.4 ms ] story [ 15.3 ms ] thread2. Because this feature ties in with others that I cannot talk about. Its optional, and meant for optimizing for Nuuton.