Simple web-crawler for Node.js (github.com)

2 points by DamonOehlman 14y ago ↗ HN

1 comment

[ 0.32 ms ] story [ 7.5 ms ] thread

zoowar 14y ago ↗

Show some respect for the web and add robots.txt and sitemap.xml support.