HN Explorer

7 comments

[ 3.4 ms ] story [ 27.0 ms ] thread
creekorful
Release 1.0.0 will be out very soon. Check it out guys!
lrossi
Can you share some examples of what it finds? As long as it is safe to post that.
creekorful
I have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces.

There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.

creekorful
I'll certainly publish an index of interesting (legal) hostnames on my GitHub account.
visox
nice, what is actually needed to crawl onion sites ?

i am not a go dev, would try it with scala/java

creekorful
Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.
visox
cool i look into that some time.