Ask HN: Anyone interested in contributing to Domo web crawler project?

2 points by hinoglu ↗ HN
Hi Hn,

http://github.com/hinoglu/Domo/ is a python based remotely manageable web crawler based on Python, multiprocessing, pyro, lxml and pycurl.

It's main purpose is to be able provide a lightweight python based alternative to archive.org's heritrix. Main structure is similar to heritrix and Domo provides almost similar but somewhat lesser features.

Though Domo is a web crawler/archiver, it can be used for anything that requires remote multiinstance / multiprocess job management features, thanks to Pyro and multiprocessing libraries.

I'd be glad if anyone would like to contribute or join the project.

0 comments

[ 3.1 ms ] story [ 11.7 ms ] thread

No comments yet.