Oh man ! I wish someone had told me this before. Back when i did research for my BSc thesis, I had an 24 core machine and wrote a script to create 24 bash scripts to split up my task which i would than run in parallel.
Sometimes I have the felling that I am wasting time when I check HN too often, but stuff like this makes me think otherwise.
Like, async spidering with parallel is a one-liner, but comparatively, have you seen how much absurdly syntaxed pycurl boilerplate you need? It's crazy. I for one will take the simpler method any day.
8 comments
[ 3.3 ms ] story [ 45.5 ms ] threadUnlike xargs? That's just plain wrong, see xargs' -P flag.
http://news.ycombinator.com/item?id=1801186
Sometimes I have the felling that I am wasting time when I check HN too often, but stuff like this makes me think otherwise.
I think that Parallel is more interesting as an instant server-farm for "taco bell programmers" -- q.v. http://teddziuba.com/2010/10/taco-bell-programming.html and http://news.ycombinator.com/item?id=1818816 for the HN discussion -- as per Mr. Dziuba's definition.
Like, async spidering with parallel is a one-liner, but comparatively, have you seen how much absurdly syntaxed pycurl boilerplate you need? It's crazy. I for one will take the simpler method any day.