HN Explorer

Ask HN: Processing multiple GBs of data on local machine

3 points by EdwardDiego ↗ HN
I recall a few blog posts on this, but I'm struggling to find them now that I need them.

I have about 100GB of Log4j formatted logs to process to find a particular needle in the haystack, and am looking for a decent way to process those files locally without breaking out Spark in EMR etc.

I recall a few blog posts on this subject, but my search fu is letting me down. Is this ringing bells for anyone?

Thanks in advance :)

6 comments

[ 2.6 ms ] story [ 24.2 ms ] thread
viraptor
I know one related one, but can't remember how to find it either. (I found a response to a response to that post - https://news.ycombinator.com/item?id=8920194)

However, if you're just after a specific needle... why not grep? With 100GB on a local machine you'll either need to do pre-processing / filtering, or you'll be IO-limited. So instead of heavy tools, why not start with basics?

EdwardDiego
Cheers, I was most likely going to, but wanted to stand on the shoulders of giants, as it were, and see what had worked for others, but that gives me a rough time-frame to look around in.

Thanks!

Andys
ripgrep ? It works on my computer at over 1GB per second
speedgoose
You could also use SQLite after some little parsing.
VirusNewbie
Spark works locally and should exploit multicore. I'd use spark instead of writing my own multithreaded system. If you don't like spark there's also apache beam.