HN Explorer

10 comments

[ 2.9 ms ] story [ 38.2 ms ] thread
bitL
Yikes, I had to stop when the author mentioned MapReduce for computing shortest distance. That's one of the problems for which MapReduce approach is extraordinarily bad.
taeric
Sadly.... I'm noticing that folks that have done one MapReduce job are convinced it is the way to go for all jobs. It is scary just how fad based development can be.
dimi-31
To be fair, abstracting that fact away, the rest of the post is still an interesting thought experiment.
ignoramous
A better approach at the scale being discussed in the article would be to...?
karussell
What does 'scale' mean :) ? Solving graph problems are best done in-memory with a big machine (RAM!), otherwise you are an order of magnitude slower (at least) if you try to distribute. But if you really have no choice I would have a look if some of the graph databases has a good distributed model. My gut feeling tells me that even a bad approach there is faster than spark ...
ddrum001
I don't think you have a choice, you'll have to use MapReduce if the data is too big to fit into memory. I believe that's what Facebook and Google do:

https://www.facebook.com/notes/facebook-engineering/scaling-...

http://googleresearch.blogspot.com/2009/06/large-scale-graph...

bitL
Your 2nd link describes Pregel, Google's distributed graph database specifically built for these kinds of tasks.

They were using Map Reduce prior to that, but it was a cascading mess.