Without details on the task he's running and the setup of his cluster, it's hard to know how meaningful these results are. Using a variation of the example, I've seen roughly 80% improvement with each doubling of the cluster size (testing up to 100 workers).
How big was the dataset that you saw the 80% improvement? Remember this test was done on a relatively small dataset, 28 MB, so it may have reached a diminishing point a lot faster than with a larger dataset.
3 comments
[ 3.8 ms ] story [ 12.7 ms ] thread