Modin uses Ray, a distributed computation library. There was a similar article on HN a year ago that hyped "making pandas faster" by replacing it with Ray:
I just learned about modin as well.
Based on this blog post [1], it seems to be a bit slower for small datasets. But this was 6 months ago. Now the documentation claims that it is suitable for datasets from 1kB to 1TB.
The Modin project should work on merging the implementation to the original Pandas project, so that parallel / non-parallel algorithms could be mixed. Drop in replacements don't work, as they are not 100% compatible with the original project.
I would prefer to pass an optional parallel=true parameter to some functions in the API, or have a configuration setting that can fall back to non-parallel implementation.
12 comments
[ 0.20 ms ] story [ 41.7 ms ] threadAlso the title "Pandas got 3X faster" seems to contradict the conclusion in the article, which reports the result was < 2x faster.
[1] https://github.com/modin-project/modin
https://github.com/modin-project/modin
Modin uses Ray, a distributed computation library. There was a similar article on HN a year ago that hyped "making pandas faster" by replacing it with Ray:
https://news.ycombinator.com/item?id=16510610
[1] https://rise.cs.berkeley.edu/blog/pandas-on-ray-early-lesson...
I would prefer to pass an optional parallel=true parameter to some functions in the API, or have a configuration setting that can fall back to non-parallel implementation.