An anecdote that cannot be extrapolated: I did a course on using Dask a few years ago (Dask was fresh then). They teach us how Dask would improve our calculations, to the point of almost promissing a time improvement of "if you chop your dataframe in 8 partitions, your times will be divided by 8", no questions asked.
Then they gave us a sample of a dataframe, to analyze firstly with Pandas, and then with Dask, and compare. The times not only didn't improve, or even equal Pandas: they got worse by more than 10x. The teachers looked the code with a mix of disbelief and horror, and only after that they started to talk about tradeoffs, overheads, data fitting in memory, etc.
So, the trouble with Spark (as I suspect you're probably aware) is that a) it's distributed and b) it's lazy.
This means that nothing happens until it needs to. So I have experienced .head() taking hours if there's a lot of upstream processing.
Another reason for this could be that your file is distributed across many nodes, and thus you need lots of network requests to get the file.
I'm not disagreeing with you that Spark can often have problems, but it's a really useful tool that can do things that are really hard to do elsewhere (like distributed ML).
I'm pretty sure the problem was that it was fragmented across workers and such.
I had already computed the filters and done a df.show() and count prior to trying save it.
It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
Oh yeah, there's stuff you can do, but again it takes tuning and ideally it shouldn't be so annoying.
I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way.
> run on a single node
yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.
I’ll totally second this. It takes away a lot of the tedium of multiprocessing programming.
I implemented it at my last job and it was a game changer. Simplified a lot of gnarly cluster management and let me focus on my code rather then figuring out how to distribute the work.
super useful in the pandas and scikit-learn realms, taken to the "cloud." Please note that DASK runs locally too! (you can run DASK on your local machine with Debian-Ubuntu or RH derivatives)
If you want built-in GPU support (and distributed), you should check out cuNumeric (released by NVIDIA in the last week or so). Also avoids needing to manually specify chunk sizes, like it says in a sibling comment.
I found out that Dask is like half-done Spark. Dask kinda works with regular dataframes, but there are just too many inconsistencies that almost always Dask breaks on a working Pandas algorithm. It's just easier to install PySpark and work with it. IMHO.
I'm not making any claims about dask being super reliable or easy to use, but let's not pretend that a spark cluster is a trivial thing to set up and deal with.
I've had to administer spark clusters in the past, and there's a good reason why databricks is a thing.
Another detail that's important is that spark is dramatically overkill for the data sets dask is meant to work on. Dask is really geared for data sets that can't fit in memory but not necessarily data sets that can't fit on disk that spark was really meant for.
> spark is dramatically overkill for the data sets dask is meant to work on
I disagree with this, I have used spark standalone/single node both locally and in the cloud for numerous use cases because you can easily cook up spark sql or pyspark cleaning and ETL scripts for small data (a couple TB) and reach high throughput (>260mb/s). I am also someone who reaches for dask (and prefect) first for small data but in cases where the data is medium sized (100gb-10tb) I always reach for spark first, its simply easier to write fast code with spark (especially spark sql) without needing to think hard about optimizations. I also have not run into any operational issues running a spark standalone instance for a few hours or days to process small data I would use dask for.
I reach for dask first for most small data (<100gb) because its better integrated with python libraries like prefect that improve the dev experience.
Spark seems like it will be good, but always disappoints.
I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve both:
1) the automatic configuration of single node systems
2) the changing of config throughout the processing
The fact that I don't even really know if this is the problem however is the real problem.
I get that setting stuff up for a huge network of computers may be challenging, but standard defaults to use all processors on a single machine should be automatic.
Spark pisses me off that the defaults for a 1TB RAM, 20 core machine as a single node can't handle reading a 20GB parquet file with one line.
And further, checking substrings can throw OOM errors in the next step without adjusting the config of worker memory, etc.
Worst of all, a 40 MB (!) file after filtering and processing can take like 10 minutes to save because of all the parallelism config. It's off course instantaneous on pandas.
They really need to fix the experience for single machines (e.g. just like a small 18 core, 512 GB ram system), so that people can write packages that work for anyone, but with a small yaml flag, it can be sent to a cluster.
Yes. I'm not a fan of Spark, dealing with JVM, new syntax everything, optimizing parallelism in a weird way but - it always works.
Dask, on the other hand, works some of the time. The rest of the time it'll keep running a calculation forever, or simply fail silently over and over, or some other unpleasant outcome.
I had a similar experience, I usually avoid the dask dataframe where possible and instead use bag and dask delayed. But its hard to get scientists to give up the dataframe mindset. Thats one of the reasons I wanted to try modin, I have heard the dataframe in modin is a bit easier to use.
I tried to use Dask to do a formst conversion of a 5 TB binary dataset coming from astrophysical simulation. It should have been a trivial job, since it was a single huge 3D array coming from a lot of files that were slicing the array in one direction, and I needed to do a conversion in each point, do a couple of transposes and reshapes, and then output a single file.
After two weeks tearing my hair out at the appalling speed, I scrapped that solution and spent one day throwing together a pipeline of classical shell tools (dd, sed, xxd etc.), orchestrated using GNU Parallell, and it did the whole job in three hours.
Funny and I had a similar real life experience with a hadoop cluster. I replaced a job that needed 10 machines with a few lines of C, some python and GNU parallel, which was perl I believe at the time. The process was taking over 24 hours on the cluster and became unworkable. I got it back to 2 hours on my slow Macbook air.
On Dask, it may not be super fast for very large data sets but it is sure easy to use.
I was mainly looking at Dask for the purpose of a more better Pandas DataFrame, but it seems like it's actually not that much better, just more distributed. It's like Dask is trying to do too many things and there doesn't do most of them well enough.
While I haven't tried it, vaex[1] seems much better, since it specifically to only address one pain point, which is to address the inefficiencies of the pandas DataFrame and aims to do that one thing well.
a path I haven't explored yet is Koalas, which touted a 80% API compatibility with pandas at version 1.0. Now its at version 1.8, and it has the pandas API as the main design principle for function interface design.
Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutely necessary.
My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads.
The nice thing about dask is really the way you can effectively use it on pandas dataframe; Without any large overhead.
Having said that, we opted to write our own parallelization for this library:
As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck and easier to maintain the threading ourselves given the particular approaches taken.
That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which couldn’t be stored in memory of use ray.io
To me, Dask is the solution to the wrong question. Rearchitecting the data loading + processing is better than trying to force data frames into a distributed world and will perform a lot better too.
Dask is well built but is fundamentally flawed by tacking together pandas structures with multiprocessing, rather than having a fundamentally efficient underlying structure.
I love Dask for parallel computing because instead of threads or agents, you’re primarily using distributed data structures like distributed DataFrames, Series, and Bags.
It’s made our code very small, clean, and bug free.
Apologies for being pessimistic. I've never had a reason to use dask. I like Pandas only for very small sample datasets. Nothing more than a few MBs. I deal with large amount of alpha-numeric data that can no way fit on a single computer. Even with multi-processing abilities, it's never really that useful and pandas does the job. I've to use large clusters and spark for any real datasets before I even think about using pandas. Spark has its quirks but I'd argue that it has a lot more features when it comes to distributed computing.
Data in the 1-15 GB range is more common than you'd think. Spark is overkill and comparatively a nightmare to configure. You can fit 15 GB in memory nowadays on a single core, but you probably want things to move a little quicker.
You're definitely correct about Spark being a nightmare to configure. Luckily we've got data platform teams who take care of that. Analysts like I still do configure parameters depending on the scenario. You might be correct about data in the 1-15 GB being more common. I'm an analyst so most of the time, I've to aggregate data before I do any analysis. But data scientists in my team typically work on smaller datasets and use Sagemaker instances.
I work at a University HPC centre and our new 72 core Icelake nodes have 512gb of RAM, and even our older 40 core Cascade Lake nodes had 256gb as standard, so I always scratch my head at who these libraries serve. That's pretty standard hardware for a HPC cluster, which is where people doing this stuff should be working anyway.
I wasn't thinking about HPC at all. I more had in mind "single-node" use cases like a workstation PC.
But even in a small cluster type of use case, I think the main users are scientists or other practitioners who, for whatever reason, don't have the support of a platform team or data engineering team that can keep a Spark (or other) cluster fed and cared for.
Off the top of my head, I can't remember what size nodes we use. I think it's m5x large AWS nodes with 16 cores and 128 GB RAM. Depending on the size of the data I use anywhere between 20 to 80 of these nodes. Sometimes even higher. That's why I've never had a need for these libraries.
I have seen european companies (claiming) doing big data on 16gb, 4-cpu, 15 node clusters and claiming that their spark jobd were not failing due to 90s hardware. I will send them your configuration. :)
I don't think that's being pessimistic. I think it reflects dask's documentation. Some items I looked up because I remember from the past:
> If your data fits comfortably in RAM and you are not performance bound, then using NumPy might be the right choice. Dask adds another layer of complexity which may get in the way.
> In many workloads it is common to use Dask to read in a large amount of data, reduce it down, and then iterate on a much smaller amount of data. For this latter stage on smaller data it may make sense to stop using Dask, and start using normal Python again.
Spark is massive overkill for the vast majority of needs. Can't fit the data on a computer, like your use case? Yep, good for that.
Anything else, and it's not worth the overhead. Dask doesn't require a JVM cluster with executors and name nodes and hdfs and everything else that keeps databricks in business.
yeah, like others have pointed out, you might be correct about it being a massive overkill for most users. It takes me at least 10 to 15 minutes to spin up a cluster and configure it before I can start anything.
In the academic world you can run Dask on the typical clusters using slurm or grid-engine, without any additional setup. You're not going to get a university IT department to create a Spark cluster for you.
The integration with xarray[0] and the Pangeo[1] community at large are pretty good. Not good enough for sub second response on large datasets but very good for analytic workloads, especially when you are dealing with larger than memory datasets.
If you want fast(ish) performance but you can wait a few minutes or longer its a great tool. Climate scientists love it because it lets them focus on their problem domain instead of focusing on dealing with developing parallel software.
If you are using it to magically speedup your api backend you will probably be disappointed, but you will also be just as disappointed trying to use a jackhammer to hammer in a nail.
I rely heavily on Dask/Pangeo stack to serve time-series weather data via Rest API primarily based on ERA5. You’re correct, it won’t give you sub-second responses but turns out this is more than sufficient for data analysis work.
I recently switched from Spark => Dask cause I like it so much. Disclosure: work for Coiled, the company founded by Matt Rocklin, the creator of Dask.
Here's what I've found thus far with Dask:
* way easier to build custom distributed compute systems with Dask than other alternatives. Dask futures & delayed APIs give you access to the "engine" so you can build your own custom race car
* Lots of data scientists are a lot more productive with Dask compared to Java / Scala technologies they're not comfortable with (I have a popular Spark blog / Spark books & love Spark, but some ppl just can't get productive with Spark)
* Dask cluster visualizations are so nice. So easy to understand how clusters are being computed in real time.
* We've been working closely with NVIDIA folks to provide real cutting edge GPU support.
I'm a believer in multi-tech data pipelines in the future. A data engineering team that loves Scala may write some ETL code in Spark and then pass off the baton to a data science team that loves Python and uses Dask to train machine learning models.
Seems like most other players in the data space want to take over an organization's entire data platform. I like how Dask likes to play nicely as part of the overall PyData ecosystem. I've always liked the Unix philosophy of building little tools that can be easily composed to solve a variety of different problems and feel at home in the PyData ecosystem.
Dask is really good for scientists, when I worked with bioinformaticians it was way easier to get them to use dask for out of memory processing than spark. In particular dask bag offers a great amount of flexibility for ETL use cases.
The problem with dask for me though (as a user of dask and prefect) is that I was never able to get the throughput of pySpark out of dask. For dask I can reach 100 mb/s on my laptop while pyspark can each 260 mb/s on my laptop for the same workload (cleaning and restructuring). This was around 2 years ago that I tested that I’m curious what the community has experienced now.
Also for most use cases your data isn’t really big data and dask is much simpler and fast enough which is why I reach for dask first.
We use it to go faster than spark via dask_cudf: bottleneck becomes pci/ssd, which is in GB/s.
For CPU, have not benchmarked latest CPU dask vs CPu spark. Would expect to see spark win on simple kernels (pandas vector ops) and lose on ML/C++ ones (ex: igraph vs graphx)
Used Dask for various research experiments and scripts, and it's very good when you have to handle larger-than-memory datasets. When coming to algorithm parallelization, some quirks emerged, and debugging sent me completely nuts. I love Dask, but I think that the problem of parallelizing Python code is not solvable by Dask alone.
My dissertation used a lot of dask code, and I found it to be an amazingly powerful tool for helping me to glue a few dozen other libraries together and orchestrate them as a single distributed system. sure it's half done spark, but if it were fully done spark I couldn't have used it as flexibly.
We have been using dask to support our computational pathology workflows [1], where the images are so big that they cannot be loaded in memory, let alone analyzed (standard pathology whole slide images are ~1GB; some microscopy techniques generate images >1TB). We divide each image into a bunch of smaller tiles and process each tile independently. The dask.distributed scheduler lets us scale up by distributing the tile processing across a cluster.
Benefits of dask.distributed: easy to get up and running, and has support for spinning up clusters on lots of different computing platforms (local machines, HPC cluster, k8s, etc.)
One difficulty is optimizing performance - there are so many configuration details (job size, number of workers, worker resources, etc. etc.) that it's been hard to know what is best.
Actually not sure if anyone noticed this https://github.com/amzn/amazon-ray, that means a good amount of benchmark has been done and Ray won. Have a commitment from Amazon (and even setup that repo) is no small feat
Dask is useful on a single laptop as well, it can be used to turn a computation that's too large for memory in xarray, to a chunked one that just crunches through, with the same code.
modin + ray would have been easier, and not required a daskification of the code, because modin's stated aim is complete compatibility with the pandas API. still, I've managed to limit it to one module in the project, and I am planning to try modin + ray vs dask to see if it is faster for this dataset.
idk, for me dask was more of a pain to set up than a single node spark cluster, and now that databricks has added a fully compatible pandas api to pyspark, there is even less reason for me to use dask locally.
I started looking at Dask recently... its scheduler/worker mechanism reminded me of the old Docker Swarm. I liked that!
But when I moved to a Kubernetes deployment (set up via the community provided Helm charts), I was completely mystified and got slowdowns as reported by others. At one point my physical worker node was using 1 core, and at other times, it was using all cores. When I dug into the default config they provide, it should have been 2 vcpus. So .. all in all .. I was completely mystified what was going on under the hood, which speaks to the lack of debugging mentioned by people.
86 comments
[ 3.1 ms ] story [ 154 ms ] threadBut "naïve programming" (the art of writing a dumb algorithm with no regards to performance or optimizations) can bite you hard: https://docs.dask.org/en/latest/best-practices.html#avoid-ve...
I implemented an algorithm that was slower than the non-parallelized version this way :D
amen. people should learn to write fast code instead of "spraying and praying"
https://users.ece.cmu.edu/~franzf/papers/gttse07.pdf
Then they gave us a sample of a dataframe, to analyze firstly with Pandas, and then with Dask, and compare. The times not only didn't improve, or even equal Pandas: they got worse by more than 10x. The teachers looked the code with a mix of disbelief and horror, and only after that they started to talk about tradeoffs, overheads, data fitting in memory, etc.
This means that nothing happens until it needs to. So I have experienced .head() taking hours if there's a lot of upstream processing.
Another reason for this could be that your file is distributed across many nodes, and thus you need lots of network requests to get the file.
I'm not disagreeing with you that Spark can often have problems, but it's a really useful tool that can do things that are really hard to do elsewhere (like distributed ML).
It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way.
> run on a single node
yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.
You can build really ergonomic distributed programs that are small wrappers around python classes.
I would prefer this to multiprocessing even if you are only on a single host.
I implemented it at my last job and it was a game changer. Simplified a lot of gnarly cluster management and let me focus on my code rather then figuring out how to distribute the work.
https://github.com/nv-legate/cunumeric
I've had to administer spark clusters in the past, and there's a good reason why databricks is a thing.
Another detail that's important is that spark is dramatically overkill for the data sets dask is meant to work on. Dask is really geared for data sets that can't fit in memory but not necessarily data sets that can't fit on disk that spark was really meant for.
If you're in the JVM you can basically embed it in-process.
I disagree with this, I have used spark standalone/single node both locally and in the cloud for numerous use cases because you can easily cook up spark sql or pyspark cleaning and ETL scripts for small data (a couple TB) and reach high throughput (>260mb/s). I am also someone who reaches for dask (and prefect) first for small data but in cases where the data is medium sized (100gb-10tb) I always reach for spark first, its simply easier to write fast code with spark (especially spark sql) without needing to think hard about optimizations. I also have not run into any operational issues running a spark standalone instance for a few hours or days to process small data I would use dask for.
I reach for dask first for most small data (<100gb) because its better integrated with python libraries like prefect that improve the dev experience.
I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve both: 1) the automatic configuration of single node systems 2) the changing of config throughout the processing
The fact that I don't even really know if this is the problem however is the real problem. I get that setting stuff up for a huge network of computers may be challenging, but standard defaults to use all processors on a single machine should be automatic.
Spark pisses me off that the defaults for a 1TB RAM, 20 core machine as a single node can't handle reading a 20GB parquet file with one line. And further, checking substrings can throw OOM errors in the next step without adjusting the config of worker memory, etc.
Worst of all, a 40 MB (!) file after filtering and processing can take like 10 minutes to save because of all the parallelism config. It's off course instantaneous on pandas. They really need to fix the experience for single machines (e.g. just like a small 18 core, 512 GB ram system), so that people can write packages that work for anyone, but with a small yaml flag, it can be sent to a cluster.
See dask-sql: https://dask-sql.readthedocs.io/en/latest/pages/api.html
Dask, on the other hand, works some of the time. The rest of the time it'll keep running a calculation forever, or simply fail silently over and over, or some other unpleasant outcome.
After two weeks tearing my hair out at the appalling speed, I scrapped that solution and spent one day throwing together a pipeline of classical shell tools (dd, sed, xxd etc.), orchestrated using GNU Parallell, and it did the whole job in three hours.
On Dask, it may not be super fast for very large data sets but it is sure easy to use.
While I haven't tried it, vaex[1] seems much better, since it specifically to only address one pain point, which is to address the inefficiencies of the pandas DataFrame and aims to do that one thing well.
[1] https://vaex.io/docs/index.html
https://databricks.com/blog/2020/06/24/introducing-koalas-1-...
My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads.
The nice thing about dask is really the way you can effectively use it on pandas dataframe; Without any large overhead.
Having said that, we opted to write our own parallelization for this library:
https://github.com/capitalone/DataProfiler
As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck and easier to maintain the threading ourselves given the particular approaches taken.
That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which couldn’t be stored in memory of use ray.io
https://www.prefect.io/
Dask is well built but is fundamentally flawed by tacking together pandas structures with multiprocessing, rather than having a fundamentally efficient underlying structure.
It’s made our code very small, clean, and bug free.
But even in a small cluster type of use case, I think the main users are scientists or other practitioners who, for whatever reason, don't have the support of a platform team or data engineering team that can keep a Spark (or other) cluster fed and cared for.
> If your data fits comfortably in RAM and you are not performance bound, then using NumPy might be the right choice. Dask adds another layer of complexity which may get in the way.
from https://docs.dask.org/en/latest/array-best-practices.html#us...
> In many workloads it is common to use Dask to read in a large amount of data, reduce it down, and then iterate on a much smaller amount of data. For this latter stage on smaller data it may make sense to stop using Dask, and start using normal Python again.
https://docs.dask.org/en/latest/best-practices.html#stop-usi...
So, dask even says not to use it for small data sets. And in their comparison to spark, they say spark has a lot more features.
Anything else, and it's not worth the overhead. Dask doesn't require a JVM cluster with executors and name nodes and hdfs and everything else that keeps databricks in business.
Answer: it you can distribute way beyond a single computer with Dask
I think the time and energy that is spent on Dask would be better spent to get proper concurrency support into python.
If you want fast(ish) performance but you can wait a few minutes or longer its a great tool. Climate scientists love it because it lets them focus on their problem domain instead of focusing on dealing with developing parallel software.
If you are using it to magically speedup your api backend you will probably be disappointed, but you will also be just as disappointed trying to use a jackhammer to hammer in a nail.
[0] https://xarray.pydata.org/en/stable/ [1] https://pangeo.io/
https://openeo.org/documentation/1.0/developers/backends/ope...
That's the issue with these tools.
Here's what I've found thus far with Dask:
* way easier to build custom distributed compute systems with Dask than other alternatives. Dask futures & delayed APIs give you access to the "engine" so you can build your own custom race car
* Lots of data scientists are a lot more productive with Dask compared to Java / Scala technologies they're not comfortable with (I have a popular Spark blog / Spark books & love Spark, but some ppl just can't get productive with Spark)
* Dask cluster visualizations are so nice. So easy to understand how clusters are being computed in real time.
* We've been working closely with NVIDIA folks to provide real cutting edge GPU support.
I'm a believer in multi-tech data pipelines in the future. A data engineering team that loves Scala may write some ETL code in Spark and then pass off the baton to a data science team that loves Python and uses Dask to train machine learning models.
Seems like most other players in the data space want to take over an organization's entire data platform. I like how Dask likes to play nicely as part of the overall PyData ecosystem. I've always liked the Unix philosophy of building little tools that can be easily composed to solve a variety of different problems and feel at home in the PyData ecosystem.
The problem with dask for me though (as a user of dask and prefect) is that I was never able to get the throughput of pySpark out of dask. For dask I can reach 100 mb/s on my laptop while pyspark can each 260 mb/s on my laptop for the same workload (cleaning and restructuring). This was around 2 years ago that I tested that I’m curious what the community has experienced now.
Also for most use cases your data isn’t really big data and dask is much simpler and fast enough which is why I reach for dask first.
For CPU, have not benchmarked latest CPU dask vs CPu spark. Would expect to see spark win on simple kernels (pandas vector ops) and lose on ML/C++ ones (ex: igraph vs graphx)
Would be interesting to see carefully done!
Benefits of dask.distributed: easy to get up and running, and has support for spinning up clusters on lots of different computing platforms (local machines, HPC cluster, k8s, etc.)
One difficulty is optimizing performance - there are so many configuration details (job size, number of workers, worker resources, etc. etc.) that it's been hard to know what is best.
[1] https://github.com/Dana-Farber-AIOS/pathml
Depending on your workload, you may get far better throughput not worrying about distributing the work and data
But when I moved to a Kubernetes deployment (set up via the community provided Helm charts), I was completely mystified and got slowdowns as reported by others. At one point my physical worker node was using 1 core, and at other times, it was using all cores. When I dug into the default config they provide, it should have been 2 vcpus. So .. all in all .. I was completely mystified what was going on under the hood, which speaks to the lack of debugging mentioned by people.