I use polars or duckdb now exclusively. Better syntax, better performance. But pandas is deeply entrenched - I try to get my team off it but it’s an uphill battle. It’s not going anywhere anytime soon.
> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
At my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization.
If a library is performant at large datasets, it is likely performant at small ones too.
Not to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse.
Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.
It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant.
I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down. Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.
I've been saying this since using Databricks at a company almost a decade ago. Most folks do not need big data tools, and it's just so entrenched because everyone wanted to be a "big data" company and pandas was how you handled big data.
Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.
DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.
Did you ever report these issues to polars developers?
I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
Makes sense, especially with AI coding tools the rewrite and familiarity arguments hold less water. Similar for the rustify everything crazy.
The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"
Only in the last few years did I start using SQL properly. Before that my pipelines would live in python. Now I offload as much to the db as possible, and keep my python simple glue. I'm very happy with this compared to other methods in pandas or polars.
If I still need to do db-like things in python I think duckdb is better.
Most of the time it works, but there are still corner cases throughout the python ML ecosystem where polars fails.
This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)
Besides the performance benefits, I use Polars at work because it’s just (subjectively) nicer to work with. The “pl.col” API lets you create arbitrary generated/virtual columns anywhere you want, declaratively. You can throw in these column expressions in wherever without actually computing their values and storing that in memory. Very powerful stuff.
Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your
For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries
How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???
And in the end save processed data in aws s3 in another format like iceberg or whatever
In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
If you mean the tables in the https://eddie.codes/posts/pandas-should-go-extinct/#i-have-b... section, it looks like those came straight from the Amazon paper, in which case __eddie__ shouldn't fix them, on general "quote accurately even if the quoted document has mistakes in it" principles. (Though a [sic] might help).
Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.
We run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool.
While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.
In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
The problem is a small change in the question can force you to make huge changes in the code in pandas.
I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas.
I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)
Pandas is not perfect or has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).
If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.
The reality is most of the pandas audience don't care about performance. Whenever performance is in the question people have always used polars, duckdb, dask etc. These users are usually software engineers and not data analysts. Agreed, that there is a real gap in latency and performance though!
84 comments
[ 0.25 ms ] story [ 10.4 ms ] threadAssumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
If a library is performant at large datasets, it is likely performant at small ones too.
This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things
Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.
I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down. Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.
I see a lot of work towards Geo-Data Science.
PostGIS was a first generation. Now we have GeoPandas, DuckDB Geoanalysis tools and Sedona DB.
IMDB data set is also up.
In many cases I’ve found directly using python primitives to be less confusing than pandas.
Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.
https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/
Published two posts at the same time and total PEBCAK
Something going wonky on their blog, where two posts got their links swapped.
DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.
I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"
If I still need to do db-like things in python I think duckdb is better.
This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)
> TODO: rewrite this entire service
remains.
I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.
Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.
For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries
Use SQL if you need to move data around.
And in the end save processed data in aws s3 in another format like iceberg or whatever
In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.
I agree that the API has warts, though typically it is more concise than polars.
[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
Pandas is not perfect or has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).
If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.