I do almost all of my day job in pandas. I consider myself very good at it. My number one recommendation to new data scientists learning the ropes is to just not use NumPy almost at all. I’m not sure where people learn it but they do all of this complicated nonsense. Just map simple Python lambda funcs with pd.Series.map and that’s most of what you need. Memorize your pd.DataFrame methods.
If your code feels like it dealing with a matrix and not a table, it’s probably doing something funny.
If by "vectorized" you mean: "able to delegate the task of performing mathematical operations on the array's contents to optimized, compiled C code." then I do not think you are correct (unless perhaps you are supplying map with a dict or Series).
Series.map is not compiling your lambda's to C and running it. If there is a built-in method available it usually will be faster. Notable exception are pandas str methods which devolve into Python code but generally with more overhead than map/apply.
I mean not writing your own loops. A built in function is indeed better. But usually not what you need. And also, readability > speed when the execution time is trivial, which is it probably should be for pandas scale data.
Hey, I also use Pandas every day and I would definitely recommend keeping up with your Numpy skills sharp. A lot of Pandas is built on top of Numpy so there's one good reason, but another is that it would prevent you from footguns like thinking Series.map is vectorized.
Those are small datasets. Also numpy != just operations on a matrix, but as mentioned above, proper vectorization can have profound speed differences for larger datasets.
They are indeed small. But most real world problems operate on small data.
And if it’s bigger you probably ought to be using a different toolset than Python altogether.
In my experience it is exceedingly rare to find a situation where you have > 1M rows of data and need to do tabular data manipulation, and it’s not coming from a managed database sort of setup where the manipulation could be better done in the database.
I think it really depends on the scale of data. If you're dealing with anything less than a GB, it probably doesn't matter all that much, but once you're dealing with larger datasets there is a pretty massive difference with using vectorized operation. Some of the pandas dataframes methods map to underlying numpy ones, but I don't believe that is always the case
With the availability of things like pyspark the grey zone between pandas scale and pyspark scale is small and uncommon though. Especially for the awkward tabular data manipulation tasks where you actually need to be mapping custom functions and what not.
Pandas built ins can cover everything with good performance except dataframe.apply imo
So, given the title and how stressful the last couple of weeks have been, I was sadly disappointed when this wasn't about drawing cute black and white bears.
I mean, data analysis is useful and all, but not what the heart wanted at the moment.
Brains are funny things. I saw the link for the first time a few hours ago, while I was on a lunch break, and immediately thought about the animal. I didn't click the link but did have a thought of "okay, that's kind of a strange topic". I went back to work, did some data analysis using pandas. Opened up HN again, and I saw the link and my brain didn't even register that this was a link I'd seen before -- it immediately assumed that it was about the python library. I didn't even think about it until I saw some comments about the animal.
Thanks. This is so clean and also fast! I just can't get used to pandas so I do most data wrangling in R and then bring the dataframe over to Python for analysis. But looks like Polars will streamline my workflow
45 comments
[ 5.0 ms ] story [ 117 ms ] threadhttps://betterprogramming.pub/pandas-illustrated-the-definit...
If your code feels like it dealing with a matrix and not a table, it’s probably doing something funny.
Pretty much everything you need in pandas is as performant as you ought to need for doing tabular data manipulation in Python. Except dataframe.apply
pandas apply:
17.9 ms ± 46.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)Vectorised function, using underlying numpy operations:
267 µs ± 11.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)Series.map is not compiling your lambda's to C and running it. If there is a built-in method available it usually will be faster. Notable exception are pandas str methods which devolve into Python code but generally with more overhead than map/apply.
Vectorized, choice between lazy optimization and eager.
And if it’s bigger you probably ought to be using a different toolset than Python altogether.
In my experience it is exceedingly rare to find a situation where you have > 1M rows of data and need to do tabular data manipulation, and it’s not coming from a managed database sort of setup where the manipulation could be better done in the database.
Pandas built ins can cover everything with good performance except dataframe.apply imo
That and using dicts as maps.
I mean, data analysis is useful and all, but not what the heart wanted at the moment.
One of my daughters is a panda bear fanatic and I thought this would be a resource I could share with her.
https://www.pola.rs/
https://news.ycombinator.com/item?id=34452526
> Pandas is an industry standard for analyzing data in Python.
https://www.pearson.com/en-us/subject-catalog/p/pandas-for-e...