> A killer feature of siuba is that the same analysis code can be run on a local DataFrame, or a SQL source.
Everyone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has.
E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator flavor of SQL basically nukes all that.
I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason.
In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.
I think this is a port of the dplyr syntax/verbs only. None of the dplyr C++ code is used, it's all syntactic sugar on top of Pandas. So any SQL translation is going to be similar to what Pandas does.
What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally.
I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.
This approach seems a little complicated, though I'm sure with some use I could learn to enjoy it.
Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class.
Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do [1]! One obstacle here is that the DataFrame has 300+ methods, which can be overwhelming to learners.
I've spent a lot of time wondering whether the piping syntax feels like too much vs chaining. It's still an open question in my mind, so it's really helpful to hear what feels most natural!
> What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally.
> I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.
I think I agree. Pandas has a lot of overhead/baggage that I don't want 90% of the time. Being able to chain _simple_ verbs on a data frame would be great -- something like mtcars.groupby(cyl).summarize(avg_hp = hp.mean())
kudos for doing this. Whilst I haven't used Suiba, dplyr in R is a very compelling way to do data munging. If Siuba brings the same thing to python, it's a very welcome addition to the ecosystem.
I know pipes split opinion; I'm definitely in the 'pro' camp. Chaining ops in a dataflow pipeline fits my mental model well.
Dplyr and data.table are two libraries that make R shine in terms of data manipulation.
Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.
Siuba uses type hints to dispatch the appropriate versions of custom functions!
For example, siuba allows users to create custom functions using a thin wrapper around functools.singledispatch.
When deciding how to run...
df >> filter(my_custom_func(_.x))
It requires that the return type be compatible with the backend being run (e.g. pandas, a SQL dialect).
Would be super interesting to try and lay out what would be needed to do static analysis via mypy. I think it'd require some plugins for singledispatch at least, probably some reworking things in ways myoy expects.
This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai).
However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.:
It also plays nicely with the black autoformatter.
Anonymous functions are verbose and limited in Python, but you can still do many things and use a regular function when a lambda won't do.
I guess one of the thing I need the most when doing data analysis is column name autocompletion, inside groupby, lambdas, for column selection...
I wonder if one could do it in IPython, similar to the string autocompletion of file/directory paths. Basically a parsing of dataframe column names in order to autocomplete strings.
If the necessary information is there, as in you've mentioned a column name in a dataframe at least once, PyCharm will now do column name autocompletion. It's actually pretty solid in my experience.
Didn't know pycharm would do that... thanks for the info! Yeah the best things about dplyr in opinion are 1) less verbose than pandas 2) much better autocomplete.
As long as there isn't a space in the column name. This is riding on a Pandas trick of making the column name accessible as an attribute of the dataframe, which breaks down when there's a space.
Hey, thanks for pointing out Self--I definitely need to dig into fastcore more!
One motivation for developing siuba is that the grouped agg you show requires users specify only one operation on one column.
E.g.
1. Calculate mean of x
However, common operations like demeaning a column are multiple operations:
1. Calculate mean of x
2. Subtract result of (1) from x
In siuba you can just write mutate(res = _.x -_.x.mean()). This isn't possible from something like gdf.x.agg("mean"), and from what I can tell deeply confusing to analysts :/.
In vanilla pandas I really like to use the chaining method you laid out, and siuba to me is mostly a utility library for making the approach a little more succinct / performant[1].
siuba has experimental autocompletion (thanks to Tim Mastny!), and there's a pretty hefty technical write up on how it uses IPython machinery for that in siuba's architectural desicion record folder[2].
Fastcore is great. I've been going ape with it recently for a work project, and I've particularly fallen in love with patch. Made it extremely easy to assemble a collection of pure functions into pseudo methods attached to an existing class.
Hey y'all, creator of siuba here--happy to answer any questions!
One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].
I'm a big fan of pandas though. It's what siuba is built on!
Using the experimental fast grouped pandas functions, it should run at the speed of optimized pandas code!
Since siuba functions just run on pandas DataFrames, you can always hand tune for performance, but imo most of the time pandas code runs slow it's because of something like .agg(lambda ...) somewhere.
31 comments
[ 2.8 ms ] story [ 74.7 ms ] threadEveryone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has.
E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator flavor of SQL basically nukes all that.
In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.
Thanks for bringing this up--the docs could be clearer here
I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.
This approach seems a little complicated, though I'm sure with some use I could learn to enjoy it.
Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do [1]! One obstacle here is that the DataFrame has 300+ methods, which can be overwhelming to learners.
I've spent a lot of time wondering whether the piping syntax feels like too much vs chaining. It's still an open question in my mind, so it's really helpful to hear what feels most natural!
https://pandas.pydata.org/pandas-docs/stable/development/ext...
Edit: oh that's pretty much what the linked decorators do.
I think I agree. Pandas has a lot of overhead/baggage that I don't want 90% of the time. Being able to chain _simple_ verbs on a data frame would be great -- something like mtcars.groupby(cyl).summarize(avg_hp = hp.mean())
I know pipes split opinion; I'm definitely in the 'pro' camp. Chaining ops in a dataflow pipeline fits my mental model well.
Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.
For example, siuba allows users to create custom functions using a thin wrapper around functools.singledispatch.
When deciding how to run...
It requires that the return type be compatible with the backend being run (e.g. pandas, a SQL dialect).Would be super interesting to try and lay out what would be needed to do static analysis via mypy. I think it'd require some plugins for singledispatch at least, probably some reworking things in ways myoy expects.
https://nbviewer.jupyter.org/github/machow/siuba/blob/master...
However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.:
df.loc[lambda f: ...].groupby(...).agg(["mean", "count"])
It also plays nicely with the black autoformatter.
Anonymous functions are verbose and limited in Python, but you can still do many things and use a regular function when a lambda won't do.
I guess one of the thing I need the most when doing data analysis is column name autocompletion, inside groupby, lambdas, for column selection...
I wonder if one could do it in IPython, similar to the string autocompletion of file/directory paths. Basically a parsing of dataframe column names in order to autocomplete strings.
[0]: https://fastcore.fast.ai/basics.html#Self-(with-an-uppercase...
[1]: https://tomaugspurger.github.io/method-chaining
One motivation for developing siuba is that the grouped agg you show requires users specify only one operation on one column.
E.g.
1. Calculate mean of x
However, common operations like demeaning a column are multiple operations:
1. Calculate mean of x
2. Subtract result of (1) from x
In siuba you can just write mutate(res = _.x -_.x.mean()). This isn't possible from something like gdf.x.agg("mean"), and from what I can tell deeply confusing to analysts :/.
In vanilla pandas I really like to use the chaining method you laid out, and siuba to me is mostly a utility library for making the approach a little more succinct / performant[1].
siuba has experimental autocompletion (thanks to Tim Mastny!), and there's a pretty hefty technical write up on how it uses IPython machinery for that in siuba's architectural desicion record folder[2].
[1]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
[2]: https://github.com/machow/siuba/blob/master/examples/archite...
The architecture necessary to pull off executing either pandas or SQL also makes it very extensible (e.g. to spark or dask in the future :).
https://siuba.readthedocs.io/en/latest/key_features.html
I use method chaining as the composing notation.
One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].
I'm a big fan of pandas though. It's what siuba is built on!
[1]: https://m.youtube.com/c/chowthedog
[2]: https://mchow.com/posts/2020-02-11-dplyr-in-python/
[3]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
Since siuba functions just run on pandas DataFrames, you can always hand tune for performance, but imo most of the time pandas code runs slow it's because of something like .agg(lambda ...) somewhere.
There's an example with timings here:
https://siuba.readthedocs.io/en/latest/developer/pandas-grou...