I've been trying to learn python for a while now, and I have a grasp of it in a similar way that I have a grasp of golf- I know what to do, but I'm still missing a few fundamentals.
All these intro to X things seem to gloss over the high level stuff, but I've hit a few walls in certain concepts that I just can't possibly Google. Maybe HN could help me out:
take a pandas dataframe. There's a .describe() method that I can chain to get the mean of the columns in the data frame grouped by var1:
df.groupby(df.var1).describe().mean
But then, there's also:
df.describe(df.var1).mean()
They largely do the same thing, but one gives me the mean across all variables, and the other one doesn't.
Now, how do I possibly Google this question? One method has parentheses, the other doesn't.... but It's clear I'm missing something fundamental here that I thought an intro online resource should address... but none do.
Something without parenthesis cannot be a method call, at least not from a normal viewpoint. Instead you are trying to access the attribute "mean" of the result of describe() if you run df.groupby(df.var1).describe().mean. This attribute might very well be a method itself (a method object).
Glad to, as I need to learn Pandas in more depth than I currently know.
Also, I noticed something very very dangerous you did:
.describe() will return a DataFrame with all those statistics. One column per numeric column of the input. One row per statistical value (like mean, std, etc). If you call .mean() on THAT DataFrame, you will get the mean of the statistical values per column, so e.g. the mean of columnA's count, mean, std, etc values. Definitely not what you want!
Instead, use either .mean() directly on your "data DataFrame" (not the one returned by .describe()). Or access the specific value from the result via .describe()['columnname']['mean'].
The question you're asking is really about the Pandas library and its API, not so much about Python the language as such.
Python objects can have both callable methods (e.g. mean()) and properties/attributes that don't need an explicit call (e.g. mean). For whatever reason, it looks like Pandas' describe() method returns a result with a different interface when called with an argument than without one.
My first thought exactly. Pandas API is complicated enough that it requires a good grasp of Python in general to be able to tell what is regular Python, and what is the convoluted convention that Pandas uses to achieve its ends. This is not a criticism of pandas -- I think it extremely very well in being an R-like data analysis framework given the limitations/features of the Python syntax. But learning it before being confident with Python is like learning Ruby at the same time as learning Ruby on Rails.
This is very Pandas specific, but I’m going to give you my intuition with a bit of Python knowledge (10yrs):
1. Anything that doesn’t have parens is either an attribute or a property. A property is basically an attribute, except that a computation of some sort is usually executed to set/get its value, whereas an attribute has its value set explicitly. Look at the source to identify if the result of the first code block sets an attribute named `mean` or if it has an `@property` declaration named `mean`. Either way, they’re roughly the same when using them in code.
2. The fact that both `describe()` method examples can have another function and attribute/property chained to their result means `describe()` is returning an object of some type that is not a stdlib data type.
3. The `df.groupby().describe()` is likely returning a different type of object than the second `df.describe()`, as evidenced by the first result having a `mean` attr/property, while the second has a `mean()` function.
From here, I’d start by inspecting the object type returned by `df.groupby()`, `df.groupby().describe()`, and the `df.describe()` to discover the differences.
Adding my upvote to this recommendation. One of the few books I've read mid-career in which the material was as intellectually revelatory as it was applicable.
The thing is that "advanced Python" is too large a topic for a single book, or even for a single person to cover completely. It might be a series of books though; off the top of my head I can think of the following topics, each of which will cover many advanced areas:
- Python concurrency: multiprocessing, multithreading and asynchronous programming
- data science and numerical computation in Python
- embedded Python
- Python GUI development
- packaging and deployment of Python applications and libraries
Other than that, I don't know of a comprehensive source, in fact have been thinking about writing something myself (if time would allow...). This topic is complex because there is a lot of nuances, and there is no-one-size-fits all solution for numerous Python's use cases.
O'Reilly's High Performance Python touches on the first two items on your list, but it is by no means comprehensive. It's also Python 2 specific, but its principles carry over to Python 3.
The book you write. It contains all the code you wrote using python as well as all the bug fixes and refactors that you put time into.
Its going to take about 3 years to write , but youll eventually be intermediate. Getting advanced is more about problem space then the actual programing language. We all piggy back on things like Hadoop eventually.
There is not automating becoming a good programmer. Its really a craft. Especially when doing it for profit.
24 comments
[ 2.1 ms ] story [ 41.8 ms ] threadAll these intro to X things seem to gloss over the high level stuff, but I've hit a few walls in certain concepts that I just can't possibly Google. Maybe HN could help me out:
take a pandas dataframe. There's a .describe() method that I can chain to get the mean of the columns in the data frame grouped by var1:
df.groupby(df.var1).describe().mean
But then, there's also:
df.describe(df.var1).mean()
They largely do the same thing, but one gives me the mean across all variables, and the other one doesn't.
Now, how do I possibly Google this question? One method has parentheses, the other doesn't.... but It's clear I'm missing something fundamental here that I thought an intro online resource should address... but none do.
Use type() to see what you are getting:
Something without parenthesis cannot be a method call, at least not from a normal viewpoint. Instead you are trying to access the attribute "mean" of the result of describe() if you run df.groupby(df.var1).describe().mean. This attribute might very well be a method itself (a method object).Last in that chain was describe(), so let's look at that: https://pandas.pydata.org/pandas-docs/stable/generated/panda...
It returns a "summary: Series/DataFrame of summary statistics", so calling .mean() on that, will call https://pandas.pydata.org/pandas-docs/stable/generated/panda...
Accessing .mean will instead return that method object itself.
By default pandas will return a string describing the data itself when printing a method:
I assume this is what threw you off? You are not using the "results" of .mean in further code but were just looking at them, right?Also, I noticed something very very dangerous you did:
.describe() will return a DataFrame with all those statistics. One column per numeric column of the input. One row per statistical value (like mean, std, etc). If you call .mean() on THAT DataFrame, you will get the mean of the statistical values per column, so e.g. the mean of columnA's count, mean, std, etc values. Definitely not what you want!
Instead, use either .mean() directly on your "data DataFrame" (not the one returned by .describe()). Or access the specific value from the result via .describe()['columnname']['mean'].
Python objects can have both callable methods (e.g. mean()) and properties/attributes that don't need an explicit call (e.g. mean). For whatever reason, it looks like Pandas' describe() method returns a result with a different interface when called with an argument than without one.
1. Anything that doesn’t have parens is either an attribute or a property. A property is basically an attribute, except that a computation of some sort is usually executed to set/get its value, whereas an attribute has its value set explicitly. Look at the source to identify if the result of the first code block sets an attribute named `mean` or if it has an `@property` declaration named `mean`. Either way, they’re roughly the same when using them in code.
2. The fact that both `describe()` method examples can have another function and attribute/property chained to their result means `describe()` is returning an object of some type that is not a stdlib data type.
3. The `df.groupby().describe()` is likely returning a different type of object than the second `df.describe()`, as evidenced by the first result having a `mean` attr/property, while the second has a `mean()` function.
From here, I’d start by inspecting the object type returned by `df.groupby()`, `df.groupby().describe()`, and the `df.describe()` to discover the differences.
But there is a great, great need for REALLY well done books on intermediate python and advanced python.
It can be complemented by:
Talking about it, can you recommend a resource for that? Related: https://xkcd.com/1987/
Other than that, I don't know of a comprehensive source, in fact have been thinking about writing something myself (if time would allow...). This topic is complex because there is a lot of nuances, and there is no-one-size-fits all solution for numerous Python's use cases.
Its going to take about 3 years to write , but youll eventually be intermediate. Getting advanced is more about problem space then the actual programing language. We all piggy back on things like Hadoop eventually.
There is not automating becoming a good programmer. Its really a craft. Especially when doing it for profit.
http://composingprograms.com/