Yeah I can see why 3500 pages are needed to document this inconsistent piece of software. Can never tell whether the method I have in mind is static or not, or how to access elements, (is it loc, iloc, query, [] directly, etc), will the results be a dataframe or a series or straight up numpy array/lists, index issues, etc etc.
If you think that these complaints are not valid or stem from a lack of experience, I urge you to use R first.
Huh, so it wasn’t just me who was confused when I tried to use pandas. I found pretty much anything non-trivial to be difficult at best and impossible at worst. (IIRC I needed to sum rows grouped by time period or something like that.) Then again, I haven’t used pandas much at all.
(I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)
Yeah, I'm not sure I understand the "I urge you to use R first" comment in the grandparent post, because I've used both and R's tidyverse was at least as difficult to approach as pandas for me. And what's the alternative, Excel?
So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts.
That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
A lot of learning curve come from them essentially being array programming languages (like APL). They are powerful, but just very different if you aren't used to them before.
The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep.
Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience.
I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases.
Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc.
I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit.
Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.
Data analysis isn't object-oriented nor imperative. Transformations are functions on datasets, not a sequence of stateful actions.
Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.).
It is about as hostile as it is possible to be, from "ideas" pov, to data science.
However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits.
Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls).
I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
When working with data, I like to avoid stateful data mutations at all cost. OOP's focus is mutating state. These are at odds. The identity semantics I hold for data are not invariant under transformation.
Understanding the history of OOPs development, what motivated it, what problems it seeks to solve, it's guiding principles, and design semantics, are not invalidated by copy on modify Uri classes (or Strings!).
As I mentioned in the comment to mjburgess, it does not matter, unless you can show it does. Fortunately, knowing the details of history is entirely optional for the ability to discuss the current state of affairs.
No, just validating my assumptions. What sort of evidence would change your mind? Accounts from language designers? Expert testimony? Reference material? Interviews?
You are extraordinarily ignorant of the history and design of programming languages to be making such statements.
`object = state` is not a paraphrasing of first order logic; it is a paraphrasing of the english sentence,
"The intention motivating OOP is that the object represents state; the object's API represents changes to its state; an encapsulation ensures those changes are privately managed."
which if you understood any of the context to this discussion, you would have captured -- see, eg., the other commenter who captured this accurately.
I have taught hundreds of hours on programming language theory, and it is what I am doing today; and it is something I have been doing for a decade (not least, programming for two decades).
I take this tone with you because assuming a person cannot /reason/ basically is an extraordinarily arrogant view which presumes you have precisely captured the context of their statements.
In cases of apparent "basic failures to reason appropriately" it is almost always the observer, not the speaker, who has failed to arrive at the correct "propositional model" of their spoken/written language.
You would do well to remember this, and include more humility in your communications.
Admittedly, I turn arrogant when I (potentially mis-)smell bullshit.
First of all, it does not matter what the initial intention was when OOP started growing to where it is now. The whole historical argument makes no sense whatsoever. The fact that something was like X in the past does not mean it is going to be like X in the future.
Second, I have a lot of doubt about the statement, that the idea of object API was solely to represent changes to object state. Unless you can link a supporting quote, I am ready to dismiss it as made up.
I beg to differ. If you are talking about dplyr, it has a much smaller and consistent API. And I learned python first.
It is certainly interesting though, how there are so many people firmly in opposite camps.
I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple.
In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
dplyr is nice, but the fact there is an entire company writing packages to get around R's inconsistent standard library might be a bit of a hint. Also, as soon as you stray away from dataframes R is completely lost.
I recently inherited an R codebase at work. It was not fun understanding it, and trying to hunt down docs to explain some of the functions was a much worse experience than it is with Pandas[1]. So I did the sane thing: I figured it all out and rewrote it in Pandas.
[1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.
Cannot agree more. Even using R just to read csvs, transform the data, and output another csv, can be an improvement, and it lets you still use python for other things.
Pandas is marvelous. As is numpy, scipy, and all of that family. And they run essentially as C libraries, and approximate that level of performance. Someday I hope there will be golang analogues of these
Once generics ship, there could be progress in this direction. Until then, though, very high level data structure type libraries, along the lines of pandas, are always going to be terrible in Go.
As someone new to python trying to use these, I find them often too magic. Soo much overloading, magic, parameters etc, with virtually no help from a type system.
Like this standard usage which misuses all the operators:
a = b[c == 1]
One would think that would be equivalent to b[true] or false, and what's that supoosed to mean? But if c is a numpy array it's instead doing element wise comparison on c, and the new array of booleans will be used to index into b. If you don't know this, it's impossible to figure out looking at the code. Too many footguns imo. Same with pytorch, broadcasting is nice, but makes it hard to figure out what's going to happen in practice with the shapes being calculated upon.
Your complaint applies to array processing libraries in general. It applies to numpy. Yet those syntactic niceties are exactly what make them so convenient. Sure, it takes time to get comfortable writing array-oriented code. But it isn’t a good idea to optimize for complete beginners. Some familiarity with the general domain is a prerequisite to using array (and dataframe) libraries.
But instead of having X ways of using the [] operator, it would be almost just as convenient if that were actual named functions. Then the editor could understand whats going on, one could jump to definitions, look up stuff, more easily actually learn that it's possible to do etc.
I feel Notebooks aren't used as much because they're a great tool, but that they are used so much as a necessity since you need to look at live data to have any hopes of understanding what's going on. Why couldn't a compiler instead tell me what I'm trying to do will fail?
It's just like learning any unfamiliar programming approach. Array processing has a long history, and it's supported by languages like R which are heavily used in data science. Once you understand the different uses of [] operator, it becomes second nature, and you realize how useful vectorization and broadcasting can be. Also, manipulating live data is an excellent way to learn about your data and figure out what you need to do. Python is a dynamic language, so interacting with a rich REPL environment makes a lot of sense.
I agree with you in spirit. Having separate names is preferable to context-specific semantics for the []. In pandas, using [] alone is discouraged, in favor of using .loc or .iloc. But admittedly, that only partially improves the situation. Both accept multiple types (int vs int-list vs bool-list).
Last year, I had an in-person discussion with a numpy core developer in which I proposed adding pandas-like syntax to numpy, e.g. allowing the author to use:
a.boolmask[b == c]
(but maybe not so verbose).
A problem immediately arises: An array can have multiple axes, and each could be indexed with a different type within the same slicing call.
One idea might be to allow the author to explicitly wrap the arguments with some dummy class to make clear what types they’re using:
a[np.boolmask(b == c), d, np.intlist(e)]
(Again, perhaps choosing shorter names in practice.)
The idea would be that the wrapper merely “looks like” the thing it wraps, but contains an assertion to verify the type of the argument.
Anyway, I agree that notebooks (or REPLs in general) are great for verifying your assumptions about how little snippets will behave. But I think they would be useful even in a fully statically-typed compiled language, too.
I think that this is something that you can get accustomed to pretty quickly. Vectorised operations and broadcasting are very common across languages too, it's not just the python data stack that uses them.
A more weird but still basic example from pandas is the way it handles indexing and the confusion between .loc, .iloc, .ix, .at and plain '[]'. That part of the API should be really basic but for some reason it's a giant mess.
If you're expecting a numerical library's operations/semantics to be intuitive to a reader not familiar with the library, you'll be disappointed by most such libraries.
What you've shown here is pandas 101. If it is confusing, it's a sign you need to understand the underlying basics of NumPy and pandas.
This seems to come up every single time these libraries are brought up around here. People who are primarily programmers don't seem to understand these libraries weren't written for them. Wes McKinney was working at a hedge fund when he wrote pandas. The history of Numpy dates back to researchers at MIT and LLNL petitioning Guido to extend the Python syntax to facilitate array programming, because it is supported in programming languages primarily used by scientists and engineers like MATLAB, R, S-Plus. Broadcasting in these languages is fairly intuitive and well understood by the users, and not considered surprising, because it matches pretty closely the way you symbolically manipulate vectors, matrices, and tensors on paper.
The array syntax is not what you expect if you're coming from a background primarily in programming applications using other dynamically-typed languages or even Python itself, but it's exactly what you'd expect if you're a scientist used to using MATLAB, or even just used to doing math on paper. And at least back in the early days, that was how hedge funds and quantitative finance worked. They weren't looking for application programmers. They were looking for physicists and electrical engineers. It was a lot easier to teach them how to program than to teach programmers about Brownian motion and stochastic processes.
For their intended use cases, Numpy and pandas are among the greatest tools ever created, honestly comparable to MATLAB itself except also free.
One funny thing, it's only in the Python world that C is a defined speed. When we implement libraries in actual C, C++, Rust etc, we benchmark and compare different libraries, they can have wildly different performance - depending on you know, work put into them. But Python is just a two-speed world: Python or C ;-)
Honestly "a great CSV reader" is the only thing it has going from my experience. Other then that, it is not intuitive, a kitchen sink of features, memory greedy.
I'm relatively new to Pandas after years of working with Apache Spark. I've found it to be relatively intuitive. From what I understand, Pandas has gotten much less memory greedy over time as PyArrow has been integrated. They just released a new type to make strings much more efficient for example: https://youtu.be/_zoPmQ6J1aE
It's probably good for someone who hasn't really done programming before, but for me I can certainly agree with it being unintuitive. Like putting filters within square brackets. It felt so unnatural to put something that was evaluated across all rows into what should just take an index or key.
And there's gotchas like join() not actually working like you'd expect a join to coming from SQL. IIRC a SQL-like join is done via merge().
And then I still don't understand when I need to differentiate between indexes and columns. I don't even know why they need to be differentiated at all. Would it really have been that hard for them to design indexes to always project out to a column so you can treat them as such?
So this thing is 3500 pages. Either pandas is overly complex and over engineered or it's a meticulously planned library that thoughtfully provides tools and use case coverage for most data analysis jobs.
What's worse is the docs often don't document all values for 'enums' or other types of parameters. Most is copy and paste from someone else. Very painful to debug and upgrade.
Honestly, I thought you were going to say it’s the latter. It’s great. It has a huge API because it solves a lot of problems.
Does some stuff get complicated? Sure. Sometimes there are edge cases that get messy. The other day I was surprised to see my categorical labels implicitly converted to strings if they were used in a multi-level index. But for me, at least, that’s a fairly esoteric use-case. It stands out to me now precisely because pandas so rarely does something I didn’t expect.
This manual is pretty nice. I usually just google when I don't know (or just forgot) how something works. Over half the time, I end up on the right page in the official docs. The rest of the time, on SO.
But I could imagine that, if I were about to do something quite different from things I've done before, that reading the relevant chapter in the book. For example, of course I've dealt with dates and times using Pandas. But I haven't used time spans. Instead of googling when I get stuck, I could read section 2.20.
Yeah I think the language server doesn't help if you don't know (or can't guess) about the existence of something.
e.g. if you don't know about pandas.Series.str then you probably wouldn't find df['colname'].str.upper just by hitting tab. So, yes good to have Google and also an offline manual that has approximately everything.
I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data.
I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient.
Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin.
The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).
The issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function:
You can use PySpark without ever writing or thinking about Java code. The library handles all the interop for you and even does vectorized serialization with PyArrow. The same way machine learning code is actually mostly C++ or CUDA but you almost never think of that when using Python libraries.
I haven't tried Julia yet, but suspect there is a large class of problems that are best suited for Julia. Your question motivated me to do a bit a research and it looks like a cloud based Julia runtime exists (JuliaHub). It looks like a promising solution, especially if you need to leverage a Julia lib.
Pandas is not a Python library, really, it is a DSL for vectorized ETL work. Like any language you have to learn the idioms.
Pandas is the swiss army knife of data science prep work. There are very few things it can't do. But to achieve that flexibility, it has a very complex API. That can be confusing when starting out. A lot of beginner work feels like just memorizing arbitrary recipes. Once you understand it, it becomes much more powerful. When I first started, the gratuitous use of nested square brackets drove me crazy. Now I understand what's happening beneath the hood, and I use that expressive power on a daily basis.
The critiques of memory usage are well founded and hopefully the underlying tech implementations continue to improve over time.
pandas is a very powerful library, but it has the highest stackoverflow-search-per-lines-of-code-written ratio I've ever experienced. Regularly exceeds one.
It's funny, I have no problem understanding basic syntax of numpy, but when I meet a DataFrame, my brain just can't.
I'm not sure where I'm going with this, and I may be completely off the mark but: Pandas, like SQL, feels like it doesn't compose very well. If you know some basic programming language — including Basic! — you've got a set of building blocks that let you, at least in principle, implement any algorithm. But in Pandas (and SQL) everything needs a new keyword or method or something. We need a better way of expressing high-level concepts, but if I knew what that was, I'd be a high-flying professor somewhere, not an anonymous commenter on HN!
SQL expresses programs via the relational model. Pandas is similar. I’m not an expert on the areas, but it seems that relational modeling is the best tradeoff between performance and abstraction.
To your point, people did have a “programming language” for databases and it was called CODASYL. It died in the 1970s in favor to the relational model. NoSQL and other recent trends have similarly been criticized pretty heavily compared to relational databases.
I guess the point is that SQL and friends may feel restrictive but alternatives sacrifice on performance or safety.
69 comments
[ 3.2 ms ] story [ 112 ms ] threadIf you think that these complaints are not valid or stem from a lack of experience, I urge you to use R first.
(I’ve heard tell that R is worse, but I wouldn’t have any opinion on that since I’ve never used the language.)
So it stands to reason there must be a common explanation whey they are all so obtuse, which I would venture is something like: they solve a set of problems so broad and complex that the tools themselves must also be massively broad and complex, and so the only way to avoid API inconsistencies is to have the system designed whole-cloth from the very beginning, but doing so would make the development process so huge that it would take a decade before you had something ready for users and so it would never get off the ground. The tools that succeed in reality are the ones that solve a common subset of problems, and then grow organically as they are extended to solve other subsets, and the varying subsets have logical inconsistencies and conflicts, thus the state of tools available today having APIs with inconsistencies and conflicts.
That said, pandas is really good at solving problems with tabular data, and even though it's not easy to do complex work with I still like and appreciate it very much.
The alternative really is to write simple imperative style scripts with for loops in your favorite scripting language. Script will be much longer and likely more prone to errors, but the learning curve for simple tasks is just a lot less steep.
Eventually learning the array oriented approach is worth it though. It's the usual trade off in how much time is it worth learning a new tool vs kludging some old tool that doesn't quite fit for the task.
Take this with a grain of salt from someone who needs data manipulation occasionally every now and then (as opposed to being a full time number-cruncher, data-scientist, statistician etc.), using krangl[1] for Kotlin has been a great experience.
I was drawn to this library because I use Kotlin in my dayjob for backend development, but I love how well Kotlin's succinct syntax & features like extension functions lends itself to data transformation & ETL kind of use cases.
Also it doesn't hurt that JVM has a plethora of libraries available for things like DB access, plotting, etc.
I am sure that Pandas has many features I am unaware of, and for a lot of people the high-ish startup time can be a deterrant, but for most of my day to day data munging the combination of jbang, krangl & kravis has been a pretty good fit.
[1] https://github.com/holgerbrandl/krangl
Lots of GUI based alternatives for data munging such as: Easy Data Transform, Alteryx, Knime. Whether they are better than text based tools, such as R or Python+Pandas, depends a lot on your use case and experience.
Data Science happened to python largely against its will (ie., that of GvR) -- and language-level support simply doesn't exist.
It can be both, but it doesn’t have to.
Python is an imperative, object-oriented language largely concerned with side-effecting programs (web severs, os automation, etc.).
It is about as hostile as it is possible to be, from "ideas" pov, to data science.
However those ideas are just a syntax skin over something much more flexible: pointers and dictionaries (/hashmaps). These are like the "Python VM Language" on which "Data Science" sits.
Data Science libraries essentially use the "actual" semantics of python and ignore the "official" syntax, as that's the only way of providing an API that makes any sense. (Ie., isnt just reams of for-loops with 5 sequential obj.method() calls).
I don't believe there is a "non-confusing" way of ignore python's syntax, *within python*; and that's just what is required.
Your second and third statements are simply false.
Also, lacking skills in first order logic casts doubt on any kind of "understanding".
But please, do not assume I haven't read your arguments.
`object = state` is not a paraphrasing of first order logic; it is a paraphrasing of the english sentence,
"The intention motivating OOP is that the object represents state; the object's API represents changes to its state; an encapsulation ensures those changes are privately managed."
which if you understood any of the context to this discussion, you would have captured -- see, eg., the other commenter who captured this accurately.
I have taught hundreds of hours on programming language theory, and it is what I am doing today; and it is something I have been doing for a decade (not least, programming for two decades).
I take this tone with you because assuming a person cannot /reason/ basically is an extraordinarily arrogant view which presumes you have precisely captured the context of their statements.
In cases of apparent "basic failures to reason appropriately" it is almost always the observer, not the speaker, who has failed to arrive at the correct "propositional model" of their spoken/written language.
You would do well to remember this, and include more humility in your communications.
First of all, it does not matter what the initial intention was when OOP started growing to where it is now. The whole historical argument makes no sense whatsoever. The fact that something was like X in the past does not mean it is going to be like X in the future.
Second, I have a lot of doubt about the statement, that the idea of object API was solely to represent changes to object state. Unless you can link a supporting quote, I am ready to dismiss it as made up.
It is certainly interesting though, how there are so many people firmly in opposite camps.
I really cannot imagine how pandas could be considered well put together; I'm always searching stackoverflow for data transformations I assume should be common and simple.
In dplyr, on the other hand, each function does one and only one thing (the touted Unix principle) and you just pass the data along via the pipe %>% operator
[1] I have never failed to find an answer for Pandas on SO for even the most convoluted manipulations I'm attempting. The first hit I look at usually has my answer.
Like this standard usage which misuses all the operators:
One would think that would be equivalent to b[true] or false, and what's that supoosed to mean? But if c is a numpy array it's instead doing element wise comparison on c, and the new array of booleans will be used to index into b. If you don't know this, it's impossible to figure out looking at the code. Too many footguns imo. Same with pytorch, broadcasting is nice, but makes it hard to figure out what's going to happen in practice with the shapes being calculated upon.I feel Notebooks aren't used as much because they're a great tool, but that they are used so much as a necessity since you need to look at live data to have any hopes of understanding what's going on. Why couldn't a compiler instead tell me what I'm trying to do will fail?
Last year, I had an in-person discussion with a numpy core developer in which I proposed adding pandas-like syntax to numpy, e.g. allowing the author to use:
a.boolmask[b == c]
(but maybe not so verbose).
A problem immediately arises: An array can have multiple axes, and each could be indexed with a different type within the same slicing call.
One idea might be to allow the author to explicitly wrap the arguments with some dummy class to make clear what types they’re using:
a[np.boolmask(b == c), d, np.intlist(e)]
(Again, perhaps choosing shorter names in practice.)
The idea would be that the wrapper merely “looks like” the thing it wraps, but contains an assertion to verify the type of the argument.
Anyway, I agree that notebooks (or REPLs in general) are great for verifying your assumptions about how little snippets will behave. But I think they would be useful even in a fully statically-typed compiled language, too.
A more weird but still basic example from pandas is the way it handles indexing and the confusion between .loc, .iloc, .ix, .at and plain '[]'. That part of the API should be really basic but for some reason it's a giant mess.
What you've shown here is pandas 101. If it is confusing, it's a sign you need to understand the underlying basics of NumPy and pandas.
The array syntax is not what you expect if you're coming from a background primarily in programming applications using other dynamically-typed languages or even Python itself, but it's exactly what you'd expect if you're a scientist used to using MATLAB, or even just used to doing math on paper. And at least back in the early days, that was how hedge funds and quantitative finance worked. They weren't looking for application programmers. They were looking for physicists and electrical engineers. It was a lot easier to teach them how to program than to teach programmers about Brownian motion and stochastic processes.
For their intended use cases, Numpy and pandas are among the greatest tools ever created, honestly comparable to MATLAB itself except also free.
One funny thing, it's only in the Python world that C is a defined speed. When we implement libraries in actual C, C++, Rust etc, we benchmark and compare different libraries, they can have wildly different performance - depending on you know, work put into them. But Python is just a two-speed world: Python or C ;-)
Apart from using xarray whenever feasible, but that's just one small part (and always in conjunction with pandas).
And there's gotchas like join() not actually working like you'd expect a join to coming from SQL. IIRC a SQL-like join is done via merge().
And then I still don't understand when I need to differentiate between indexes and columns. I don't even know why they need to be differentiated at all. Would it really have been that hard for them to design indexes to always project out to a column so you can treat them as such?
[0]:https://github.com/ankane/rover
[1]:https://github.com/SciRuby/daru
Spoiler alert: it's the former.
Does some stuff get complicated? Sure. Sometimes there are edge cases that get messy. The other day I was surprised to see my categorical labels implicitly converted to strings if they were used in a multi-level index. But for me, at least, that’s a fairly esoteric use-case. It stands out to me now precisely because pandas so rarely does something I didn’t expect.
But I could imagine that, if I were about to do something quite different from things I've done before, that reading the relevant chapter in the book. For example, of course I've dealt with dates and times using Pandas. But I haven't used time spans. Instead of googling when I get stuck, I could read section 2.20.
e.g. if you don't know about pandas.Series.str then you probably wouldn't find df['colname'].str.upper just by hitting tab. So, yes good to have Google and also an offline manual that has approximately everything.
Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/
I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient.
Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-a...
Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin.
The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).
Pandas is the swiss army knife of data science prep work. There are very few things it can't do. But to achieve that flexibility, it has a very complex API. That can be confusing when starting out. A lot of beginner work feels like just memorizing arbitrary recipes. Once you understand it, it becomes much more powerful. When I first started, the gratuitous use of nested square brackets drove me crazy. Now I understand what's happening beneath the hood, and I use that expressive power on a daily basis.
The critiques of memory usage are well founded and hopefully the underlying tech implementations continue to improve over time.
It's funny, I have no problem understanding basic syntax of numpy, but when I meet a DataFrame, my brain just can't.
To your point, people did have a “programming language” for databases and it was called CODASYL. It died in the 1970s in favor to the relational model. NoSQL and other recent trends have similarly been criticized pretty heavily compared to relational databases.
I guess the point is that SQL and friends may feel restrictive but alternatives sacrifice on performance or safety.
Pandas is powerful, but I can never get a command to work on the first try.