PandaPy has the speed of NumPy and the usability of Pandas (github.com)

And I also. The other poster, in my opinion, has it backwards. Seeing how each option can be used in a myriad of examples is particularly helpful and saves me much time struggling to work out what the right method/ tool is, and how to use it.

Pandas has enough gotchas that it looks friendly until you hit one of them. Examples of gotchas:

Want to join two dataframes together like you'd join two database tables? df.join(other=df2, on='some_column') does the wrong thing, silently, what you really wanted was df.merge(right=df2, on='some_column')

Got a list of integers that you want to put into a dataframe? pd.DataFrame({'foo': [1,2,3]}) will do what you want. What if they're optional? pd.DataFrame({'foo': [1,2,3,None]}) will silently change your integers to floating point values. Enjoy debugging your joins (sorry, merges) with large integer values.

Want to check if a dataframe is empty? Unlike lists or dicts, trying to turn a dataframe into a truth value will throw ValueError.

  left = pd.DataFrame({"abcd": ["a", "b", "c", "d"], "something": [1,2,3,4]})
  right = pd.DataFrame({"abcd": ["d", "c", "a", "b"], "something_else": [4,3,1,2]})
  left.join(other=right, on="abcd")
  
  ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

  left = pd.DataFrame({"abcd": ["a", "b", "c", "d"], "something": [10,20,30,40]})
  right = pd.DataFrame({"abcd": ["d", "c", "a", "b"], "something": [40,30,10,20]})
  
  left.join(other=right, on="something", rsuffix="_r")
  
    abcd  something abcd_r  something_r
  0    a         10    NaN          NaN
  1    b         20    NaN          NaN
  2    c         30    NaN          NaN
  3    d         40    NaN          NaN

  left = pd.DataFrame({"abcd": ["a", "b", "c", "d"], "something": [1,2,3,4]})
  right = pd.DataFrame({"abcd": ["d", "c", "a", "b"], "something": [4,3,1,2]})
  left.join(other=right, on="something", rsuffix="_r")
  
    abcd  something abcd_r  something_r
  0    a          1      c          3.0
  1    b          2      a          1.0
  2    c          3      b          2.0
  3    d          4    NaN          NaN

  left.merge(right=right, on="something", suffixes=['', '_r'])
  
    abcd  something abcd_r
  0    a         10      a
  1    b         20      b
  2    c         30      c
  3    d         40      d

  pd.DataFrame({"foo": [1,2,3,4,None,9223372036854775807]})
  
              foo
  0  1.000000e+00
  1  2.000000e+00
  2  3.000000e+00
  3  4.000000e+00
  4           NaN
  5  9.223372e+18

  df = pd.DataFrame({"foo": [1,2,3,4,None,9223372036854775807,9223372036854775806]})
  
              foo
  0  1.000000e+00
  1  2.000000e+00
  2  3.000000e+00
  3  4.000000e+00
  4           NaN
  5  9.223372e+18
  6  9.223372e+18
  
  df["foo"].unique()
  
  array([1.00000000e+00, 2.00000000e+00, 3.00000000e+00, 4.00000000e+00, nan, 9.22337204e+18])

    pd.DataFrame({"foo": [1,2,3,4,None]}, dtype=pd.Int64Dtype())

        foo
    0     1
    1     2
    2     3
    3     4
    4  <NA>

    pd.DataFrame({"foo": [1,2,3,4,None,9223372036854775807,9223372036854775806]}, dtype=pd.Int64Dtype())

                       foo
    0                    1
    1                    2
    2                    3
    3                    4
    4                 <NA>
    5  9223372036854775807
    6  9223372036854775806

You can’t pick up a dictionary and determine how difficult it is to speak a language.

One of the most difficult things about pandas is knowing how to shape your data frame before you plug it in to something. Most of my frustration comes from finding the magic incantation of symbols that will reshape my data frame the right way.

Yeah, I agree: it's super straightforward. It's hard for me to understand how people could find it inscrutable / confusing. Pandas is a LIFESAVER for anyone who needs to manipulate datasets programmatically.

I learned Python by using pandas so for awhile I was one of the devs you speak of. I can't imagine I'm alone.

I feel like the documentation is great if you already know how it works. Having learned Pandas recently, I'll confess there are parts that are a bit tricky to get past the basics. I know R dataframes which was helpful.

Pandas is well worth it to learn however if your using python.

I imagine OP is referring to things like conditional selection on multiple columns, not by AND && operators but by bitwise & operators.

I've mostly use comprehensions to filter and manipulate data and I find the Panda API to be a bit clunky and esoteric.

There are nice built in functions and it displays well in a jupyter notebook but I'm not a fan of the interface.

I mean, whenever I have this problem, I fall back on my (terrible) SPSS habits and just recode NaN to -999. With integer datasets that mostly works fine. You could come up with some alternative solution, too.

Personally, I'm an amateur at best, but Pandas has made it possible for me to do some really handy things.

Personal favorite feature: multindexing on rows AND Columns. Multiindex is such a common pattern, and so poorly handled in things like excel. Pandas really saves a lot of time with re-indexing, pivoting, stacking, or transposing data with multi indexes.

You should upgrade your version of pandas if possible - that's been fixed for a few versions now.

You can use "Int64" for nullable integer data type.

https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...

And then they learn how to game the benchmarks.

You just can’t win.

I am writing a small benchmarking site (aimed at asymptotic performance rather than singular tasks) which should hopefully have a fairly generic API/Specification, so I'll look into the more "competitive" side of it (Project vs Project as opposed "Uh oh, Commit f4r0adfja has made the build slower and [test 3] slower for large n".

The logic itself is easy but I'm having to step into the dark arts of DevOps and web-cancer - should be online by the summer.

This is a real problem, akin to how modern JITs will have diff run modes for the same code. We're overdue for pandas-without-huge-overhead.

See this thread for more on why pandas overhead adds up for real settings, and is crazy huge once you go to dask/ray/modin, and even worse, say spark: https://twitter.com/lmeyerov/status/1218093436286296069 + https://twitter.com/lmeyerov/status/1220847229537157121 . For interactive analytics apps, we want scale for bigger workloads, yet also less overhead to achieve < 100ms: ideally moving a slider will trigger lots of calcs over many widgets and their subcomponents, and the underlying stats code looks normal (e.g., df > sql > some imperative/functional lang.) Internally we write C-like buffer manips & hand-written GPU kernels & weird streaming code to achieve < 100ms wall-clock, and are trying to get tech like these up to snuff for interactive data software.

As per the JIT comment, feels like a matter of time, so happy to see this (and the pressure it creates!)

Most of those disclaimers were added tonight, after seeing my comment :) https://github.com/firmai/pandapy/commit/692c968771bb19d4f12...

> Lots of built in statistical stuff and powerful visualization makes exploring datasets easy

I see. For linear algebra stuff it does not offer anything essential. You rarely see a matrix as a "dataset".

I don't know exactly what you mean, but numpy's hstack and vstack seem to be enough for me.

I think it's partly just the audience. HN seems to have a lot of professional programmers, which is similar to but not quite the same as computational scientists. That doesn't mean there's no overlap, but for example I haven't met a people (other than one Google person) at a recent physics-informed learning or scientific machine learning conference who is a HN reader, which really explained why none of the stories in those domains have been getting HN traction even though HN seems very ML/AI-positive. Julia is in this weird middle ground where its ML/AI stories seem to get a lot of HN traction as a programming languages topic given the HN audience, but in reality it has a lot more people doing things like research in numerical linear algebra, numerical differential equations, statistics, bioinformatics, etc. but fast methods for block-banded matrices won't hit science press.

I’m not sure I just do

using Pkg; Pkd.add(“foo”)

And it works. I’m sure there’s some dependency file for projects but I don’t know how it works. Thinking about build tools and package management makes me sad, so I try and avoid it as long as possible.

Usually pretty good. The package management is completely centered around Pkg.jl, which is integrated in the REPL and you can also import it to your program for more advanced scripting. If you don't create an environment, everything is added to the global user library, and if you do create it, it will automatically manage your projects dependency files, and each environment/package can have it's own independent versions of each library (so you don't really have dependency hell issues, but you might end up with more disk space due to multiple versions of the same library, although it will respect semver when keeping multiple libraries).

Pkg.jl is based on git/github with a central registry that can I believe it's automatically updated with new packages using bots. The current version also has native supports for automatically deploying binaries and other stuff like datasets, that can be optionally loaded on demand.

Most troubles I hear is with more strict enterprise firewall scenarios and perhaps Julia JIT making it compile the libraries every time (though that's not an issue with the package manager).