when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot.
Compare and contrast with languages with built-in N-dimensional array for scientific computing.
I'm confused by what you mean. In Matlab, everything is a matrix. A naked integer is a 1x1 matrix, hence why the size is reported as [1 1]. Those are the semantics of the language, not a lie.
a naked integer is not a 1x1 matrix, it’s a 0x0. Matlab silently promotes it to one, though, which is a deliberately misleading thing to do because it’s convenient for matlab’s data model.
it should be clear that i don’t care very much about what matlab has to represent things as. i don’t mean “matlab integers”, i mean the mathematical objects; matlab’s representation of them is poor.
Okay, well that's not Matlab lying, that's you not understanding how the tool works. It would be like saying that 0.1 + 0.2 == 0.3 evaluating to false is a lie, because floating point numbers don't accurately represent real numbers as a concept. But floating point numbers aren't actually real numbers, and integers in Matlab (they're actually stored as floating point doubles, is that a lie too?) aren't actually mathematical objects.
You have to work within the constraints of the tool. Matlab makes it very clear upfront that the semantics of the language mean that naked numbers are in fact matrices, because everything in the language is a matrix. Maybe you feel that's not a good representation, but it's not a lie, and in fact it's very consistent and useful in the context of the language itself.
Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.
Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.
My point is that even the trivial implications of the definition are absurd; without inspecting non-trivial broadcasting cases we can already determine the API is poorly designed.
Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity.
My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.
But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.
For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood.
I'm happy I moved on to Go, which actually adopts this as a core design tenet.
To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.
Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.
That does not necessarily follow. Python may be striking a balance between the needs of numerical and non-numerical computing, while NumPy arrays do not need to consider non-numerical computing.
Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing:
This has a high-startup cost which makes NumPy use inappropriate for short-lived programs like shell scripts, even in cases where Python's own startup cost is otherwise appropriate.
I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.
Meh. Footguns are underrated. Let’s return to the Wild West of the 90s.
I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
All these talk about code safety just want me to stop wearing my bike helmet.
For most of my programs safety is of zero concern. I'll trade it for convenience any day. Just let me implicit cast signed to unsigned thank you very much.
Because the typing system used in Python is a big footgun. If the basis of the language you are using encourages footguns then it's not surprising that a large library in that ecosystem doesn't care about that either.
Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.
Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare.
They changed the language to change what question `<` asks.
It really doesn't if you think about it. a==b cannot by itself be used as an if condition. You would always need to use (a==b).all(). So it would be very difficult to make the mistake you are suggesting
I think the general point is that it was a bad idea to implement the operators as elementwise because it breaks all of the properties they are supposed to hold.
Really there should be separate elementwise operators like `.>` and so on but you can only do that in some languages.
Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.
Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.
Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute!
No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.
Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.
51 comments
[ 3.2 ms ] story [ 115 ms ] thread> Numpy doesn't distinguish "vector of vector" from "matrix"
though to be fair, it’s on its way out.Compare and contrast with languages with built-in N-dimensional array for scientific computing.
Numpy can:
i think it’s fair to call that a (white) lie
Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.
it should be clear that i don’t care very much about what matlab has to represent things as. i don’t mean “matlab integers”, i mean the mathematical objects; matlab’s representation of them is poor.
You have to work within the constraints of the tool. Matlab makes it very clear upfront that the semantics of the language mean that naked numbers are in fact matrices, because everything in the language is a matrix. Maybe you feel that's not a good representation, but it's not a lie, and in fact it's very consistent and useful in the context of the language itself.
I don't think so. The array module was included even in Python 1.0.
Or did you mean to use another verb?
but, cough, mature and stable scientific computing foundation?
Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.
My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans.
But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence.
Reading https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.
I'm happy I moved on to Go, which actually adopts this as a core design tenet.
[1]: Even though PEP-20 is largely a meme.
Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing:
This has a high-startup cost which makes NumPy use inappropriate for short-lived programs like shell scripts, even in cases where Python's own startup cost is otherwise appropriate.I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
For most of my programs safety is of zero concern. I'll trade it for convenience any day. Just let me implicit cast signed to unsigned thank you very much.
They changed the language to change what question `<` asks.
https://stackoverflow.com/questions/10062954/valueerror-the-...
Really there should be separate elementwise operators like `.>` and so on but you can only do that in some languages.
Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container
https://github.com/dkogan/numpysane
Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html