Everyone uses numpy, but not because they want to or the API itself provides some massive value. They use it because unless you're calling into C for literally every operation, Python is far too slow to get any scientific computing work done.
And because of this, numpy has to be much larger and less orthogonal. It needs to provide a wide range of functions that ensure that the user will never have to drop into regular Python and suffer a 100x performance hit. This is unfortunate, and prevents numpy by having the beauty and orthogonality of true array languages like apl, j, or kdb+/q.
This problem makes it difficult for new scientific users of Python to write fast code. Everything can be fast and then you add one if statement and the performance falls off a cliff. I've helped many people in various labs as this is a recurring problem that researchers face.
As such, the ease of use that Python supposedly has for scientific computing is a lie. The APIs are extremely complex, from numpy to Pandas to statsmodel and scikit. And they have to be, because scientific Python can't compose code due to performance reasons.
In contrast, Julia is much easier to learn, easier to use, and is pretty much better than Python for scientific applications in every way imaginable. In other words, it is strictly superior. The performance is consistently fast and array manipulation is built into the core language, unlike with Python. You are free to write Julia code instead of shelling out to C.
I'll probably get down voted, but Python should not be used for new scientific computing projects. I've had tons of experience with ml/financial Python and after trying Julia, I'll never go back willingly. The difference in speed, complexity, ergonomics, and expressivity is so stark it's hard to imagine why anyone would choose Python for a greenfield scientific computing project in 2020, especially since Julia can trivially access all of Python's ecosystem.
I started a new project last year (finance not scientific but I'd expect lots of similar characteristics - lots of use of dataframes etc). I'm not a massive fan of Python - much prefer Ruby for scripting for example. Had a look at Julia but I still chose Python though as:
- It exposes an API and I want as big an audience as possible and widespread familiarity with Python means that there is a lower barrier to adoption;
- The interop works both ways - if really someone wants to use Julia they can;
- I felt the JIT startup times would lead to a poorer user experience in some circumstances.
Essentially, I accept it's not as elegant as Julia but I'm prepared to accept the extra complexity etc behind the scenes to try to help users.
Not saying that this is great (or even the right decision in this case) just that there can be straightforward reasons for choosing Python.
I don't think there's anything with acknowledging the engineering trade-offs that are made between the various approaches. Julia has a lot of advantages in terms of code speed and its multiple dispatch paradigm leads to a lot more code reuse and composability of packages, but there is a trade-off of JIT loading time. I personally think it's the right trade-off, and with the JIT loading time massively decreasing in v1.6 I think it's hard for me to recommend starting new students in anything but Julia. This is especially since, in the fields of scientific computing and their connections to machine learning, model-specific kernels and the speed of calculations are becoming common place, so no matter what someone in this field really needs to know how to write their own high performance code. However, I can still see that it's possible for someone to rationally look at the trade-offs that are made and making a different decision that matches their specific professional use cases.
I think you may be overgeneralizing. If the speed of number crunching is the only concern, then sure, few scientists would use Python, R, or MATLAB. This is not the case in reality. Whether you would run into speed limit of course depends on the size of the problem. It's fine to use a "slow" scripting language to process a small data set on a personal laptop. And I suspect that this is what most everyday numpy users do.
Once you move to the other end of the spectrum where the size of the problem requires a supercomputer, that is when the Python ecosystem becomes clunky and painful. I think that could be an area that Julia shines in. But even that, I'm not sure if Julia is a clear winner yet, as the field is still largely dominated by C, C++, and Fortran.
Also, check ref. 56 in the article. They did cite Julia, among others.
The point is that Julia is easier to use and faster than Python for just about any scientific task. Even if Python is fast enough for you (which it probably isn't), it's harder to use and less expressive than Julia. This is why I consider it strictly inferior: it is not better at anything at all, only worse.
Python is so slow it is even unusable on trivially small data sets. Also it uses a massive amount of ram. I loaded a 2gb csv file the other day in Pandas, and it consumed over 40gb of ram. That's totally unacceptable in my view.
> The point is that Julia is easier to use and faster than Python for just about any scientific task.
Try to make a plot. Last time I checked, the time to first plot was still a pain point of Julia.
> it's harder to use and less expressive than Julia.
I agree that Julia is more expressive because it's a Lisp in disguise. This is definitely a plus. But in practice, I don't find Python to be "harder to use." There is good consistency among the mainstream packages in terms of API. For example, if you have a NumPy function np.mean, you can assume that Pandas would have a method .mean() for the DataFrame, and Dask would have that for DaskArray as well. Not always, but things are moving toward that direction.
> I loaded a 2gb csv file the other day in Pandas, and it consumed over 40gb of ram.
Have you tried the `engine='c'` option when calling pandas.read_csv? Pretty sure there is also another option of chunking that may be useful.
Time to first plot is still painful even in julia 1.5, sure.
But after paying that startup cost, the speed boost can be transformative in terms of the flexibility and dynamism that it buys. I've been using Pluto.jl lately, and interactive data analysis feels like way less of a burden than working in jupyter.
As to harder to use, writing fast vectorized numpy code can take a lot of mental effort. For one project I ended up forcing things into this 5-D array broadcasting mess, where in julia it's way simpler to just write intuitive code that's performant without bending over backwards to avoid explicit loops. For numpy code you can "just" drop down to cython, but for pytorch you can get stuck with thinking up some clever broadcasting solution.
Yeah, string processing in python is slow. But for csv files of size, you are probably better of using SQLite or sql server over even Julia. If pytorch wasn’t able to keep the gpu busy, I would be more inclined to try Julia or lisp again.
to be fair, the Julia citation is sort of a token sentence in the discussion section.
> New generation languages, interpreters and compilers, such as Rust [55], Julia [56] and LLVM [57], will create new concepts and data structures, and determine their viability.
I don't disagree with the statement, but there's not a lot of meat there either.
I'm a hardcore Python user, but am gradually coming around to your point of view. It's frustrating that I can't just use loops to perform array operations, but instead call some very specific Numpy method I have to search for in the docs.
If your loop is gpu bound, it’s going to end up unrolled in a warp anyway...
When we get to a point that machine learning is dominated by tree traversal, than python will be toast.
I think that Numpy with Numba for routines that aren't easily vectorized and or need if statements is just about the perfect combination. https://numba.pydata.org/. Numba functions are very close to C/C++ performance.
EDIT: You can write loops explicitly and have OpenMP style for-loops and they are fast.
Honest question from a heavy python user who would switch if it made sense:
Are there any comprehensive benchmarks that show Julia outperforming Pandas or PyTorch or SciKit?
Obviously pure Python is terrible. But the library algorithms written in C seem fairly competitive.
I’m a fairly boring user who doesn’t do new science, and is fine just composing existing boring algorithms to solve problems in my subject matter domain.
The argument is if you need to do something slightly different, but straight forward (and this doesn't even have to be in you algorithm, it could be in your data prep) you either have to accept a slow loop or go hunt the documentation and come up with some obscure API or some very clever combination of API's.
To my knowledge if you stick to things that calls optimized C and fortran code, it's a draw between the compiled code and Julia.
But even boring problems ends up doing things that are easily expressed in a loop, but ends up being a hard to read chain of pandas.
In the uncommon event I need to write a loop from scratch, and I need it to be really fast, I just rewrite that one jupyter cell in cython or numba. But that is a small piece of my codebase.
I agree that Julia code is aesthetically superior to a long chain of Pandas code. But at this point I’m used to reading a bunch of chained pandas code. Often I think of myself as more of a Pandas programmer than a Python programmer.
I've come to the realization that the real relationship of Python and Numpy is not a language/library relationship, but a script/engine relationship, similar to writing scripts in a game or map editor.
Numpy is not just a helping hand to python; instead, it is this efficient, giant, comprehensive engine, and python is merely the interface used to feed it data, operate it, and read from it.
Of course, under the hood, it's still just language and library. But the mental model is quite different.
The real value of NumPy is that it sets the standard for what we expect in an array processing interface. Of course NumPy did not invent the idea, it builds on many others, but it is the de facto standard for what we expect. In time, another language or array processing library will probably replace Python / NumPy/SciPy, but due to the convenience of the API, I predict whatever takes over will look a lot like NumPy.
23 comments
[ 3.8 ms ] story [ 66.0 ms ] threadAnd because of this, numpy has to be much larger and less orthogonal. It needs to provide a wide range of functions that ensure that the user will never have to drop into regular Python and suffer a 100x performance hit. This is unfortunate, and prevents numpy by having the beauty and orthogonality of true array languages like apl, j, or kdb+/q.
This problem makes it difficult for new scientific users of Python to write fast code. Everything can be fast and then you add one if statement and the performance falls off a cliff. I've helped many people in various labs as this is a recurring problem that researchers face.
As such, the ease of use that Python supposedly has for scientific computing is a lie. The APIs are extremely complex, from numpy to Pandas to statsmodel and scikit. And they have to be, because scientific Python can't compose code due to performance reasons.
In contrast, Julia is much easier to learn, easier to use, and is pretty much better than Python for scientific applications in every way imaginable. In other words, it is strictly superior. The performance is consistently fast and array manipulation is built into the core language, unlike with Python. You are free to write Julia code instead of shelling out to C.
I'll probably get down voted, but Python should not be used for new scientific computing projects. I've had tons of experience with ml/financial Python and after trying Julia, I'll never go back willingly. The difference in speed, complexity, ergonomics, and expressivity is so stark it's hard to imagine why anyone would choose Python for a greenfield scientific computing project in 2020, especially since Julia can trivially access all of Python's ecosystem.
- It exposes an API and I want as big an audience as possible and widespread familiarity with Python means that there is a lower barrier to adoption;
- The interop works both ways - if really someone wants to use Julia they can;
- I felt the JIT startup times would lead to a poorer user experience in some circumstances.
Essentially, I accept it's not as elegant as Julia but I'm prepared to accept the extra complexity etc behind the scenes to try to help users.
Not saying that this is great (or even the right decision in this case) just that there can be straightforward reasons for choosing Python.
Once you move to the other end of the spectrum where the size of the problem requires a supercomputer, that is when the Python ecosystem becomes clunky and painful. I think that could be an area that Julia shines in. But even that, I'm not sure if Julia is a clear winner yet, as the field is still largely dominated by C, C++, and Fortran.
Also, check ref. 56 in the article. They did cite Julia, among others.
Python is so slow it is even unusable on trivially small data sets. Also it uses a massive amount of ram. I loaded a 2gb csv file the other day in Pandas, and it consumed over 40gb of ram. That's totally unacceptable in my view.
Try to make a plot. Last time I checked, the time to first plot was still a pain point of Julia.
> it's harder to use and less expressive than Julia.
I agree that Julia is more expressive because it's a Lisp in disguise. This is definitely a plus. But in practice, I don't find Python to be "harder to use." There is good consistency among the mainstream packages in terms of API. For example, if you have a NumPy function np.mean, you can assume that Pandas would have a method .mean() for the DataFrame, and Dask would have that for DaskArray as well. Not always, but things are moving toward that direction.
> I loaded a 2gb csv file the other day in Pandas, and it consumed over 40gb of ram.
Have you tried the `engine='c'` option when calling pandas.read_csv? Pretty sure there is also another option of chunking that may be useful.
But after paying that startup cost, the speed boost can be transformative in terms of the flexibility and dynamism that it buys. I've been using Pluto.jl lately, and interactive data analysis feels like way less of a burden than working in jupyter.
As to harder to use, writing fast vectorized numpy code can take a lot of mental effort. For one project I ended up forcing things into this 5-D array broadcasting mess, where in julia it's way simpler to just write intuitive code that's performant without bending over backwards to avoid explicit loops. For numpy code you can "just" drop down to cython, but for pytorch you can get stuck with thinking up some clever broadcasting solution.
Julia's GPU programming stack is fantastic and very advanced btw.
> New generation languages, interpreters and compilers, such as Rust [55], Julia [56] and LLVM [57], will create new concepts and data structures, and determine their viability.
I don't disagree with the statement, but there's not a lot of meat there either.
EDIT: You can write loops explicitly and have OpenMP style for-loops and they are fast.
Are there any comprehensive benchmarks that show Julia outperforming Pandas or PyTorch or SciKit?
Obviously pure Python is terrible. But the library algorithms written in C seem fairly competitive.
I’m a fairly boring user who doesn’t do new science, and is fine just composing existing boring algorithms to solve problems in my subject matter domain.
To my knowledge if you stick to things that calls optimized C and fortran code, it's a draw between the compiled code and Julia.
But even boring problems ends up doing things that are easily expressed in a loop, but ends up being a hard to read chain of pandas.
I agree that Julia code is aesthetically superior to a long chain of Pandas code. But at this point I’m used to reading a bunch of chained pandas code. Often I think of myself as more of a Pandas programmer than a Python programmer.
Numpy is not just a helping hand to python; instead, it is this efficient, giant, comprehensive engine, and python is merely the interface used to feed it data, operate it, and read from it.
Of course, under the hood, it's still just language and library. But the mental model is quite different.