I had previously read an article about making all computation happen through linear algebra, they seemed pretty close to that goal which would allow any python operation to run on GPUs.
Heuristic parsing of title: probably benchmarking x=a*b; that's not how you do heavy math in Python, A100 needs lots of set up, oh wait, Carmack, he knows all that. Moving on.
I don’t think you’ve actually read the blog post behind the Twitter thread. For starters, it’s not Carmack making that claim but merely quoting it, and secondly, it’s much more interesting than benchmarking “a*b”.
It's actually saying Python can do roughly 32 million additions per second on his local hardware setup. I just tested mine, and it's a bit faster than that, but those aren't floating point operations anyway, and should actually be faster than FLOPs.
You've got multiple for loops over every digit, array dereferencing, plus the actual adds. Thus, "overhead." Python takes care of a lot for you, making it possible to do whatever you want with thousand digit numbers and it'll just work, but is definitely not meant for doing lots of math with small numbers. Hence why something like PyTorch even exists.
I agree, the point isn't really about how Python is super slow compared to C++, or any language.
The point is about how easy it is for overhead to creep in when you're working with these unbelievably fast accelerators. Like, the point is that if you perform a single addition in Python, and then have your GPU crunching away at a matmul simultaneously, your GPU is going to get through 9.75 million FLOPS before Python finishes its single add.[1]
Even if you did it in C++, how many single-threaded adds could you do? Probably on the order of a couple billion (let's round (generously) up to 10 billion). In that case, in the time you finish one add, an A100 will finish 31 thousand FLOPS.
Of course, you can leverage multithreading/SIMD if it really is parallelizable. But say, if you're doing a ref-count bump, your A100 will finish 31 thousand FLOPS before you finish that ref-count bump.
Yes but if everything from the ground up is faster you might avoid some of the wrapper overhead. This is the problem with "legacy" languages like python. There's still a big difference between 9.75 million and 31 k, especially if you multiply that out.
I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.
Oh, wrapper overhead of Python is certainly annoying.
There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU.
No Flops are floating point operations. GPUs are better at them because they run floating point operations in parallel on thousands of optimized cores. While python will run a floating point operation serially on (likely) a single general purpose CPU core. None of this really pertains to websites where the main bottle neck is IO so python is as good as anything else.
I think it's a joke, they're likely aware of what you said. It's also possibly a sarcastic way of pointing out what you said, that a lot of things Python is used for are things where the main bottle neck is IO, so things like this should be read with such context taken into account.
And if it wouldn’t have tried manipulating my emotions and gain access to my attention center, there were a chance that I would have read the article. Now I won’t, as the article apparently isn’t interesting on its own merits.
A non sequitir consider no one uses pure Python for numeric compute. People use numpy which is a wrapper around optimized BLAS code, which is usually running at the peak capacity of the CPU. So really this is a statement about how much faster GPUs are than CPUs for doing numeric compute.
Even using numpy you leave a LOT of performance on the table because you are till using python to do at least some things. It's not uncommon to see a complete function rewritten in C to be 5x+ faster then python numpy.
24 comments
[ 3.2 ms ] story [ 64.0 ms ] threadhttps://horace.io/brrr_intro.html
If anyone has a link to that please post.
Python uses arbitrary precision integers, though, so it's doing more than just a single hardware add per Python "add." You can see the actual implementation here: https://github.com/python/cpython/blob/main/Objects/longobje...
You've got multiple for loops over every digit, array dereferencing, plus the actual adds. Thus, "overhead." Python takes care of a lot for you, making it possible to do whatever you want with thousand digit numbers and it'll just work, but is definitely not meant for doing lots of math with small numbers. Hence why something like PyTorch even exists.
I agree, the point isn't really about how Python is super slow compared to C++, or any language.
The point is about how easy it is for overhead to creep in when you're working with these unbelievably fast accelerators. Like, the point is that if you perform a single addition in Python, and then have your GPU crunching away at a matmul simultaneously, your GPU is going to get through 9.75 million FLOPS before Python finishes its single add.[1]
Even if you did it in C++, how many single-threaded adds could you do? Probably on the order of a couple billion (let's round (generously) up to 10 billion). In that case, in the time you finish one add, an A100 will finish 31 thousand FLOPS.
Of course, you can leverage multithreading/SIMD if it really is parallelizable. But say, if you're doing a ref-count bump, your A100 will finish 31 thousand FLOPS before you finish that ref-count bump.
I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.
There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU.
But yeah, the Python overhead is annoying (see https://dev-discuss.pytorch.org/t/where-we-are-headed-and-wh...), just giving an explanation why it hasn't been a dealbreaker.
Not sure what I'm supposed to take away from this tweet...
[*] I like my joke, so I'll leave it be, but fwiw I missed an M ><