I'm reminded of how Carmack talked about the extra efficiencies available when targeting consoles, because you knew exactly what hardware was available.
It's great that the efficiencies available can be shown to be extractable. The real, much harder, trick is putting together a sufficiently smart compiler to enable them for heterogeneous compute setups.
Only a matter of time until we start seeing bogus Hard Science papers like that, now that we've given the Social Text people the tools they need to take their revenge.
They will argue that we had it coming, and that it serves us right, and maybe they're not wrong.
> It is sensitive to compiler versions, GPU setup, and sometimes even being looked at the wrong way, and we have no intention whatsoever of supporting it.
Excellent writeup. I like the interpreter. But I can only assume all these ideas have been widely implemented at all significant labs for years, so I'm surprised to see this written in 2025. This is all about taking things to their logical conclusions, not arcane magic. If you're going to spend billions on GPUs, why wouldn't you spend a little on CUDA programmer hours?
MIG virtualization is IMHO weak sauce. Only seven slices. Seven? Extremely limited hardware support. Difficult to configure - like the early days of CUDA. It’s been in the works for what 7 years now and barely functional.
Meanwhile, don’t forget that if your workloads are cooperative, you can put all the processes you want on a single GPU and they’ll happily multitask. No security boundary of course, but who knows how good MIG is at that.
I’d greatly prefer better tools for cooperative GPU sharing like per process memory limits or compute priority levels. Also seems like it should be way easier to implement. As containerization and k8 have proven, there’s a ton of utility in bin packing your own workloads better without rock solid security boundaries.
The sentiment in the title resonates, but for consumer GPUs (the article is about server cards).
The recently leaked M5 benchmarks reveal a 35% faster GPU. These improvements compound, so you can get a GPU that's effectively twice as fast by waiting a couple of years.
Modern GPUs are the equivalent of local supercomputers, but the drivers, languages and libraries are still playing catch up. Imagine the audio processing you could do if only you could target that hardware.
It would be nice if the article header would actually be clear that they are optimizing a CUDA chip. There is a difference between a GPU and a CUDA chip.
Not using the NVDEC and NVJPG units to decompress weights into registers? And you say you're using the whole GPU. There are entire blocks on the silicon going idle!
I bought a car with side impact airbags, so we’re damn well going to use the side impact airbags.
Maybe… you don’t actually want or need to use all the features of something you bought. Particularly given that GPUs previously used for cryptocurrency mining may have damaged themselves while being run full out for a year straight.
> please be warned that this really is research code; it is sensitive to compiler versions, GPU setup, and sometimes even being looked at the wrong way
the writeup is a classic example of what we lose through abstraction and how writing custom (and optimized) code still beats sticking to high-level implementations.
i would go further and say that the "megakernel" written as part of the optimization is highly-model dependent as well.
the whole "cuda moat" is from the generic implementations of the moving parts of the model architecture. at the same time, you lose a lot of performance through the generic code. it is like comparing writing a stock trading algo in next.js vs assembly.
training models is another landscape altogether, so props to those who can quickly adapt to the hardware they got.
18 comments
[ 0.20 ms ] story [ 34.2 ms ] threadAccept!
It's great that the efficiencies available can be shown to be extractable. The real, much harder, trick is putting together a sufficiently smart compiler to enable them for heterogeneous compute setups.
Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity
https://physics.nyu.edu/faculty/sokal/transgress_v2/transgre...
They will argue that we had it coming, and that it serves us right, and maybe they're not wrong.
My favorite type of code
* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
* https://www.nvidia.com/en-us/technologies/multi-instance-gpu...
Or having multiple processes from one user share it:
* https://docs.nvidia.com/deploy/mps/index.html
Meanwhile, don’t forget that if your workloads are cooperative, you can put all the processes you want on a single GPU and they’ll happily multitask. No security boundary of course, but who knows how good MIG is at that.
I’d greatly prefer better tools for cooperative GPU sharing like per process memory limits or compute priority levels. Also seems like it should be way easier to implement. As containerization and k8 have proven, there’s a ton of utility in bin packing your own workloads better without rock solid security boundaries.
The recently leaked M5 benchmarks reveal a 35% faster GPU. These improvements compound, so you can get a GPU that's effectively twice as fast by waiting a couple of years.
Modern GPUs are the equivalent of local supercomputers, but the drivers, languages and libraries are still playing catch up. Imagine the audio processing you could do if only you could target that hardware.
Maybe… you don’t actually want or need to use all the features of something you bought. Particularly given that GPUs previously used for cryptocurrency mining may have damaged themselves while being run full out for a year straight.
the writeup is a classic example of what we lose through abstraction and how writing custom (and optimized) code still beats sticking to high-level implementations.
i would go further and say that the "megakernel" written as part of the optimization is highly-model dependent as well.
the whole "cuda moat" is from the generic implementations of the moving parts of the model architecture. at the same time, you lose a lot of performance through the generic code. it is like comparing writing a stock trading algo in next.js vs assembly.
training models is another landscape altogether, so props to those who can quickly adapt to the hardware they got.