> By disassembly of ptxas, it is indeed hard-coded that they have logic like: strstr(kernel_name, "cutlass").
> it is likely that, this is an unstable, experimental, aggressive optimization by NVIDIA, and blindly always enabling it may produce some elusive bugs.
Often not elusive bugs, but elusive performance. GPU compilers are hard: Once you've done the basics, trying to do further transforms in a mature compiler will almost always produced mixed results. Some kernels will go faster, some will go slower, and you're hoping to move the balance and not hit any critical kernel too hard in your efforts to make another go faster.
An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup.
My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.
Someone really needs to learn to use `git commit --amend`. Almost 100 commits with pointless commit messages like "wip" or "x"? Be kinder to your reviewers...
This is weirdly common; phone chipset manufacturers did it with phone benchmarks [0], VW with emissions [1], nVidia did it with 3DMark [2], Intel with the SPEC benchmark for its Xeon processors [3], etc.
When it comes to computer graphics, iirc it's pretty normalized now - graphics drivers all seem to have tweaks, settings, optimizations and workarounds for every game.
(As an aside, I hate that I have to link to archive.org, there's a lot of dead links nowadays but these are important things to remember).
Every vendor does this to this day - and its a morally grey practice, drivers hijack and modify the rendering loops of popular games, fixing bugs, replacing shaders with more optimized versions, enabling faster codepaths in the driver etc.
These changes are supposed to have minimal to no impact on the actual output, but sometimes vendors are really aggressive, and significantly degrade the outputs so that the game can run faster on their hardware.
Reminds me of when ~10 years ago with a particular version of Webpack the build would fail if I had one SVG called add.svg so I had to rename it plus.svg
It would be nice if we could find economics that allowed us to share code instead of all the bullshit with the binary blob drivers. Same for basebands and everything else. How many collective hours and months of our society’s finest minds has been wasted reverse engineering binary blobs, controllers through IO pins, trying to reverse engineer circuit schematics —- when all of this is already sitting on someone’s computer somewhere and they could just GIVE you the docs. CUDA and NVIDIA can go to hell.
Sometimes you write some heavily tuned code in a high level language like C++ that you know could be translated into very specific GPU assembly, then find that the compiler isn't producing the exact assembly that you had in mind.
When you talk to the computer team about it they may offer a range of solutions, some of which may not be applicable to open source code. Picture proprietary #pragmas, intrinsics, or whatnot. What do you do? You can't ship a high performance library that doesn't deliver high performance. It is then when you rely on things like function names to enable specific code transformations that can't be used in general because they would sometimes break third party code.
I never worked on Cutlass, but this is the sort of thing that is done in the real world.
There is nothing nefarious about this sort of optimizatkon. People comparing this to cheating on benchmarks by rendering lower quality images are not on the right track.
21 comments
[ 4.4 ms ] story [ 46.1 ms ] threadhttps://news.ycombinator.com/item?id=44530581
> By disassembly of ptxas, it is indeed hard-coded that they have logic like: strstr(kernel_name, "cutlass").
> it is likely that, this is an unstable, experimental, aggressive optimization by NVIDIA, and blindly always enabling it may produce some elusive bugs.
An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup.
My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.
Nvidia seems to get a pass. Whys that?
They "made something ~100 tflops faster" and peoples' comments are "their commit messages are bad"? You guys would hate how John Carmack worked, too
https://web.archive.org/web/20230929180112/https://techrepor...
https://web.archive.org/web/20011108190056/https://hardocp.c...
https://web.archive.org/web/20011118183932/www.3dcenter.de/a...
Funnily, it's under an older submission of the same cutlass optimizations.
When it comes to computer graphics, iirc it's pretty normalized now - graphics drivers all seem to have tweaks, settings, optimizations and workarounds for every game.
(As an aside, I hate that I have to link to archive.org, there's a lot of dead links nowadays but these are important things to remember).
[0] https://web.archive.org/web/20250306120819/https://www.anand...
[1] https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
[2] https://web.archive.org/web/20051218120547/http://techreport...
[3] https://www.servethehome.com/impact-of-intel-compiler-optimi...
These changes are supposed to have minimal to no impact on the actual output, but sometimes vendors are really aggressive, and significantly degrade the outputs so that the game can run faster on their hardware.
- Intel faced a "cheating compiler" controversy when SPEC, the benchmark standard-setter, invalidated over 2,600 benchmark results for Intel Xeon processors in early 2024. ( https://www.tomshardware.com/pc-components/cpus/spec-invalid... )
- microsoft doing similar things (java benchmarks, C compiler benchmarks)
- and everybody cheating on AI benchmarks (https://www.thestack.technology/ai-benchmarking-scandal-were...)
And despite it not being nice, some optimizations rely on type or function names schemas/substrings/etc
It sucks, but thats how it works
It doesnt have to be malicious just sometimes it is safer to deploy optimization only for your libs than risk breaking stuff
Or your frontend is not giving you more data which you can rely on
When you talk to the computer team about it they may offer a range of solutions, some of which may not be applicable to open source code. Picture proprietary #pragmas, intrinsics, or whatnot. What do you do? You can't ship a high performance library that doesn't deliver high performance. It is then when you rely on things like function names to enable specific code transformations that can't be used in general because they would sometimes break third party code.
I never worked on Cutlass, but this is the sort of thing that is done in the real world.
There is nothing nefarious about this sort of optimizatkon. People comparing this to cheating on benchmarks by rendering lower quality images are not on the right track.