TIL, I didn’t know that. I always assumed it came from level zero”, the Intel computer layer that ZLUDA was translating to before its developer was hired by AMD to target HIP.
RDNA1 isn't good for a whole lot, even flagship RDNA2 cards are a stretch for many things. The lack of WMMA/matrix multiply/BF16 is too severe of a penalty.
The FP16 throughput on RDNA1 is both shader reliant and requires everything to be packed first. Even with 2 or 4 or 1000 cards, you would be consuming all of the available memory and memory bandwidth just packing and unpacking values, and if you really want to dump a hundred billion tokens into making it work anyways, you're only going to find out that even if you bother to sit there ferrying packed values to ram or disk before then issuing the instructions, paying that already severe penalty again when the values then have to be unpacked is so steep of a cost that the 256 BF16 flops/cu/clock's effective throughput is outright lower than simply doing it on a Zen 2 processor. You also don't have INT8 (or really INT4) on RDNA1 so the other RNS/CRT tricks aren't viable.
Sadly RDNA1's VCN2 also lacks actually good x264 bframe encoding support, or even P010 for 10 bit color, so what I'm saying is you should sell them. Used Radeon VII's are like $260, you'll go a lot further with those especially if you throw in a 7900XTX, and then augment that further with a 9070 CRE (you only want it for its int8 cores), and of course 128GB of ram.
It's volatile as all get out and I'm not going to call out any token by name but if you needed another nudge, you can absolutely cover the cost of power and the 9070 GRE (or even XT) itself before the warranty expires by mining (without holding/speculating), which has been my breakeven point for sidestepping any guilt I might feel from buying another flagship GPU.
But to reason in other direction, unless you absolutely need cards right now, you could throw that ~$1440 in a 6 month CD and let the 4.5% pay for the tax or shipping on a 10090 XT or whatever RDNA 5 flagship when those drop in about as much time. If it lands anywhere close to what the rumors are indicating it should be a fucking monster.
I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.
I think it was on qwen 3.5 9B or thereabouts that I saw a ~30% decode rate improvement on my M2 Mac vs MLX. That’s probably the best I’ve gotten.
Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.
All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game.
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
Note that Vulkan is only somehow better than anything else out of Khronos, because at least for that API someone is paying LunarG for the Vulkan SDK, and NVidia has a vested interest in now pushing slang as the shading language.
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
Consider that for the others you are supposed to build your own SDK from random pieces of libraries, not all of them playing well together.
In that regard, the Vulkan SDK is definitely an improvement.
However using Vulkan on Android is actually hardly any different, given that there is no SDK as such, only the libs and headers get dumped into the NDK, there is a github with samples from Google and that's it.
So bad that they failed to convince anyone to drop OpenGL ES from Java/Kotlin, and use NDK alongside C or C++ for Vulkan, thus now there is WebGPU implementation for Java and Kotlin consumption.
I do agree with the Vulkan code itself, related extension spaghetti, and overall state of drivers, which has become visible enough to have been part of Vulkanised talks for two years in a row now.
It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
C ran on computers with 8/12/16/18/24/32/36/48/60 bit words, computers with 8/9-bit bytes, 6/9 bit characters, computers with word and byte addressing, with one's and two's compliment integers, with flat and non-flat memory models, with different endianness & different alignment rules.
Early graphics APIs ran on GPUs that were just about as different form one another as that.
We could easily figure it out, were it not for the trillion dollars incentives to not figure it out.
AI will take down Nvidia’s moat. When it becomes trivial to translate CUDA/PTX to HIP, SYCL or Metal, CUDA is no longer the moat, it becomes the intermediate representation.
I don't think we're at a point yet where anyone would trust ZLUDA enough to ship commercial products that rely on it. I would be delighted though, if anyone can prove me wrong.
Because while many want to work on the same project generally, the people who can or want to do that has so little time.
I personally had to put all my toy projects into hiatus to be able to keep up with life. Maybe if things get a little lighter, we can thaw them or start working on a couple of interesting ones which I stumble on.
i swear people who are outsiders here have only clickbait takes; if you've never had to ship GPU code professionally you should just not comment on these things.
the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.
Wait, I have an idea:
What is your qualification for talking about graphics programming? If you cannot provide credentials, I will have no other choice than to claim you don’t know what you’re talking about, based on the criteria by which you judged my comments so far.
But claude will not be able to debug the whole stack? I have no experience in the domain. But as an outsider, it seems to me, that if the rate of progress is maintained, i don't see why agents wouldn't be able to do it in a couple of years
If your stack is open source, why not? I’ve had good results with pointing an LLM to a debug build of the GPU compiler when I saw it crash. It just seems you need to periodically remind them that gdb exists to prevent it from littering your drivers with printfs.
yeah yeah, "when" an often keyword with AI it seems. As Mr. E. Nigma put it - what always comes but never arrives? Meanwhile the moat deepens and it's build on inertia and laziness and Nvidia knows this really REALLY well.
Oh, absolutely. Nvidia is the modern day “nobody gets fired for buying IBM”. The reason we’re still using Unix is not because it’s the best, but because it had to much inertia to let any alternative become its successor. Similarly, C and HTML are maybe the most terrible yet extremely useful languages we have.
trivial to translate (or transpile) - okay. trivial to understand the result - not so much. trivial to then evolve it - hm... perhaps a different story. still, it seems very likely now, that such "quick rewrites" are viable, not sure if an open approach to them is viable. a newly born open project that was LLM-derived, and not by a credible author, which spans hundreds of files no human eye has ever looked at, can only work for a closed organization, but will never be trusted by the general audience... just like that.
I don't see a hard reason. If it works it works. No hard need for a good, universal, and long lasting solution. At some point you just stack slop on top of slop and it works for your use case - and if it doesn't you'll slop it out yourself.
tbh... it seems to work contrary to all expectation that slop shouldn't work. but, then, just when I think about the incredible corporate slop i've seen here and there, that not only works, but pays. well I guess we have to agree - slop works. and that's fine.
man i can't say how much i used to like cuda when i had a nvidia gpu it made ml so much fun and on amd its a war especially on rdna 2 cards which i have. hopefully one day we will be able to properly translate cuda for its amd counter parts
Followed by (from NVidia) compilers for C, C++, Fortran, Python JIT and more recently Julia JIT.
Additionally there are third party support for .NET, Java, Haskell tooling to also target CUDA.
Then you have the libraries ready to plug into the official NVidia compilers to reduce the need to explicitly write CUDA code manually.
Finally you have the graphical debugging and profiling tools that make targeting GPUs as easy as regular CPU programming, ignoring the hardware differences for a moment.
All wannabe replacements fail, because they focus only on the CUDA C++ compiler and nothing else.
Right, it reminds me of this website back in the day, when people really were coding these things as well instead of just generating 100% of it. Being sarcastic here, but stuff like "How I coded Java with Rust" were a common thing hah!
I think projects like this one are a clear indication that CUDA *is* the de facto standard and that aspiring open standards are competing for their niche. The rest of the world is still living in the CUDA ecosystem. Asking accelerated HPC devs to port every project to a different language (and possibly maintain two codebases) is something we haven't asked of anyone else for decades now. And yes, LLMs can help (although CUDA makes up the overwhelming majority of their training data, with the hilarious result that you need to give them a disambiguation table like "you'll want to write $CUDA, but write $else"), but AI is clearly not the correct tool for the job. A compiler is.
ZLUDA is JIT (it's working with the binary, not the source, which usually means you've lost some information) and it doesn't support CDNA, which is a showstopper for enterprises, tho it's a great project and it's awesome seeing it work on Windows (I wonder whether MS Defender complains about it due to library injection? IIRC, it used to. That'd be another enterprise showstopper)
Someone here mentioned scale-lang.com already as a portability tool - it also beats HIP in the performance game, on average. scale-lang.com/s/discord is where its community lives.
disclaimer: I am part of SCALE's team, Spectral Compute
81 comments
[ 122 ms ] story [ 283 ms ] threadThe FP16 throughput on RDNA1 is both shader reliant and requires everything to be packed first. Even with 2 or 4 or 1000 cards, you would be consuming all of the available memory and memory bandwidth just packing and unpacking values, and if you really want to dump a hundred billion tokens into making it work anyways, you're only going to find out that even if you bother to sit there ferrying packed values to ram or disk before then issuing the instructions, paying that already severe penalty again when the values then have to be unpacked is so steep of a cost that the 256 BF16 flops/cu/clock's effective throughput is outright lower than simply doing it on a Zen 2 processor. You also don't have INT8 (or really INT4) on RDNA1 so the other RNS/CRT tricks aren't viable.
Sadly RDNA1's VCN2 also lacks actually good x264 bframe encoding support, or even P010 for 10 bit color, so what I'm saying is you should sell them. Used Radeon VII's are like $260, you'll go a lot further with those especially if you throw in a 7900XTX, and then augment that further with a 9070 CRE (you only want it for its int8 cores), and of course 128GB of ram.
But to reason in other direction, unless you absolutely need cards right now, you could throw that ~$1440 in a 6 month CD and let the 4.5% pay for the tax or shipping on a 10090 XT or whatever RDNA 5 flagship when those drop in about as much time. If it lands anywhere close to what the rumors are indicating it should be a fucking monster.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
I would really appreciate your input!
Can you tell us what kind of gain you got ie. what was the speed bump
Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable __VK_ENABLE_DEVICE_GROUPS=1.
* manual pipelining
* shared memory staging
* tiling
* bounds checking
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
See https://www.bit-tech.net/news/gaming/pc/carmack-directx-bett...
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
In that regard, the Vulkan SDK is definitely an improvement.
However using Vulkan on Android is actually hardly any different, given that there is no SDK as such, only the libs and headers get dumped into the NDK, there is a github with samples from Google and that's it.
So bad that they failed to convince anyone to drop OpenGL ES from Java/Kotlin, and use NDK alongside C or C++ for Vulkan, thus now there is WebGPU implementation for Java and Kotlin consumption.
I do agree with the Vulkan code itself, related extension spaghetti, and overall state of drivers, which has become visible enough to have been part of Vulkanised talks for two years in a row now.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
Early graphics APIs ran on GPUs that were just about as different form one another as that.
We could easily figure it out, were it not for the trillion dollars incentives to not figure it out.
Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.
ZLUDA is already doing that, no?
Src: https://github.com/vosen/ZLUDA
It's niche, performance and code-quality sensitive, needs deep and wide knowledge, and hardware dependent.
I personally had to put all my toy projects into hiatus to be able to keep up with life. Maybe if things get a little lighter, we can thaw them or start working on a couple of interesting ones which I stumble on.
the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.
https://github.com/Zaneham/Booth
https://scale-lang.com/
AMD GPUs build for AI specs for reference: https://flopper.io/gpus?vendor=AMD&page=1
- what exactly is CUDA?
- is it patented hardware tech that AMD cannot replicate?
- is it a library that ll also work on AMD while it currently works on nvidia
- can AMD cook an equivalent of CUDA, if so why havent they done it?
At the botton you the PTX bytecode.
Followed by (from NVidia) compilers for C, C++, Fortran, Python JIT and more recently Julia JIT.
Additionally there are third party support for .NET, Java, Haskell tooling to also target CUDA.
Then you have the libraries ready to plug into the official NVidia compilers to reduce the need to explicitly write CUDA code manually.
Finally you have the graphical debugging and profiling tools that make targeting GPUs as easy as regular CPU programming, ignoring the hardware differences for a moment.
All wannabe replacements fail, because they focus only on the CUDA C++ compiler and nothing else.
This gives me strong "Bash on Ubuntu on Windows" kind of vibes
ZLUDA is JIT (it's working with the binary, not the source, which usually means you've lost some information) and it doesn't support CDNA, which is a showstopper for enterprises, tho it's a great project and it's awesome seeing it work on Windows (I wonder whether MS Defender complains about it due to library injection? IIRC, it used to. That'd be another enterprise showstopper)
Someone here mentioned scale-lang.com already as a portability tool - it also beats HIP in the performance game, on average. scale-lang.com/s/discord is where its community lives.
disclaimer: I am part of SCALE's team, Spectral Compute