However, for my use cases (running on arbitrary client hardware) I generally distrust any abstractions over the GPU api, as the entire point is to leverage the low level details of the gpu. Treating those details as a nuisance leads to bugs and performance loss, because each target is meaningfully different.
To overcome this, a similar system should be brought forward by the vendors. However, since they failed to settle their arguments, I imagine the platform differences are significant. There are exceptions to this (e.g Angle), but they only arrive at stability by limiting the feature set (and so performance).
Its good that this approach at least allows conditional compilation, that helps for sure.
What we really need is a consistent GPU ISA. If it wasn't for the fairly recent proliferation of ARM CPUs, we more or less would've rallied around x86 as the de facto ISA for general purpose compute. I'm not sure why we couldn't do the same for GPUs as well.
> Though this demo doesn't do so, multiple backends could be compiled into a single binary and platform-specific code paths could then be selected at runtime.
That’s kind of the goal, I’d assume: writing generic code and having it run on anything.
Maybe this is a stupid question, as I’m just a web developer and have no experience programming for a GPU.
Doesn’t WebGPU solve this entire problem by having a single API that’s compatible with every GPU backend? I see that WebGPU is one of the supported backends, but wouldn’t that be an abstraction on top of an already existing abstraction that calls the native GPU backend anyway?
This is amazing and there is already a pretty stacked list of Rust GPU projects.
This seems to be at an even lower level of abstraction than burn[0] which is lower than candle[1].
I gueds whats left is to add backend(s) that leverage naga and others to the above projects? Feeks like everyone is building on different bases here, though I know the naga work is relatively new.
[EDIT] Just to note, burn is the one that focuses most on platform support but it looks like the only backend that uses naga is wgpu... So just use wgpu and it's fine?
2. Backend abstracting over the cust, ash and wgpu crates
3. wgpu and co. abstracting over platforms, drivers and APIs
4. Vulkan, OpenGL, DX12 and Metal abstracting over platforms and drivers
5. Drivers abstracting over vendor specific hardware (one could argue there are more layers in here)
6. Hardware
That's a lot of hidden complexity, better hope one never needs to look under the lid. It's also questionable how well performance relevant platform specifics survive all these layers.
I write native audio apps, where every cycle matters. I also need the full compute API instead of graphics shaders.
Is the "Rust -> WebGPU -> SPIR-V -> MSL -> Metal" pipeline robust when it come to performance? To me, it seems brittle and hard to reason about all these translation stages. Ditto for "... -> Vulkan -> MoltenVk -> ...".
Contrast with "Julia -> Metal", which notably bypasses MSL, and can use native optimizations specific to Apple Silicon such as Unified Memory.
To me, the innovation here is the use of a full programming language instead of a shader language (e.g. Slang). Rust supports newtype, traits, macros, and so on.
They are doing a huge service for developers that just want to build stuff and not get into the platform wars.
https://github.com/cogentcore/webgpu is a great example . I code in golang and just need stuff to work on everything and this gets it done, so I can use the GPU on everything.
I applaud the attempt this project and the GPU Working Group are making here. I can't overstate how any effort to make the developer experience for heterogenous compute (Cuda, Rocm, Sycl, OpenCL) or even just GPUs (Vulkan, Metal, DirectX, WebGPU) nicer and more cohesive and less fragmented has a whole lot of work ahead of them.
This is a little crude still, but the fact that this is even possible is mind blowing. This has the potential, if progress continues, to break the vendor-locked nightmare that is GPU software and open up the space to real competition between hardware vendors.
Imagine a world where machine learning models are written in Rust and can run on both Nvidia and AMD.
To get max performance you likely have to break the abstraction and write some vendor-specific code for each, but that's an optimization problem. You still have a portable kernel that runs cross platform.
Do you really need to break the abstraction? Current scenario where SPIR-V is let's say compiled by Mesa into NIR and then NIR is compiled into GPU specific machine code works pretty well, where optimizations can happen on different phases of compilation.
Very interesting. I wonder about the model of storing the GPU IR in binary for a real-world project; it seems like that could bloat the binary size a lot.
I also wonder about the performance of just compiling for a target GPU AOT. These GPUs can be very different even if they come from the same vendor. This seems like it would compile to the lowest common denominator for each vendor, leaving performance on the table. For example, Nvidia H-100s and Nvidia Blackwell GPUs are different beasts, with specialised intrinsics that are not shared, and to generate a PTX that would work on both would require not using specialised features in one or both of these GPUs.
Mojo solves these problems by JIT compiling GPU kernels at the point where they're launched.
If you're like me you want to write code that runs on the GPU but you don't, because everything about programming GPUs is pain. That's the use-case I see for rust-gpu. Make CPU devs into GPU devs with an acceptable performance penalty. If you're already a GPU programmer and know cuda/vulkan/metal/dx inside out then something like this will not be interesting to you.
25 comments
[ 2.7 ms ] story [ 48.5 ms ] threadWow. That at first glance seems to unlock ALOT of interesting ideas.
However, for my use cases (running on arbitrary client hardware) I generally distrust any abstractions over the GPU api, as the entire point is to leverage the low level details of the gpu. Treating those details as a nuisance leads to bugs and performance loss, because each target is meaningfully different.
To overcome this, a similar system should be brought forward by the vendors. However, since they failed to settle their arguments, I imagine the platform differences are significant. There are exceptions to this (e.g Angle), but they only arrive at stability by limiting the feature set (and so performance).
Its good that this approach at least allows conditional compilation, that helps for sure.
(And I haven't tried the SPIR-V compilation yet, just came across it yesterday.)
That’s kind of the goal, I’d assume: writing generic code and having it run on anything.
Doesn’t WebGPU solve this entire problem by having a single API that’s compatible with every GPU backend? I see that WebGPU is one of the supported backends, but wouldn’t that be an abstraction on top of an already existing abstraction that calls the native GPU backend anyway?
This seems to be at an even lower level of abstraction than burn[0] which is lower than candle[1].
I gueds whats left is to add backend(s) that leverage naga and others to the above projects? Feeks like everyone is building on different bases here, though I know the naga work is relatively new.
[EDIT] Just to note, burn is the one that focuses most on platform support but it looks like the only backend that uses naga is wgpu... So just use wgpu and it's fine?
Yeah basically wgpu/ash (vulkan, metal) or cuda
[EDIT2] Another crate closer to this effort:
https://github.com/tracel-ai/cubecl
[0]: https://github.com/tracel-ai/burn
[1]: https://github.com/huggingface/candle/
1. Domain specific Rust code
2. Backend abstracting over the cust, ash and wgpu crates
3. wgpu and co. abstracting over platforms, drivers and APIs
4. Vulkan, OpenGL, DX12 and Metal abstracting over platforms and drivers
5. Drivers abstracting over vendor specific hardware (one could argue there are more layers in here)
6. Hardware
That's a lot of hidden complexity, better hope one never needs to look under the lid. It's also questionable how well performance relevant platform specifics survive all these layers.
Is the "Rust -> WebGPU -> SPIR-V -> MSL -> Metal" pipeline robust when it come to performance? To me, it seems brittle and hard to reason about all these translation stages. Ditto for "... -> Vulkan -> MoltenVk -> ...".
Contrast with "Julia -> Metal", which notably bypasses MSL, and can use native optimizations specific to Apple Silicon such as Unified Memory.
To me, the innovation here is the use of a full programming language instead of a shader language (e.g. Slang). Rust supports newtype, traits, macros, and so on.
They are doing a huge service for developers that just want to build stuff and not get into the platform wars.
https://github.com/cogentcore/webgpu is a great example . I code in golang and just need stuff to work on everything and this gets it done, so I can use the GPU on everything.
Thank you rust !!
I think GPU programming is different enough to require special care. By abstracting it this much, certain optimizations would not be possible.
Imagine a world where machine learning models are written in Rust and can run on both Nvidia and AMD.
To get max performance you likely have to break the abstraction and write some vendor-specific code for each, but that's an optimization problem. You still have a portable kernel that runs cross platform.
The implications of this for inference is going to be huge.
I also wonder about the performance of just compiling for a target GPU AOT. These GPUs can be very different even if they come from the same vendor. This seems like it would compile to the lowest common denominator for each vendor, leaving performance on the table. For example, Nvidia H-100s and Nvidia Blackwell GPUs are different beasts, with specialised intrinsics that are not shared, and to generate a PTX that would work on both would require not using specialised features in one or both of these GPUs.
Mojo solves these problems by JIT compiling GPU kernels at the point where they're launched.