This is amazing.. ive been working with custom CUDA kernels and https://crates.io/crates/cudarc for a long time, and this honestly looks like it could be a near drop-in replacement.
im especially curious how build times would compare? Most Rust CUDA crates obv rely on calling CMake or nvcc, which can make compilation painfully slow. coincidentally, just last week i was profiling build times and found that tools like sccache can dramatically reduce rebuild times by caching artifacts - but you still end up paying for expensive custom nvcc invocations (e.g. candle by hugging face calls custom nvcc command in their kernel compilation): https://arpadvoros.com/posts/2026/05/05/speeding-up-rust-whi...
This is a bit good for Rust if you want to use the language with CUDA. The problem is, it still doesn't really move the needle if you really don't like running closed source drivers and runtime binaries and care about open source.
Continuing from this discussion [0], this only makes it a Rust or a CUDA problem rather than a Python, CUDA and a PyTorch one if there bug in one of them.
Yet at the end of the day, it still uses Nvidia's closed source CUDA compiler 'nvcc' which they will never open source. A least Mojo promises to open source their own compiler which compiles to different accelerators with multiple backend support.
Small correction: cuda-oxide does not use `nvcc` to compile Rust kernels.
The device path is roughly: rustc frontend/MIR -> cuda-oxide’s Pliron-based IR/lowering pipeline -> LLVM IR -> PTX via LLVM’s NVPTX backend. Host code still goes through normal rustc/LLVM codegen.
You still need the NVIDIA driver/toolkit pieces to run CUDA code and load PTX on NVIDIA GPUs, so this does not change the broader CUDA ecosystem/dependency story. But it is not “Rust source handed to nvcc”; it is a Rust compiler backend generating device code.
I'm quite interested in how they dealt with Rust's memory model, which might not neatly map to CUDA's semantics. Curious what the differences are compared to CUDA C++, and if the Rust's type system can actually bring more safety to CUDA (I do think writing GPU kernels is inherently unsafe, it's just too hard to create a safe language because of how the hardware works, and because of the fact that you're hyper-optimizing all the time)
FWIW, Rust’s memory model is more or less completely identical to C++’s, by design. Atomics work the same, there’s provenance, and so on.
Whether it is a convenient language for GPU programming probably remains to be seen, but I definitely wouldn’t be surprised if you could make a decent DSL-like API for writing safe code that leverages the full spectrum of GPU oddities. That’s what CUDA is, right?
Why do we bother with programming languages today? Why not have the LLMs just write assembly code and skip the human readable part? We are not reviewing it anymore anyway.
Hell yea! I have been doing it with Cudarc (Kernels) and FFI (cuFFT). Using manual [de]serialization between byte arrays and rust data structs. I hope this makes it lower friction!
Does anyone know if this will let you share structs between host and device? That is the big thing missing so far with existing rust/CUDA workflows. (Plus the serialization/bytes barrier between them)
Yes, absolutely. That is one of the advantages of cuda-oxide being single-source Rust: the host and device code can refer to the same Rust types, and the compiler has enough information to make the device-side layout match what rustc chose on the host.
So the intended workflow is not “define a Rust struct on the host, define a matching CUDA C++ struct for the device, then serialize bytes between them.” It is much closer to “define `MyStruct` once in Rust, put a `DeviceBuffer<MyStruct>` on the GPU, and write kernels that take `&[MyStruct]`, `*const MyStruct`, etc.”
There are two important pieces under the hood:
1. At the kernel boundary, cuda-oxide scalarizes aggregate parameters where needed. For example, slices become pointer + length, and simple structs can be flattened into fields for launch ABI purposes.
2. For actual struct layout, we use rustc’s computed layout rather than assuming declaration order or a C ABI. That matters because Rust is allowed to reorder/pad `repr(Rust)` structs. The device lowering carries those offsets/padding through so field access on the GPU matches the host-side layout.
So for plain data structs, nested structs, numeric fields, arrays, etc., yes, this is very much the goal: share the type directly instead of maintaining a separate CUDA representation or crossing a bytes/serialization boundary.
The caveat is the usual one: this does not make arbitrary host-owned Rust heap graphs GPU-addressable. A `Vec`, `String`, `Box`, trait object, or host pointer still contains an address, and that address has to refer to memory the GPU can actually access. For those cases you still need device allocation, unified/HMM memory, or a GPU-friendly representation.
But for the common “I have a Rust data type and want kernels to consume/update arrays of it” case: yes, that is exactly the kind of friction cuda-oxide is meant to remove.
One thing I’ve been wary about with Rust for CUDA is the bit of overhead that Rust adds that is usually negligible but might matter here, like bounds checks on arrays. Could it cause additional registers to get used, lowering the concurrency of a kernel?
So, we have stainless, which means Linux code that never rusted. Now we need someone to make phosphorus so that we can turn rusty code into old iron. Then GPL fans can run Rust boxes, Stainless machines, or future proofed iron work horses.
All software can come on three editions. Stainless drivers that were never rusty, oxidized drivers that used Rust on existing code, and Iron editions which is where someone converted the Rust back to C using the new phosphoric tool...
Diversity can be our strength.
Making Iron C/c++ code can be called acid washing if it was rusted.
I wonder what it means for Slang[0]. Presumably the point is that people want to do GPU programming with a more modern language. But now you can just use Rust...
Writing shaders is materially different from writing CUDA kernels, at least for now. Shaders are simultaneously higher and lower level, and have a lot of idiosyncrasies as a result of being designed for a specific and limited set of driver/GPU features.
Stuff like descriptor sets, resource registers, dispatch limitations, …
Oh lord. If this is the trend, I probably can't avoid improving my Rust language knowledge in the long term. I hate reading Rust so much right now. I guess I just have to get over that hump.
> (em dash) no DSLs, no foreign language bindings, just Rust.
Official CUDA port and they couldn't even bother with the introductory paragraph.
Okay, I'll try to ignore it and read the docs. Hey a custom IR, this sounds interesti-
> MLIR’s implementation, however, is C++ with a side of TableGen, a build system that requires you to compile all of LLVM, and debugging sessions that make you question your career choices.
> A GPU kernel runs thousands of threads that all see the same memory at the same time. On a CPU, Rust prevents data races through ownership and borrowing – one mutable reference, no aliases, enforced at compile time. On a GPU, you have 2048 threads per SM, all launched from the same function, all pointing at the same output buffer. The borrow checker was not designed for this.
> cuda-oxide solves the problem in layers. The common case – one thread writes one element – is safe by construction, no unsafe required. The uncommon cases – shared memory, warp shuffles, hardware intrinsics – require unsafe with documented contracts. And the frontier cases – TMA, tensor cores, cluster-level communication – are fully manual, matching the complexity of the hardware they control.
That's.. not really Rusty. In Rust, we create new safe abstractions when the existing ones don't quite map to the problem at hand. See for example what's done in Rust for Linux
If it's not safe.. what's the point of Rust?
(it's okay to offer unsafe APIs for people that need to squeeze the last bit of performance, but this shouldn't be the baseline)
I compare this with userspace libs for APIs like io_uring and vulkan. designing safe APIs for them stuff is kind of hard (there's even some unsound attempts)
Weird. There's a recent NVIDIA MLIR that is quite good and fast. Or they could target the even easier and more recent/fashionable tile IR [1] used by CuTile [2] (a little bit higher level but significantly easier to target, only loses on epilogue fusion and similar).
27 comments
[ 218 ms ] story [ 2132 ms ] threadim especially curious how build times would compare? Most Rust CUDA crates obv rely on calling CMake or nvcc, which can make compilation painfully slow. coincidentally, just last week i was profiling build times and found that tools like sccache can dramatically reduce rebuild times by caching artifacts - but you still end up paying for expensive custom nvcc invocations (e.g. candle by hugging face calls custom nvcc command in their kernel compilation): https://arpadvoros.com/posts/2026/05/05/speeding-up-rust-whi...
Continuing from this discussion [0], this only makes it a Rust or a CUDA problem rather than a Python, CUDA and a PyTorch one if there bug in one of them.
Yet at the end of the day, it still uses Nvidia's closed source CUDA compiler 'nvcc' which they will never open source. A least Mojo promises to open source their own compiler which compiles to different accelerators with multiple backend support.
Unlike this...but uses Rust.
[0] https://news.ycombinator.com/item?id=48067228
The device path is roughly: rustc frontend/MIR -> cuda-oxide’s Pliron-based IR/lowering pipeline -> LLVM IR -> PTX via LLVM’s NVPTX backend. Host code still goes through normal rustc/LLVM codegen.
You still need the NVIDIA driver/toolkit pieces to run CUDA code and load PTX on NVIDIA GPUs, so this does not change the broader CUDA ecosystem/dependency story. But it is not “Rust source handed to nvcc”; it is a Rust compiler backend generating device code.
Whether it is a convenient language for GPU programming probably remains to be seen, but I definitely wouldn’t be surprised if you could make a decent DSL-like API for writing safe code that leverages the full spectrum of GPU oddities. That’s what CUDA is, right?
Does anyone have more details on NVIDIAs use of Spark/Ada?
All I can find is what's listed below:
https://www.adacore.com/case-studies/nvidia-adoption-of-spar...
So the intended workflow is not “define a Rust struct on the host, define a matching CUDA C++ struct for the device, then serialize bytes between them.” It is much closer to “define `MyStruct` once in Rust, put a `DeviceBuffer<MyStruct>` on the GPU, and write kernels that take `&[MyStruct]`, `*const MyStruct`, etc.”
There are two important pieces under the hood:
1. At the kernel boundary, cuda-oxide scalarizes aggregate parameters where needed. For example, slices become pointer + length, and simple structs can be flattened into fields for launch ABI purposes.
2. For actual struct layout, we use rustc’s computed layout rather than assuming declaration order or a C ABI. That matters because Rust is allowed to reorder/pad `repr(Rust)` structs. The device lowering carries those offsets/padding through so field access on the GPU matches the host-side layout.
So for plain data structs, nested structs, numeric fields, arrays, etc., yes, this is very much the goal: share the type directly instead of maintaining a separate CUDA representation or crossing a bytes/serialization boundary.
The caveat is the usual one: this does not make arbitrary host-owned Rust heap graphs GPU-addressable. A `Vec`, `String`, `Box`, trait object, or host pointer still contains an address, and that address has to refer to memory the GPU can actually access. For those cases you still need device allocation, unified/HMM memory, or a GPU-friendly representation.
But for the common “I have a Rust data type and want kernels to consume/update arrays of it” case: yes, that is exactly the kind of friction cuda-oxide is meant to remove.
All software can come on three editions. Stainless drivers that were never rusty, oxidized drivers that used Rust on existing code, and Iron editions which is where someone converted the Rust back to C using the new phosphoric tool...
Diversity can be our strength.
Making Iron C/c++ code can be called acid washing if it was rusted.
(Disclaimer: I like Slang a lot.)
[0]: https://shader-slang.org/
Stuff like descriptor sets, resource registers, dispatch limitations, …
Official CUDA port and they couldn't even bother with the introductory paragraph.
Okay, I'll try to ignore it and read the docs. Hey a custom IR, this sounds interesti-
> MLIR’s implementation, however, is C++ with a side of TableGen, a build system that requires you to compile all of LLVM, and debugging sessions that make you question your career choices.
I can't take this industry seriously anymore.
> A GPU kernel runs thousands of threads that all see the same memory at the same time. On a CPU, Rust prevents data races through ownership and borrowing – one mutable reference, no aliases, enforced at compile time. On a GPU, you have 2048 threads per SM, all launched from the same function, all pointing at the same output buffer. The borrow checker was not designed for this.
> cuda-oxide solves the problem in layers. The common case – one thread writes one element – is safe by construction, no unsafe required. The uncommon cases – shared memory, warp shuffles, hardware intrinsics – require unsafe with documented contracts. And the frontier cases – TMA, tensor cores, cluster-level communication – are fully manual, matching the complexity of the hardware they control.
That's.. not really Rusty. In Rust, we create new safe abstractions when the existing ones don't quite map to the problem at hand. See for example what's done in Rust for Linux
If it's not safe.. what's the point of Rust?
(it's okay to offer unsafe APIs for people that need to squeeze the last bit of performance, but this shouldn't be the baseline)
I compare this with userspace libs for APIs like io_uring and vulkan. designing safe APIs for them stuff is kind of hard (there's even some unsound attempts)
Weird. There's a recent NVIDIA MLIR that is quite good and fast. Or they could target the even easier and more recent/fashionable tile IR [1] used by CuTile [2] (a little bit higher level but significantly easier to target, only loses on epilogue fusion and similar).
[1] https://docs.nvidia.com/cuda/tile-ir/
[2] https://developer.nvidia.com/cuda/tile