I feel like the title is a bit misleading. I think it should be something like "Using Rust's Standard Library from the GPU". The stdlib code doesn't execute on the GPU, it is just a remote function call, executed on the CPU, and then the response is returned. Very neat, but not the same as executing on the GPU itself as the title implies.
Are there any details around how the round-trip and exchange of data (CPU<->GPU) is implemented in order to not be a big (partially-hidden) performance hit?
e.g. this code seems like it would entirely run on the CPU?
print!("Enter your name: ");
let _ = std::io::stdout().flush();
let mut name = String::new();
std::io::stdin().read_line(&mut name).unwrap();
But what if we concatenated a number to the string that was calculated on the GPU or if we take a number:
print!("Enter a number: ");
[...] // string number has to be converted to a float and sent to the GPU
// Some calculations with that number performed on the GPU
print!("The result is: " + &the_result.to_string()); // Number needs to be sent back to the CPU
Or maybe I am misunderstanding how this is supposed to work?
I'm confused about this: As the article outlines well, Std Rust (over core) buys you GPOS-provided things. For example:
- file system
- network interfaces
- dates/times
- Threads, e.g. for splitting across CPU cores
The main relevant one I can think which applies is an allocator.
I do a lot of GPU work with rust: Graphics in WGPU, and Cuda kernels + cuFFT mediated by Cudarc (A thin FFI lib). I guess, running Std lib on GPU isn't something I understand. What would be cool is the dream that's been building for decades about parallel computing abstractions where you write what looks like normal single-threaded CPU code, but it automagically works on SIMD instructions or GPU. I think this and CubeCL may be working towards that? (I'm using Burn as well on GPU, but that's abstracted over)
Of note: Rayon sort of is that dream for CPU thread pools!
I work with GPUs and I'm also trying to understand the motivations here.
Side note & a hot take: that sort of abstraction never really existed for GPU and it's going to be even harder now as Nvidia et al races to put more & more specialized hardware bits inside GPUs
>What would be cool is the dream that's been building for decades about parallel computing abstractions where you write what looks like normal single-threaded CPU code, but it automagically works on SIMD instructions or GPU.
I've had that same dream at various points over the years, and prior to AI my conclusion was that it was untenable barring a very large, world-class engineering team with truckloads of money.
I'm guessing a much smaller (but obviously still world-class!) team now has a shot at it, and if that is indeed what they're going for, then I could understand them perhaps being a bit coy.
It's one heck of a crazy hard problem to tackle. It really depends on what levels of abstraction are targeted, in addition to how much one cares about existing languages and supporting infra.
It's really nice to see a Rust-only shop, though.
Edit: Turns out it helps to RTFA in its entirety:
>>Our approach differs in two key ways. First, we target Rust's std directly rather than introducing a new GPU-specific API surface. This preserves source compatibility with existing Rust code and libraries. Second, we treat host mediation as an implementation detail behind std, not as a visible programming model.
In that sense, this work is less about inventing a new GPU runtime and more about extending Rust's existing abstraction boundary to span heterogeneous systems.
That last sentence is interesting in combination with this:
>>Technologies such as NVIDIA's GPUDirect Storage, GPUDirect RDMA, and ConnectX make it possible for GPUs to interact with disks and networks more directly in the datacenter.
Perhaps their modified std could enable distributed compute just by virtue of running on the GPU, so long as the GPU hardware topology supports it.
Exciting times if some of the hardware and software infra largely intended for disaggregated inference ends up as a runtime for [compiled] code originally intended for the CPU.
To the author (or anyone from vectorware team), can you please give me, admittedly a skeptic, a motivating example of a "GPU-native" application?
That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU?
From the "Announcing VectorWare" page:
> Even after opting in, the CPU is in control and orchestrates work on the GPU.
Isn't it better to let CPUs be in control and orchestrate things as GPUs have much smaller, dumber cores?
> Furthermore, if you look at the software kernels that run on the GPU they are simplistic with low cyclomatic complexity.
Again, there's a obvious reason why people don't put branch-y code on GPU.
I think it is possible to run CPU code on GPU (including the whole OS), because GPU has registers, memory, arithmetic and branch instructions, and that should be enough. However, it will be able to use only several cores from many thousands because GPU cores are effectively wide SIMD cores, grouped into the clusters, and CPU-style code would use only single SIMD lane. Am I wrong?
Merely mislead by marketing. The x64 arch has 512bit registers and a hundred or so cores. The gpu arch has 1024bit registers and a few hundred SMs or CUs, being the thing equivalent to an x64 core.
The software stacks running on them are very different but the silicon has been converging for years.
GPUs aren't fast because they run standard CPU code with magic pixie dust, they're fast because they're specialised vector processors running specialised vector code.
Cuda can also do C++ new, delete and virtual functions and exception handling and all the rest. And if you use that stuff, you're basically making an aeroplane flap its wings, with all the performance implications that come with such an abomination.
inb4 these guys start running Python and Ruby on GPU for "speed", and while they're at it, they should send a fax to Intel and AMD saying "Hey guys, why do you keep forgetting to put the magic go-fast pixie dust into your CPUs, are you stupid?"
Considering that we live in the age of megakernels where the cost of CPU->GPU->CPU data transfer and kernel launch overhead are becoming ever bigger performance bottlenecks I would have expected more enthusiasm in this comment section.
Surely there is some value in the ability to test your code on the CPU for logic bugs with printf/logging, easy breakpoints, etc and then run it on the GPU for speed? [0]
Surely there is some value in being able to manage KV caches and perform continuous batching, prefix caching, etc, directly on the GPU through GPU side memory allocations?
Surely there is some value in being able to send out just the newly generated tokens from the GPU kernel via a quick network call instead of waiting for all sessions in the current batch to finish generating their tokens?
Surely there is some value in being able to load model parameters from the file system directly into the GPU?
You could argue that I am too optimistic, but seemingly everyone here is stuck on the idea of running existing CPU code without ever even attempting to optimize the bottlenecks rather than having GPU heavy code interspersed with less GPU heavy code. It's all or nothing to you guys.
[0] Assuming that people won't write GPU optimized code at all is bad faith because the argument I am presenting here is that you test your GPU-first code on the CPU rather than pushing CPU-first code on the GPU.
What’s the latency on a hostcall? A PCIe round-trip for something like File::open is fine—that’s slow I/O anyway. But if a println! from GPU code blocks on a host round-trip every time, that completely changes how you’d use it for debugging.
Is there device-side buffering, or does each write actually wait for the host?
There's indeed an actual toggle at the top to switch on pedantic mode, FYI.
But still a bit odd that the article doesn't show assembly/cuda/opencl output of the compiler - nor show an example of anything parallel - like maybe a vector search, Mandelbrot calculation or something like that?
Something like this 2019 article on cuda for Julia:
I've been building something similar (GPU-native OS research project) and wanted to share a mental model shift that unlocked things for me.
The question "why run CPU code on GPU when GPU cores are slower?" assumes you're running ONE program. But GPUs execute in SIMD wavefronts of 32 threads - and here's the trick: each of those 32 lanes can run a DIFFERENT process. Same instruction, different data. Calculator on lane 0, text editor on lane 1, file indexer on lane 2. No divergence, legal SIMD, full utilization.
Suddenly you're not running "slow CPU code on GPU" - you're running 32 independent programs in parallel on hardware designed for exactly this pattern.
The win isn't throughput for compute-heavy code. It's eliminating CPU roundtrips for interactive stuff. Every kernel launch, every synchronization, every "GPU done, back to CPU, dispatch next thing" adds latency. A persistent kernel that polls for input, updates state, and renders - all without returning to CPU - changes the responsiveness equation entirely.
A few things to try at home if you're curious:
1. Write a Metal/CUDA kernel with while(true) and an atomic shutdown flag. See how long it runs.
(Spoiler: indefinitely, if you do it right)
2. Put 32 different "process states" in a buffer and have each SIMD lane execute instructions for its
own process. Watch all 32 make progress simultaneously.
3. Measure the latency from "input event" to "pixel on screen" with CPU orchestration vs GPU polling an input queue directly. The difference surprised me.
The persistent kernel thing has a nasty gotcha though - ALL 32 threads must participate in the while loop. If you do if (tid != 0) return; then while(true), it'll work for a few million iterations then hard-lock. Ask me how I know.
If you're running vastly different processes in different ALU lanes, the single master "program" that comprises them all is effectively an interpreter. And then it's hard to have the exact same control flow lead to vastly different effects in different processes, especially once you account for branches. This works well for inference batches since those are essentially about straight-line processing, but not much else.
It'll go much faster if you give each process a warp instead of a thread. That means each process has its own IP and set of vector registers, and when your editor takes a different branch to your browser, no cost.
20 comments
[ 3.0 ms ] story [ 40.3 ms ] threadUPDATE: Oh, that's a post from maintainers or rust-gpu.
e.g. this code seems like it would entirely run on the CPU?
But what if we concatenated a number to the string that was calculated on the GPU or if we take a number: Or maybe I am misunderstanding how this is supposed to work?I do a lot of GPU work with rust: Graphics in WGPU, and Cuda kernels + cuFFT mediated by Cudarc (A thin FFI lib). I guess, running Std lib on GPU isn't something I understand. What would be cool is the dream that's been building for decades about parallel computing abstractions where you write what looks like normal single-threaded CPU code, but it automagically works on SIMD instructions or GPU. I think this and CubeCL may be working towards that? (I'm using Burn as well on GPU, but that's abstracted over)
Of note: Rayon sort of is that dream for CPU thread pools!
Side note & a hot take: that sort of abstraction never really existed for GPU and it's going to be even harder now as Nvidia et al races to put more & more specialized hardware bits inside GPUs
I've had that same dream at various points over the years, and prior to AI my conclusion was that it was untenable barring a very large, world-class engineering team with truckloads of money.
I'm guessing a much smaller (but obviously still world-class!) team now has a shot at it, and if that is indeed what they're going for, then I could understand them perhaps being a bit coy.
It's one heck of a crazy hard problem to tackle. It really depends on what levels of abstraction are targeted, in addition to how much one cares about existing languages and supporting infra.
It's really nice to see a Rust-only shop, though.
Edit: Turns out it helps to RTFA in its entirety:
>>Our approach differs in two key ways. First, we target Rust's std directly rather than introducing a new GPU-specific API surface. This preserves source compatibility with existing Rust code and libraries. Second, we treat host mediation as an implementation detail behind std, not as a visible programming model.
In that sense, this work is less about inventing a new GPU runtime and more about extending Rust's existing abstraction boundary to span heterogeneous systems.
That last sentence is interesting in combination with this:
>>Technologies such as NVIDIA's GPUDirect Storage, GPUDirect RDMA, and ConnectX make it possible for GPUs to interact with disks and networks more directly in the datacenter.
Perhaps their modified std could enable distributed compute just by virtue of running on the GPU, so long as the GPU hardware topology supports it.
Exciting times if some of the hardware and software infra largely intended for disaggregated inference ends up as a runtime for [compiled] code originally intended for the CPU.
That is, where does it truly make a difference to dispatch non-parallel/syscalls etc from GPU to CPU instead of dispatching parallel part of a code from CPU to GPU?
From the "Announcing VectorWare" page:
> Even after opting in, the CPU is in control and orchestrates work on the GPU.
Isn't it better to let CPUs be in control and orchestrate things as GPUs have much smaller, dumber cores?
> Furthermore, if you look at the software kernels that run on the GPU they are simplistic with low cyclomatic complexity.
Again, there's a obvious reason why people don't put branch-y code on GPU.
Genuinely curious what I'm missing.
The software stacks running on them are very different but the silicon has been converging for years.
Cuda can also do C++ new, delete and virtual functions and exception handling and all the rest. And if you use that stuff, you're basically making an aeroplane flap its wings, with all the performance implications that come with such an abomination.
inb4 these guys start running Python and Ruby on GPU for "speed", and while they're at it, they should send a fax to Intel and AMD saying "Hey guys, why do you keep forgetting to put the magic go-fast pixie dust into your CPUs, are you stupid?"
Surely there is some value in the ability to test your code on the CPU for logic bugs with printf/logging, easy breakpoints, etc and then run it on the GPU for speed? [0]
Surely there is some value in being able to manage KV caches and perform continuous batching, prefix caching, etc, directly on the GPU through GPU side memory allocations?
Surely there is some value in being able to send out just the newly generated tokens from the GPU kernel via a quick network call instead of waiting for all sessions in the current batch to finish generating their tokens?
Surely there is some value in being able to load model parameters from the file system directly into the GPU?
You could argue that I am too optimistic, but seemingly everyone here is stuck on the idea of running existing CPU code without ever even attempting to optimize the bottlenecks rather than having GPU heavy code interspersed with less GPU heavy code. It's all or nothing to you guys.
[0] Assuming that people won't write GPU optimized code at all is bad faith because the argument I am presenting here is that you test your GPU-first code on the CPU rather than pushing CPU-first code on the GPU.
Is there device-side buffering, or does each write actually wait for the host?
But still a bit odd that the article doesn't show assembly/cuda/opencl output of the compiler - nor show an example of anything parallel - like maybe a vector search, Mandelbrot calculation or something like that?
Something like this 2019 article on cuda for Julia:
https://nextjournal.com/sdanisch/julia-gpu-programming
The question "why run CPU code on GPU when GPU cores are slower?" assumes you're running ONE program. But GPUs execute in SIMD wavefronts of 32 threads - and here's the trick: each of those 32 lanes can run a DIFFERENT process. Same instruction, different data. Calculator on lane 0, text editor on lane 1, file indexer on lane 2. No divergence, legal SIMD, full utilization. Suddenly you're not running "slow CPU code on GPU" - you're running 32 independent programs in parallel on hardware designed for exactly this pattern.
The win isn't throughput for compute-heavy code. It's eliminating CPU roundtrips for interactive stuff. Every kernel launch, every synchronization, every "GPU done, back to CPU, dispatch next thing" adds latency. A persistent kernel that polls for input, updates state, and renders - all without returning to CPU - changes the responsiveness equation entirely.
1. Write a Metal/CUDA kernel with while(true) and an atomic shutdown flag. See how long it runs. (Spoiler: indefinitely, if you do it right)2. Put 32 different "process states" in a buffer and have each SIMD lane execute instructions for its own process. Watch all 32 make progress simultaneously.
3. Measure the latency from "input event" to "pixel on screen" with CPU orchestration vs GPU polling an input queue directly. The difference surprised me.
The persistent kernel thing has a nasty gotcha though - ALL 32 threads must participate in the while loop. If you do if (tid != 0) return; then while(true), it'll work for a few million iterations then hard-lock. Ask me how I know.