Show HN: Attaching to a virtual GPU over TCP (thundercompute.com)
We developed a tool to trick your computer into thinking it’s attached to a GPU which actually sits across a network. This allows you to switch the number or type of GPUs you’re using with a single command.
116 comments
[ 3.8 ms ] story [ 262 ms ] threadHmm... well I just watched you run nvidia-smi in a Mac terminal, which is a platform it's explicitly not supported on. My instant assumption is that your tool copies my code into a private server instance and communicates back and forth to run the commands.
Does this platform expose eGPU capabilities if my host machine supports it? Can I run raster workloads or network it with my own CUDA hardware? The actual way your tool and service connects isn't very clear to me and I assume other developers will be confused too.
Going into more details for how this works, we intercept communication between the CPU and the GPU so only GPU code and commands are sent across the network to a GPU that we are hosting. This way we are able to virtualize a remote GPU and make your computer think it's directly attached to that GPU.
We are not copying your CPU code and running it on our machines. The CPU code runs entirely on your instance (meaning no files need to be copied over or packages installed on the GPU machine). One of the benefits of this approach is that you can easily scale to a more / less powerful GPU without needing to setup a new server.
will that cause system instability, say, if the network suddenly dropped?
Given that, if the network suddenly dropped then only the process using the GPU would fail.
Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.
As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!
https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork...
Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?
I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.
this is awesome. can it do 3d rendering (vulkan/opengl)
> is this a remote nvapi
Essentially yes! Just to be clear, this covers the entire GPU not just the NVAPI (i.e. all of cuda). This functions like you have the physical card directly plugged into the machine.
Right now we don't support vulkan or opengl since we're mostly focusing on AI workloads, however we plan to support these in the future (especially if there is interest!)
I bet you saw this https://github.com/mikex86/LibreCuda
they implemented the cuda driver by calling into rmapi.
My understanding is if there is a remote rmapi, other user mode drivers should work out of the box?
I think essentially this is solving the same problem Ray (https://www.ray.io/) is solving, but in a more generic way.
it potentially can have finer grained gpu sharing, like a half-gpu.
I'm very excited about this.
1. If you're actively developing and need a GPU then you typically would be paying the entire time the instance is running. Using Thunder means you only pay for the GPU while actively using it. Essentially, if you are running CPU only code you would not be paying for any GPU time. The alterative for this is to manually turn the instance on and off which can be annoying.
2. This allows you to easily scale the type and number of GPUs you're using. For example, say you want to do development on a cheap T4 instance and run a full DL training job on a set of 8 A100. Instead of needing to swap instances and setup everything again, you can just run a command and then start running on the more powerful GPUs.
> 1. If you're actively developing and need a GPU [for fractional amounts of time]...
Why would I need a GPU for a short amount of time during development? For testing?
I don't get it - what would testing an H100 over a TCP connection tell me? It's like, yeah, I can do that, but it doesn't represent an environment I am going to use for real. Nobody runs applications to GPUs on buses virtualized over TCP connections, so what exactly would I be validating?
From the developer perspective, you wouldn't know that the H100 is across a network. The experience will be as if your computer is directly attached to an H100. The benefit here is that if you're not actively using the H100 (such as when you're setting up the instance or after the training job completes) you are not paying for the H100.
I would never run a training job on a GPU virtualized over TCP connection. I would never run a training job that requires 80GB of VRAM on a 24GB VRAM device.
Whom is this for? Who needs to save kopecks on a single GPU who needs H100s?
So I wrote https://github.com/steelbrain/ffmpeg-over-ip and had the server running in the windows machine and the client in the media server (could be plex, emby, jellyfin etc) and it worked flawlessly.
As an aside, are there any uses for GPU-over-network other than video encoding? The increased latency seems like it would prohibit anything machine learning related or graphics intensive.
I found Juice to work decently for graphical applications too (e.g., games, CAD software). Latency was about what you'd expect for video encode + decode + network: 5-20ms on a LAN if I recall correctly.
[1] - https://github.com/Juice-Labs/Juice-Labs
IPMI and such could use it. Like, for example, Proxmox could use it. Machine learning tasks (like Frigate) and hashcat could also use such. All in theory, of course. Many tasks use VNC right now, or SPICE. The ability to extract your GPU in the Unix way over TCP/IP is powerful. Though Node.js would not be the way I'd want such to go.
https://gist.github.com/tzmartin/88abb7ef63e41e27c2ec9a5ce5d...
https://news.ycombinator.com/showhn.html
https://news.ycombinator.com/item?id=22336638
One of our main goals for the near future is to allow GPU sharing. This would be better than MIG or vGPU since we'd allow users to use the entire GPU memory instead of restricting them to a fraction.
What are you doing to reset the GPU to clean state after a run? It's surprisingly complicated to do this securely (we're writing up a back-to-back sequence of audits we did with Atredis and Tetrel; should be publishing in a month or two).
Does anyone know if this is possible with USB?
I have a Davinci Resolve license USB-dongle I'd like to not plugging into my laptop.
Down the line we want to move to a pay-as-you-go model.
Down the line, we could see this being used for batched render jobs (i.e. to replace a render farm).
I think you'd want to run the blender GUI locally and only call out to a headless rendering server ("render farm") that uses your service under the hood to get the actual render.
This separation is already something blender supports, and you could for instance use Blender on Windows despite your render farm using Linux servers.
Cloud rendering is adjacent to what you're offering, and it should be trivial for you to expand into that space by just figuring out the setup and preparing a guide for users wishing to do that with your service.
[1] https://pastebin.com/raw/kCYmXr5A
LD_PRELOAD trick allows you to intercept and virtualize calls to the CUDA runtime.
The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.
The same way one "approaches the sun" when they take the stairs?
Curious where you see this in the CLI, may be an oversight on our part. If you can join the Discord and point us to this bug we would really appreciate it!
Another solution is qCUDA [3] which is more specialized towards CUDA.
In addition to these solutions, various virtualization solutions today provide some sort of serialization mechanism for GPU commands, so they can be transferred to another host (or process). [4]
One example is the QEMU-based Android Emulator. It is using special translator libraries and a "QEMU Pipe" to efficiently communicate GPU commands from the virtualized Android OS to the host OS [5].
The new Cuttlefish Android emulator [6] uses Gallium3D for transport and the virglrenderer library [7].
I'd expect that the current virtio-gpu implementation in QEMU [8] might make this job even easier, because it includes the Android's gfxstream [9] (formerly called "Vulkan Cereal") that should already support communication over network sockets out of the box.
[1] https://github.com/pocl/pocl
[2] https://portablecl.org/docs/html/remote.html
[3] https://github.com/coldfunction/qCUDA
[4] https://www.linaro.org/blog/a-closer-look-at-virtio-and-gpu-...
[5] https://android.googlesource.com/platform/external/qemu/+/em...
[6] https://source.android.com/docs/devices/cuttlefish/gpu
[7] https://cs.android.com/android/platform/superproject/main/+/...
[8] https://www.qemu.org/docs/master/system/devices/virtio-gpu.h...
[9] https://android.googlesource.com/platform/hardware/google/gf...