I think I will stick with the wine+proton hoops. I am even planning on selling my nvidia card to go with a RNDA2 GPU so as to have gpu drivers in the kernel.
Not taking away from all the work that goes into this -- and I love hacking away as much as the next nerd -- this is just a bit janky, or rather, I'm hesitant to try this on my main workstation.
Most package updates don't disrupt anything. Depending on the vendor driver you use (Intel i915, AMD GPU-IOV Module, Nvidia) kernel updates should be okay if we use DKMS. GPU-IOV Module works okay with DKMS if you know what you're doing. I can't speak too much to i915. On Nvidia DKMS is hit or miss but that depends on which driver version is used. Some folks report DKMS doesn't work at all, others say they've never had a problem with it. I think it also depends on the kernel version DKMS is compiling against as in some cases the driver hasn't been updated yet to support the latest kernel version.
We try to help with that by making patch files that run ahead of official driver/kernel version support (they basically just make the driver work on a newer kernel - that's all):
NVIDIA's vGPU technology has been cracked to work on most Maxwell and later devices. AMD and Intel have not been cracked and researched in the same ways.
There are some limitations involved in LibVF.IO such as pre-defined mdev types. GVM is entirely free/libre open source software and it supports arbitrary mdev types. :)
We'll do our best to add more supported vendors to GVM/mdev-gpu in the near future.
Nvidia consumer cards like the 1060 can now be shared between host and VM? I though that was limited to datacenter cards and consumer cards must use exclusive PCI-E passthrough?
This seems to be at the wrong level of abstraction.
I want a virtual Vulkan, one per CPU VM. I think I read somebody was working on that, or something like. That way it works on any GPU, not just NVidia or NVidia/Intel.
Looking Glass is kind of the closest you'll get for a trivial "I want shared GPU virtualization on my workstation", but GPU partitioning doesn't really work that way. Outside of the baseline support in the hardware itself, which is nowhere near generic enough for "works on any GPU" (it took supervisory frameworks to even get CUDA/OpenCL/etc to a point where you can stop worrying about writing transforms from scratch and just let PyTorch abstract it a little), this model of GPU partitioning doesn't perform well.
How do you allocate vGPU memory between a ML/AI VM, a VDI VM, and a gaming/CAD VM? All have dramatically different requirements. You also can't think of shader/GPU cores in any way similar to CPU cores. They're essentially just vector/linear algebra accelerators with little to no branch prediction, speculative execution, or anything else you'd expect.
There's an effort, but it's far from where you want, and there's no indication it will get there unless you can get all the vendors to agree on a standard at some point in the future.
It appears to be an open-source implementation of nvidia-cli for managing mdevs, which is arbitrarily nice, but it's not clear to end-users what this means.
Pre-Ampere GPUs (Ampere+ is MIG) were able to use the mediated device subsystem to partition cards into time slices. It's similar to SR-IOV, except that you can specify the size of the partition with more granularity than "give me a new virtual device".
Intel has GVT-g, which is reasonably widely supported on Gen12 and above (edit: too late/early -- Gen11 and earlier, not Gen12 and later -- thanks to my123 for the correction). nVidia had vGPU/mdev, and newer generations (Ampere and later) use MIG. It's unclear whether this supports MIG at all. AMD uses MxGPU, and they've never really cared about/pursued anything related to this, probably because their datacenter penetration is about 1%.
MxGPU is only supported on some FirePro cards. mdev was largely on GRID cards (mostly Tesla, some Quadros). MIG is on AXXX cards.
It's unclear why anyone should use this over mdevctl, which already supports GVT-g, and it's also unclear whether this is tied to the (very much "don't use in production") open source nvidia drivers.
For end-users, GVT-g, getting a cheap older GRID card, or using Looking Glass for your GPU are all more reasonable options.
This effort is great, but the readme is appallingly short on information even for someone who knows the problem domain.
> GVT-g or a new GPU is probably a better solution, though. Are you sure GVT-g is Gen12 though? I always thought Intel switched to SRIOV by then.
Currently, for Gen11 and later (in Ice Lake), neither GVT-g nor SR-IOV-based GPU virt options are available. As we're several years past Gen11 launch, seems that Intel isn't interested in GPU virtualisation much anymore on a Linux host.
More complex than that. MIG covers compute use cases only. For workloads where graphics are needed (or even the graphics APIs), you have to use preemptive scheduling, even on Ampere.
> which is reasonably widely supported on Gen12 and above
No. It's gone on Gen11 and later. :/ And no replacement yet.
> MxGPU is only supported on some FirePro cards
Forget about AMD cards for GPU virtualisation. The modern GIM drivers aren't public. So you're stuck with very old out of support GPUs.
> and it's also unclear whether this is tied to the (very much "don't use in production") open source nvidia drivers.
The OSS NV KM stack doesn't support GPU virt at all yet.
> More complex than that. MIG covers compute use cases only. For workloads where graphics are needed (or even the graphics APIs), you have to use preemptive scheduling, even on Ampere.
Thanks for the correction! Shows how much I've really used it outside of "does nvidia-smi do the right thing, and can I map those to pods". I had assumed that, since nvidia-smi on Ampere did both GPU and CPU slicing, that maybe they found a solution which allowed them to sidestep all of the inherent problems with trying to share GPU memory, but too much to hope for.
> No. It's gone on Gen11 and later. :/ And no replacement yet.
I've been relatively distanced from the nitty-gritty parts of GPU virt for a couple of years, but how/why did the end up pushing for root SR-IOV support for Xe along with patches for ReBAR at the same time as they dropped this?
I guess they'll gate it behind enterprise dGPU SKUs when/if they ever release. Welp.
> The OSS NV KM stack doesn't support GPU virt at all yet.
I would have thought this also. As bad as the doc in the link is, it does explicitly reference OpenRM verification. I wouldn't trust nVidia's "big ball of hardly-commented source" for anything yet, but at least it is the "real" driver and not Nouveau, so I'd expect GPU virt to be supported, even if the code may be inscrutable.
I'm really just waiting for the HN link to confirm/deny the long-held suspicion that nVidia suddenly got really good around the same time SGI was in its death throes, and that the reason they resisted opening their drivers was due to routines with... questionable IP.
We did some things to enable GPU virtualization on older driver revisions for most Nvidia consumer GPUs and some AMD GPUs as well. Here's that post if you'd like to take a look:
We're building a free/libre (AGPLv3 & GPLv2) virtualization stack intended to support i915 (Intel) and Nvidia (OpenRM). We're planning support for Tenstorrent TPUs and in the longer term possibly AMD GPUs.
Here's a full summary of how GPU virtualization actually works under various modes from our wiki. This includes VFIO-Mediated Devices (Mdev), Single Root I/O Virtualization (SR-IOV), and Intel's new Scalable I/O Virtualization (SIOV):
Curious about Tenstorrent.
Did you ever get your hands on any of their hardware? We are just getting excuses from them since several months and have not yet received even a single unit for evaluation.
Then a link to Intel's LTS Linux repo. Do your scripts, if it's Xe, actually clone/build this and boot into it? What does "supported" mean in this sense? Do you replace the user's kernel with the intel-lts fork? If so, do you tell them?
> We did some things to enable GPU virtualization on older driver revisions for most Nvidia consumer GPUs and some AMD GPUs as well. Here's that post if you'd like to take a look:
You have some patches to drivers. Which is great. But "we did some things" on HN is best followed with "here's what we did from a technical level" or "here's the source", not "here's how to install our product".
Is there anything at all here which isn't applicable to other KVM-based virt platforms? It looks like not.
> We're building a free/libre (AGPLv3 & GPLv2) virtualization stack intended to support i915 (Intel) and Nvidia (OpenRM).
Again, this is great. But please be clear that "we're building a free/libre virtualization stack" is "we are building a very opinionated wrapper around qemu+KVM which is not using libvirt bindings for some reason". The "stack" definitely looks like "some utility scripts to make this easier".
Is there a project homepage with a roadmap, goals, and issues/bugs, and so on?
Again as in your other comment GVM does not actually have anything to do with LibVF.IO right now. They don't even work together. GVM/Mdev-GPU is actually a free software component that virtualizes GPUs, not a wrapper for QEMU/KVM. That's misinformation.
GVM/Mdev-GPU is over here - not in the LibVF.IO codebase which you link to:
Hey, OP here. Sorry I didn't get to this sooner as I posted this just before falling asleep. This comment basically hits the nail on the head in most areas. I am happy to say though that Intel now uses SR-IOV instead of GVT-g (deprecated since 9th generation Intel). Their replacement SR-IOV driver is now Open Source (recently made public):
> Hey, OP here. Sorry I didn't get to this sooner as I posted this just before falling asleep. This comment basically hits the nail on the head in most areas. I am happy to say though that Intel now uses SR-IOV instead of GVT-g (deprecated since 9th generation Intel). Their replacement SR-IOV driver is now Open Source (recently made public):
This would legitimately be ideal information to have in the readme/top-level link.
This is pretty unhelpful. Legitimately. It's not mainlined yet, there are zero userspace docs, etc. The patch looks like it will pretty much "just work" when/if Intel bothers to get it into mainline. Until then, a patched/forked kernel is needed.
> Also to address the first comment in this thread - there are many inaccuracies here:
> Post-Ampere supports MIG and SR-IOV. VFIO-Mediated Devices (Mdev) are used both pre-Ampere and post-Ampere. This is how that works:
I maintained mdev support for a major KVM-based platform, but it's been a couple of years. That said, a link to how mdev internals work isn't useful to end-users, who just want to know "how do I partition my card"? As-in "which driver/utilities do I need to install"?
> For folks who are interested we also built LibVF.IO which enables vGPU/SR-IOV functionality on consumer GPUs:
Is there some way in which LibVF.IO differs from just being a wrapper around KVM/qemu? Because the scripts do an awful lot of stuff to your host system, and arcd.nim appears to just call qemu anyway:
https://github.com/Arc-Compute/LibVF.IO/blob/master/src/libv...
Sure, it also binds/removes mdev devices, which is a nice convenience, and you have a couple of patches applied to the nvidia driver sources, but asking users to blindly execute scripts they have to wade through to find out exactly what they're going to do in kernelspace, plus the system. It's... asking a lot.
You're adding a virtual sound card, nim, shell aliases, samba, plasma, then blindly overwriting any kernel options the user has set without even the good grace of capturing them and appending them:
https://github.com/Arc-Compute/LibVF.IO/blob/master/scripts/...
> Also finally this tool has nothing to do with nvidia-cli or mdevctl. It defines available mdev devices in the mdevctl list, it does not replace mdevctl.
It looks like it does a lot more than that, and less than that. It has nothing to do with nvidia-cli (which can also manage mdev devices), and "defines available mdev devices in the mdevctl list" only as a side effect of the fact that it's doing a whole bunch of other stuff to the system.
I'm not trying to tear down your project, but honestly, the docs could be far, far better about what this actually does, how it does it, which pieces ...
Indeed, "98 changed files with 11,276 additions and 46 deletions" and no idea if it will work on a vanilla kernel.
I would like to try running linux baremetal to virtualize Windows 11 running in fullscreen mode with control over the mouse and keyboard, but I may wait until that's mainlined.
I am also looking forward to when it's mainlined. Having said that until a short time ago everything going on with i915 SR-IOV was a secret (nobody on the i915 mailing list would talk about it, there was no open source) so I'd say we're moving in the right direction.
> That said, a link to how mdev internals work isn't useful to end-users, who just want to know "how do I partition my card"? As-in "which driver/utilities do I need to install"?
OpenMdev.io is meant for developers, not for users.
> Is there some way in which LibVF.IO differs from just being a wrapper around KVM/qemu?
No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:
>"defines available mdev devices in the mdevctl list" only as a side effect of the fact that it's doing a whole bunch of other stuff to the system.
GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.
>Similarly, it strongly appears that arcd could be more or less replaced by any given binding to the libvirt API, which would also allow the VMs to be easily migrated (assuming identical hardware), the libvirt XML shared with others, snapshotting, storage pooling, listed in virt-manager/virsh, and so on.
It's actually unrelated to GVM. You can use GVM with whatever you want, including Libvirt/Virsh/Virt-Manager because we wanted to support users of those things with GVM rather than requiring that they use LibVF.IO.
> GVM, rather than "a GPU Virtual Machine ..." is Haskell bindings for the RMAPI.
Well, we do create mediated devices exposed in mdevctl defined by a user config file, so I would say it goes a fair amount beyond Haskell bindings for the RMAPI.
I think it's reasonable to describe a GPU mediated device as a virtual GPU given you get a virtual function that represents a scheduling share and virtual BAR space with a share of the device VRAM (partition of the GPU) which you can pass to one or several guests to allow them to run an unmodified guest GPU driver. I can't really think of a better definition for a vGPU. The Mediated Device Internals article pretty much explains the APIs GVM is dealing with - I believe we even link some sample code:
https://openmdev.io/index.php/Mediated_Device_Internals
Your comment seems kind of trollish so I'm not really sure what benefit continuing this thread has. I think most of the stuff you're asking about is more or less documented and spelled out as openly as we're able to. What we're trying to do here is to make this stuff more open and available to people rather than locked away behind binary blobs. More or less everything we do is put into our wiki with very few exceptions. OpenMdev.io is made to be open to our community of folks working on Mediated Device/IO Virtualization functions on various projects so if you're a developer on this stuff and think anything is lacking you're welcome to contribute or or make suggestions in our IRC or Discord. I'm sure there's always room to improve and we put a ton of effort into trying to listen to feedback and improve upon things ourselves as well as accept contributions from others.
> OpenMdev.io is meant for developers, not for users.
Frankly, it isn't meant for developers, either. Almost every page on that site is either woefully incomplete, or crib notes from docs/talks, which is fine for a high-level overview, but it's not an API reference developers can use either. The sample code is mostly just lifted from other places (such as https://github.com/torvalds/linux/blob/master/samples/vfio-m...), so useless you're better off reading the source (https://openmdev.io/index.php/OpenRM), or just links to other people's APIs which interested devs can find.
It's fine to collate this, but it's far more like someone's personal aggregator than any kind of reference site.
> No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:
I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.
What advantages does your system offer?
> GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.
I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.
I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language they like better.
It's not clear from the outset what the advantage is over just submitting a PR to mdevctl to echo into /sys/devices/..../[create|remove], and overall, the README doesn't give any information about it whatsoever, even `--help` output to show the args and defaults.
No, it is not. Point blank, it is not. libvirt also isn't. Even qemu isn't for hardware virt, and you're not doing IOMMU operations on emulated CPU calls. kvm is. It's a toolkit to manage virtual machines, maybe.
This does not answer the question at all of "why not libvirt?"
> It's actually unrelated to GVM. You can use GVM with whatever you want, ...
> Frankly, it isn't meant for developers, either. Almost every page on that site is either woefully incomplete, or crib notes from docs/talks, which is fine for a high-level overview, but it's not an API reference developers can use either. The sample code is mostly just lifted from other places (such as https://github.com/torvalds/linux/blob/master/samples/vfio-m...), so useless you're better off reading the source (https://openmdev.io/index.php/OpenRM), or just links to other people's APIs which interested devs can find.
Well, the Mediated Device Internals article does cite every piece of information used but it's still incomplete. I made that page because it's largely a collection of everything I've learned while developing software. The "lifted" samples literally have the authors at the top of every header and the original link is the first line beneath the title in the README.md:
https://github.com/OpenMdev/VFIO-Mdev_Samples
> I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.
You define types that are presented in mdevctl via a YAML config file the user can write themselves or via a JSON file. So LibVF.IO might as well be entirely ignorant to the fact that GVM is on the system. All it sees are different arbitrary mdev types that the user decides to create. Not sure where you're going with this comment.
>I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.
>What advantages does your system offer?
Well for starters I've never argued LibVF.IO is better than libvirt. In-fact when folks bring it up I generally say there's a lot we can learn from libvirt.
GVM/Mdev-GPU isn't LibVF.IO so I'm kind of blown away that you keep bringing that up as if they're the same. GVM/Mdev-GPU controls which types the vendor driver presents to mdevctl - GVM/Mdev-GPU is neither an mdevctl replacement nor does it share code with or even integrate with LibVF.IO. I don't think you really grasp that.
GVM/Mdev-GPU enables the creation of arbitrary mdev types against the Nvidia driver as defined by the user. Those types generally have "capabilities" restricted by arbitrary policy and "available mdev types" are stored in an XML file alongside signatures which attempt to stop the user from defining their own types. GVM/Mdev-GPU gives control back to the user. Also our codebase is GPLv2 and we don't enforce any kind of "device-specific" restrictions. Nvidia's equivalent is closed source and intentionally locks out consumer cards.
>I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language the...
> GVM isn't built in a way which is usable by any other project. That's ok, but it does nothing to explain the design decision.
It is. You configure a YAML/JSON file to define the mdev types that are presented by the vendor driver in the standard sysfsdev mdev_bus path. Those appear precisely the same way any mdev type appears to any program which uses the VFIO-Mdev API. In other words you can install GVM/Mdev-GPU underneath Libvirt or talk to it using mdevctl.
> You create mediated devices for nVidia devices by sending ioctls exposed via RMAPI. This is potato/potato. The explanation you just gave STILL makes it sound as if this is a novel thing done by GVM/mdev-gpu rather than something common, and talking down to someone who is asking informed questions about why you did it this way by linking to internals (when I was physically there for most of those talks and helped write some of the docs) doesn't paint a pretty picture.
This isn't potato/potato. All of these functions are artificially locked out on closed source programs that gatekeep the ability to use your own GPU. No, we're not making it sound like something special. All of our documentation that you seem to think linking to amounts to "talking down to you" because of your position in industry references common virtualization APIs in the kernel and drivers. We link to sample code on how to talk to those APIs. If anything we're showing people this stuff isn't magic! That's the whole point of doing this as open source and making it work on everything we can rather than making it closed source and locking it out to certain devices then charging you for the use of your own hardware.
> NONE OF THE STUFF I'M ASKING ABOUT IS DOCUMENTED OR SPELLED OUT. That's the point. From someone who was a maintainer, engineering leader, etc on a major open source virtualization platform who literally wrote code which does this kind of scheduling/creation across a cluster
I'm glad you have no problem gatekeeping in your comments or propping yourself up based on your job history. Many folks have offered constructive criticism as well as guidance on where to place our attention next many of whom I strongly suspect are at least as experienced as you (those folks who notably don't feel the need to often repeat how senior they are) but thanks again for the grandstanding so plainly.
>I am telling you that your documentation is opaque, misleading, takes credit for things you did not invent
No it doesn't. I constantly link to everything we're talking about on OpenMdev and to all the original sources, talks, mailing list posts, sample code. Please find any example you like of inappropriately taking credit for things and I'll happily correct it.
> where that "missing middle" is /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen"), etc.
This comment makes no sense. /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen" has nothing to do with GVM/Mdev-GPU.
> This is, or could be, a great start to a unified ecosystem. You are going to have a very hard time getting a developer/user ecosystem if you do not provide better documentation,
I agree we need to improve our documentation a lot still yet. It's open for community contribution. You obviously have some idea of what you're talking about enough that we can have this argument so as much as this has dragged on for an uncomfortable amount of time and I still think you have several fundamental misconceptions about what GVM/Mdev-GPU actually does it would be nice if I could structure this interaction with you into something I could use to condense what you're saying here into documentation improvements rather than a flame war in a web forum comment section.
> and most of all, to acknowledge the work other have done/the knowledge they have rather than presenti...
> As a basic open source citizen, this would also help in "giving credit where credit is due". Bravo for stitching all of this together, but the project pages/repo certainly make it seem as if LibVF.IO/GVM did the work, and that's not really honest
GVM/Mdev-GPU depends on the Nvidia driver in this release. I think that's pretty clear but if you're not aware the page this thread is about does reference OpenRM driver interactions.
Not that this thread is about LibVF.IO which you've devoted most of your attention to throughout this discussion but we do give credit to Looking Glass, vGPU_Unlock, AMD GPU-IOV Module, and Intel i915 in the LibVF.IO install guide - they are discussed as part of the install process where they're mentioned explicitly by name. We also link them at the bottom of the article:
https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...
Intel has already confirmed that GVT-g is essentially dead and not supported on their Iris/Xe or anything newer graphics.. We can also confirm this via their own drivers source..
Thanks, I came here looking for an explanation, I had no idea what it was after reading the Readme, even though I'd already heard of some of the related tech.
Nearest I can tell the guy earlier thought what mdevctl is (echo uuid >> /sys/mdev_bus/$pci-device/available-types/) is the same as what we are doing.
That couldn't be further from the truth.
I tried explaining to him we arbitrarily define the available types on behalf of the vendor driver via user configuration. It’s the difference between picking from a pre-made list of things someone else says you’re allowed to do and making the list of things you’re allowed to do yourself.
54 comments
[ 4.0 ms ] story [ 124 ms ] threadHere's our current install documentation:
https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...
This will be updated against our GVM/mdev-gpu sources shortly but our current documentation still will get you to what you're asking about. :)
I think I will stick with the wine+proton hoops. I am even planning on selling my nvidia card to go with a RNDA2 GPU so as to have gpu drivers in the kernel.
Not taking away from all the work that goes into this -- and I love hacking away as much as the next nerd -- this is just a bit janky, or rather, I'm hesitant to try this on my main workstation.
One concern I had was having to uninstall the Nvidia drivers and used the patched ones?. How does that play with pkg manager updates and things?
Most package updates don't disrupt anything. Depending on the vendor driver you use (Intel i915, AMD GPU-IOV Module, Nvidia) kernel updates should be okay if we use DKMS. GPU-IOV Module works okay with DKMS if you know what you're doing. I can't speak too much to i915. On Nvidia DKMS is hit or miss but that depends on which driver version is used. Some folks report DKMS doesn't work at all, others say they've never had a problem with it. I think it also depends on the kernel version DKMS is compiling against as in some cases the driver hasn't been updated yet to support the latest kernel version.
We try to help with that by making patch files that run ahead of official driver/kernel version support (they basically just make the driver work on a newer kernel - that's all):
https://github.com/Arc-Compute/LibVF.IO/tree/master/patches/
https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...
https://news.ycombinator.com/item?id=28944426
https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...
A full list of supported devices is available here:
https://openmdev.io/index.php/GPU_Support
There are some limitations involved in LibVF.IO such as pre-defined mdev types. GVM is entirely free/libre open source software and it supports arbitrary mdev types. :)
We'll do our best to add more supported vendors to GVM/mdev-gpu in the near future.
I want a virtual Vulkan, one per CPU VM. I think I read somebody was working on that, or something like. That way it works on any GPU, not just NVidia or NVidia/Intel.
How do you allocate vGPU memory between a ML/AI VM, a VDI VM, and a gaming/CAD VM? All have dramatically different requirements. You also can't think of shader/GPU cores in any way similar to CPU cores. They're essentially just vector/linear algebra accelerators with little to no branch prediction, speculative execution, or anything else you'd expect.
Otherwise, you can sort of follow along here: https://openmdev.io/index.php/GPU_Support
There's an effort, but it's far from where you want, and there's no indication it will get there unless you can get all the vendors to agree on a standard at some point in the future.
https://openmdev.io/index.php/Mediated_Device_Internals
We're doing most of our work (almost all) as open source and we're trying to make sure we have good documentation too. :)
Our company website is over here if you're interested to take a look:
https://arccompute.io/
It appears to be an open-source implementation of nvidia-cli for managing mdevs, which is arbitrarily nice, but it's not clear to end-users what this means.
Pre-Ampere GPUs (Ampere+ is MIG) were able to use the mediated device subsystem to partition cards into time slices. It's similar to SR-IOV, except that you can specify the size of the partition with more granularity than "give me a new virtual device".
Intel has GVT-g, which is reasonably widely supported on Gen12 and above (edit: too late/early -- Gen11 and earlier, not Gen12 and later -- thanks to my123 for the correction). nVidia had vGPU/mdev, and newer generations (Ampere and later) use MIG. It's unclear whether this supports MIG at all. AMD uses MxGPU, and they've never really cared about/pursued anything related to this, probably because their datacenter penetration is about 1%.
MxGPU is only supported on some FirePro cards. mdev was largely on GRID cards (mostly Tesla, some Quadros). MIG is on AXXX cards.
It's unclear why anyone should use this over mdevctl, which already supports GVT-g, and it's also unclear whether this is tied to the (very much "don't use in production") open source nvidia drivers.
For end-users, GVT-g, getting a cheap older GRID card, or using Looking Glass for your GPU are all more reasonable options.
This effort is great, but the readme is appallingly short on information even for someone who knows the problem domain.
GVT-g or a new GPU is probably a better solution, though. Are you sure GVT-g is Gen12 though? I always thought Intel switched to SRIOV by then.
Currently, for Gen11 and later (in Ice Lake), neither GVT-g nor SR-IOV-based GPU virt options are available. As we're several years past Gen11 launch, seems that Intel isn't interested in GPU virtualisation much anymore on a Linux host.
Here's the commit to Intel i915 containing SR-IOV feature enablement! :)
https://github.com/intel/linux-intel-lts/commit/41ef979f0894...
If you'd like to checkout a full list of supported GPUs that's over here:
https://openmdev.io/index.php/GPU_Support
https://news.ycombinator.com/item?id=28944426
The full list of supported GPUs is here:
https://openmdev.io/index.php/GPU_Support
More complex than that. MIG covers compute use cases only. For workloads where graphics are needed (or even the graphics APIs), you have to use preemptive scheduling, even on Ampere.
> which is reasonably widely supported on Gen12 and above
No. It's gone on Gen11 and later. :/ And no replacement yet.
> MxGPU is only supported on some FirePro cards
Forget about AMD cards for GPU virtualisation. The modern GIM drivers aren't public. So you're stuck with very old out of support GPUs.
> and it's also unclear whether this is tied to the (very much "don't use in production") open source nvidia drivers.
The OSS NV KM stack doesn't support GPU virt at all yet.
Thanks for the correction! Shows how much I've really used it outside of "does nvidia-smi do the right thing, and can I map those to pods". I had assumed that, since nvidia-smi on Ampere did both GPU and CPU slicing, that maybe they found a solution which allowed them to sidestep all of the inherent problems with trying to share GPU memory, but too much to hope for.
> No. It's gone on Gen11 and later. :/ And no replacement yet.
I've been relatively distanced from the nitty-gritty parts of GPU virt for a couple of years, but how/why did the end up pushing for root SR-IOV support for Xe along with patches for ReBAR at the same time as they dropped this?
https://www.intel.com/content/www/us/en/support/articles/000...
I guess they'll gate it behind enterprise dGPU SKUs when/if they ever release. Welp.
> The OSS NV KM stack doesn't support GPU virt at all yet.
I would have thought this also. As bad as the doc in the link is, it does explicitly reference OpenRM verification. I wouldn't trust nVidia's "big ball of hardly-commented source" for anything yet, but at least it is the "real" driver and not Nouveau, so I'd expect GPU virt to be supported, even if the code may be inscrutable.
I'm really just waiting for the HN link to confirm/deny the long-held suspicion that nVidia suddenly got really good around the same time SGI was in its death throes, and that the reason they resisted opening their drivers was due to routines with... questionable IP.
https://openmdev.io/index.php/GPU_Support
We did some things to enable GPU virtualization on older driver revisions for most Nvidia consumer GPUs and some AMD GPUs as well. Here's that post if you'd like to take a look:
https://news.ycombinator.com/item?id=28944426
We're building a free/libre (AGPLv3 & GPLv2) virtualization stack intended to support i915 (Intel) and Nvidia (OpenRM). We're planning support for Tenstorrent TPUs and in the longer term possibly AMD GPUs.
Here's a full summary of how GPU virtualization actually works under various modes from our wiki. This includes VFIO-Mediated Devices (Mdev), Single Root I/O Virtualization (SR-IOV), and Intel's new Scalable I/O Virtualization (SIOV):
https://openmdev.io/index.php/Mediated_Device_Internals
> https://openmdev.io/index.php/GPU_Support
I'm not trying to beat a dead horse here, but your matrix links to an archive of Intel's community forum, where it's a basic question, about Windows.
https://archive.ph/0McAE#selection-4883.0-4943.35
Then a link to Intel's LTS Linux repo. Do your scripts, if it's Xe, actually clone/build this and boot into it? What does "supported" mean in this sense? Do you replace the user's kernel with the intel-lts fork? If so, do you tell them?
> We did some things to enable GPU virtualization on older driver revisions for most Nvidia consumer GPUs and some AMD GPUs as well. Here's that post if you'd like to take a look:
You have some patches to drivers. Which is great. But "we did some things" on HN is best followed with "here's what we did from a technical level" or "here's the source", not "here's how to install our product".
https://github.com/Arc-Compute/LibVF.IO/tree/master/patches
Is there anything at all here which isn't applicable to other KVM-based virt platforms? It looks like not.
> We're building a free/libre (AGPLv3 & GPLv2) virtualization stack intended to support i915 (Intel) and Nvidia (OpenRM).
Again, this is great. But please be clear that "we're building a free/libre virtualization stack" is "we are building a very opinionated wrapper around qemu+KVM which is not using libvirt bindings for some reason". The "stack" definitely looks like "some utility scripts to make this easier".
Is there a project homepage with a roadmap, goals, and issues/bugs, and so on?
GVM/Mdev-GPU is over here - not in the LibVF.IO codebase which you link to:
https://github.com/Arc-Compute/Mdev-GPU
https://docs.linux-gvm.org/mdev-gpu/
Also Intel's SR-IOV isn't my code. You have to ask them about that.
https://github.com/intel/linux-intel-lts/commit/41ef979f0894...
Also to address the first comment in this thread - there are many inaccuracies here:
Post-Ampere supports MIG and SR-IOV. VFIO-Mediated Devices (Mdev) are used both pre-Ampere and post-Ampere. This is how that works:
https://openmdev.io/index.php/Mediated_Device_Internals
Also this tool has nothing to do with nvidia-cli or mdevctl. It defines available mdev devices in the mdevctl list, it does not replace mdevctl.
For folks who are interested we also built LibVF.IO which enables vGPU/SR-IOV functionality on consumer GPUs:
https://news.ycombinator.com/item?id=28944426
Here's a full list of supported GPUs from our wiki:
https://openmdev.io/index.php/GPU_Support
This would legitimately be ideal information to have in the readme/top-level link.
> https://github.com/intel/linux-intel-lts/commit/41ef979f0894
This is pretty unhelpful. Legitimately. It's not mainlined yet, there are zero userspace docs, etc. The patch looks like it will pretty much "just work" when/if Intel bothers to get it into mainline. Until then, a patched/forked kernel is needed.
> Also to address the first comment in this thread - there are many inaccuracies here:
> Post-Ampere supports MIG and SR-IOV. VFIO-Mediated Devices (Mdev) are used both pre-Ampere and post-Ampere. This is how that works:
> https://openmdev.io/index.php/Mediated_Device_Internals
I maintained mdev support for a major KVM-based platform, but it's been a couple of years. That said, a link to how mdev internals work isn't useful to end-users, who just want to know "how do I partition my card"? As-in "which driver/utilities do I need to install"?
> For folks who are interested we also built LibVF.IO which enables vGPU/SR-IOV functionality on consumer GPUs:
> https://news.ycombinator.com/item?id=28944426
> If you're interested in a full list of supported GPUs you can read the following page from our wiki:
> https://openmdev.io/index.php/GPU_Support
Is there some way in which LibVF.IO differs from just being a wrapper around KVM/qemu? Because the scripts do an awful lot of stuff to your host system, and arcd.nim appears to just call qemu anyway: https://github.com/Arc-Compute/LibVF.IO/blob/master/src/libv...
Sure, it also binds/removes mdev devices, which is a nice convenience, and you have a couple of patches applied to the nvidia driver sources, but asking users to blindly execute scripts they have to wade through to find out exactly what they're going to do in kernelspace, plus the system. It's... asking a lot.
You're adding a virtual sound card, nim, shell aliases, samba, plasma, then blindly overwriting any kernel options the user has set without even the good grace of capturing them and appending them: https://github.com/Arc-Compute/LibVF.IO/blob/master/scripts/...
> Also finally this tool has nothing to do with nvidia-cli or mdevctl. It defines available mdev devices in the mdevctl list, it does not replace mdevctl.
It looks like it does a lot more than that, and less than that. It has nothing to do with nvidia-cli (which can also manage mdev devices), and "defines available mdev devices in the mdevctl list" only as a side effect of the fact that it's doing a whole bunch of other stuff to the system.
I'm not trying to tear down your project, but honestly, the docs could be far, far better about what this actually does, how it does it, which pieces ...
> This is pretty unhelpful. Legitimately
Indeed, "98 changed files with 11,276 additions and 46 deletions" and no idea if it will work on a vanilla kernel.
I would like to try running linux baremetal to virtualize Windows 11 running in fullscreen mode with control over the mouse and keyboard, but I may wait until that's mainlined.
OpenMdev.io is meant for developers, not for users.
> Is there some way in which LibVF.IO differs from just being a wrapper around KVM/qemu?
No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:
https://openmdev.io/index.php/LibVF.IO
>"defines available mdev devices in the mdevctl list" only as a side effect of the fact that it's doing a whole bunch of other stuff to the system.
GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.
GVM/Mdev-GPU creates the mediated devices that are exposed in the mdevctl list. Read this code instead: https://github.com/Arc-Compute/Mdev-GPU/
>Similarly, it strongly appears that arcd could be more or less replaced by any given binding to the libvirt API, which would also allow the VMs to be easily migrated (assuming identical hardware), the libvirt XML shared with others, snapshotting, storage pooling, listed in virt-manager/virsh, and so on.
Sure, arcd is a reference Virtual Machine Monitor as it says at the top of this page: https://openmdev.io/index.php/LibVF.IO
It's actually unrelated to GVM. You can use GVM with whatever you want, including Libvirt/Virsh/Virt-Manager because we wanted to support users of those things with GVM rather than requiring that they use LibVF.IO.
> GVM, rather than "a GPU Virtual Machine ..." is Haskell bindings for the RMAPI.
Well, we do create mediated devices exposed in mdevctl defined by a user config file, so I would say it goes a fair amount beyond Haskell bindings for the RMAPI.
I think it's reasonable to describe a GPU mediated device as a virtual GPU given you get a virtual function that represents a scheduling share and virtual BAR space with a share of the device VRAM (partition of the GPU) which you can pass to one or several guests to allow them to run an unmodified guest GPU driver. I can't really think of a better definition for a vGPU. The Mediated Device Internals article pretty much explains the APIs GVM is dealing with - I believe we even link some sample code: https://openmdev.io/index.php/Mediated_Device_Internals
Your comment seems kind of trollish so I'm not really sure what benefit continuing this thread has. I think most of the stuff you're asking about is more or less documented and spelled out as openly as we're able to. What we're trying to do here is to make this stuff more open and available to people rather than locked away behind binary blobs. More or less everything we do is put into our wiki with very few exceptions. OpenMdev.io is made to be open to our community of folks working on Mediated Device/IO Virtualization functions on various projects so if you're a developer on this stuff and think anything is lacking you're welcome to contribute or or make suggestions in our IRC or Discord. I'm sure there's always room to improve and we put a ton of effort into trying to listen to feedback and improve upon things ourselves as well as accept contributions from others.
Frankly, it isn't meant for developers, either. Almost every page on that site is either woefully incomplete, or crib notes from docs/talks, which is fine for a high-level overview, but it's not an API reference developers can use either. The sample code is mostly just lifted from other places (such as https://github.com/torvalds/linux/blob/master/samples/vfio-m...), so useless you're better off reading the source (https://openmdev.io/index.php/OpenRM), or just links to other people's APIs which interested devs can find.
It's fine to collate this, but it's far more like someone's personal aggregator than any kind of reference site.
> No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:
> https://openmdev.io/index.php/LibVF.IO
I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.
What advantages does your system offer?
> GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.
I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.
> GVM/Mdev-GPU creates the mediated devices that are exposed in the mdevctl list. Read this code instead: https://github.com/Arc-Compute/Mdev-GPU/
I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language they like better.
It's not clear from the outset what the advantage is over just submitting a PR to mdevctl to echo into /sys/devices/..../[create|remove], and overall, the README doesn't give any information about it whatsoever, even `--help` output to show the args and defaults.
> Sure, arcd is a reference Virtual Machine Monitor as it says at the top of this page: https://openmdev.io/index.php/LibVF.IO
No, it is not. Point blank, it is not. libvirt also isn't. Even qemu isn't for hardware virt, and you're not doing IOMMU operations on emulated CPU calls. kvm is. It's a toolkit to manage virtual machines, maybe.
This does not answer the question at all of "why not libvirt?"
> It's actually unrelated to GVM. You can use GVM with whatever you want, ...
Well, the Mediated Device Internals article does cite every piece of information used but it's still incomplete. I made that page because it's largely a collection of everything I've learned while developing software. The "lifted" samples literally have the authors at the top of every header and the original link is the first line beneath the title in the README.md: https://github.com/OpenMdev/VFIO-Mdev_Samples
> I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.
You define types that are presented in mdevctl via a YAML config file the user can write themselves or via a JSON file. So LibVF.IO might as well be entirely ignorant to the fact that GVM is on the system. All it sees are different arbitrary mdev types that the user decides to create. Not sure where you're going with this comment.
>I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.
>What advantages does your system offer?
Well for starters I've never argued LibVF.IO is better than libvirt. In-fact when folks bring it up I generally say there's a lot we can learn from libvirt. GVM/Mdev-GPU isn't LibVF.IO so I'm kind of blown away that you keep bringing that up as if they're the same. GVM/Mdev-GPU controls which types the vendor driver presents to mdevctl - GVM/Mdev-GPU is neither an mdevctl replacement nor does it share code with or even integrate with LibVF.IO. I don't think you really grasp that. GVM/Mdev-GPU enables the creation of arbitrary mdev types against the Nvidia driver as defined by the user. Those types generally have "capabilities" restricted by arbitrary policy and "available mdev types" are stored in an XML file alongside signatures which attempt to stop the user from defining their own types. GVM/Mdev-GPU gives control back to the user. Also our codebase is GPLv2 and we don't enforce any kind of "device-specific" restrictions. Nvidia's equivalent is closed source and intentionally locks out consumer cards.
>I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language the...
It is. You configure a YAML/JSON file to define the mdev types that are presented by the vendor driver in the standard sysfsdev mdev_bus path. Those appear precisely the same way any mdev type appears to any program which uses the VFIO-Mdev API. In other words you can install GVM/Mdev-GPU underneath Libvirt or talk to it using mdevctl.
> You create mediated devices for nVidia devices by sending ioctls exposed via RMAPI. This is potato/potato. The explanation you just gave STILL makes it sound as if this is a novel thing done by GVM/mdev-gpu rather than something common, and talking down to someone who is asking informed questions about why you did it this way by linking to internals (when I was physically there for most of those talks and helped write some of the docs) doesn't paint a pretty picture.
This isn't potato/potato. All of these functions are artificially locked out on closed source programs that gatekeep the ability to use your own GPU. No, we're not making it sound like something special. All of our documentation that you seem to think linking to amounts to "talking down to you" because of your position in industry references common virtualization APIs in the kernel and drivers. We link to sample code on how to talk to those APIs. If anything we're showing people this stuff isn't magic! That's the whole point of doing this as open source and making it work on everything we can rather than making it closed source and locking it out to certain devices then charging you for the use of your own hardware.
> NONE OF THE STUFF I'M ASKING ABOUT IS DOCUMENTED OR SPELLED OUT. That's the point. From someone who was a maintainer, engineering leader, etc on a major open source virtualization platform who literally wrote code which does this kind of scheduling/creation across a cluster
I'm glad you have no problem gatekeeping in your comments or propping yourself up based on your job history. Many folks have offered constructive criticism as well as guidance on where to place our attention next many of whom I strongly suspect are at least as experienced as you (those folks who notably don't feel the need to often repeat how senior they are) but thanks again for the grandstanding so plainly.
>I am telling you that your documentation is opaque, misleading, takes credit for things you did not invent
No it doesn't. I constantly link to everything we're talking about on OpenMdev and to all the original sources, talks, mailing list posts, sample code. Please find any example you like of inappropriately taking credit for things and I'll happily correct it.
> where that "missing middle" is /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen"), etc.
This comment makes no sense. /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen" has nothing to do with GVM/Mdev-GPU.
> This is, or could be, a great start to a unified ecosystem. You are going to have a very hard time getting a developer/user ecosystem if you do not provide better documentation,
I agree we need to improve our documentation a lot still yet. It's open for community contribution. You obviously have some idea of what you're talking about enough that we can have this argument so as much as this has dragged on for an uncomfortable amount of time and I still think you have several fundamental misconceptions about what GVM/Mdev-GPU actually does it would be nice if I could structure this interaction with you into something I could use to condense what you're saying here into documentation improvements rather than a flame war in a web forum comment section.
> and most of all, to acknowledge the work other have done/the knowledge they have rather than presenti...
GVM/Mdev-GPU depends on the Nvidia driver in this release. I think that's pretty clear but if you're not aware the page this thread is about does reference OpenRM driver interactions.
Not that this thread is about LibVF.IO which you've devoted most of your attention to throughout this discussion but we do give credit to Looking Glass, vGPU_Unlock, AMD GPU-IOV Module, and Intel i915 in the LibVF.IO install guide - they are discussed as part of the install process where they're mentioned explicitly by name. We also link them at the bottom of the article: https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...
https://github.com/intel/gvt-linux/blob/gvt-staging/drivers/...
Checkout their commit for that over here:
https://github.com/intel/linux-intel-lts/commit/41ef979f0894...
The full list of supported GPUs is available over on our wiki:
https://openmdev.io/index.php/GPU_Support
why even bother?
I think we have decent coverage of device support: https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw...
We'd like to improve support for AMD devices but there are some issues there that have yet to be resolved.