VirtualBox KVM Public Release (cyberus-technology.de)
For the past few months we have been working hard to provide a fast, reliable and secure KVM backend for VirtualBox. VirtualBox is a multi-platform Virtual Machine Monitor (VMM) with a great feature set, support for a wide variety of guest operating systems, and a consistent user interface across different host operating systems.
Cyberus Technology’s KVM backend allows VirtualBox to run virtual machines utilizing the Linux KVM hypervisor instead of the custom kernel module used by standard VirtualBox. Today we are announcing the open-source release of our KVM backend for Virtualbox.
204 comments
[ 2.4 ms ] story [ 277 ms ] threadSo, slow as it may be, the win for UML (which seems to still have a heartbeat) is that it can run on uP without any specific virtualisation capabilities, right? If I could run Linux on a Z80/6502 then in theory I could run a virtualised Linux on a Z80/6502.
In the early 2000s people used to use UMLs as a hosting platform - they didn't have the same security isolation as a proper VM (or even, necessarily, of a container) though.
[0] https://en.wikipedia.org/wiki/Cgroups
https://en.m.wikipedia.org/wiki/Kernel-based_Virtual_Machine
https://xkcd.com/1053/
There are two large hypervisors in the Linux world.
Xen, which extends the kernel to support virtual CPUs with time slices.
KVM, which assigns each virtual core a process that uses the Linux scheduler.
When a hardware vm vcpu core is preempted there is vmexit call that has to reset registers etc... and it is expensive.
Xen is what legacy AWS instances ran on and has advantages for being fair to guests is an easier task.
KVM has the advantage of gaining the benefits of the Linux scheduler which is red black tree based and well optimized.
When a new CPU comes out for example, KVM gains support from the upstream while Xen has to support it themselves.
Once technology like cgroups improved the benefits of letting your thread complete and not be preempted due to the time slice expiring avoided the cost of vmexit.
In theory, leveraging the inherently optimized core Linux features is what will also benefit virtualbox.
Most people who use KVM are using an abstraction layer like libvirt that hides how it is implemented.
In fact if you look at the processes you will see qemu even if KVM is how it is implemented.
Bryan Cantrill
https://www.youtube.com/watch?v=-zRN7XLCRhc&t=1980s
Much of the reason I refuse to use Virtualbox for anything is how scummy Oracle is.
FTFY.
Having spent time at another large Oracle customer that was later acquired by Oracle I've seen how they do this internally too. And you can't just send it off to Legal to write back that "we didn't actually use it"...
Guest integration (drag'n'drop, clipboard), USB passhthrough and audio support is also top-notch in VBox.
I haven't found a significative difference but if you have found one and can tune qemu to same level,why don't you share the xml template of your machine to the world and to upstream's virt-manager project?
> Guest integration (drag'n'drop, clipboard), USB passhthrough and audio support is also top-notch in VBox.
These things works well with libvirt too provided you are using the spice-guest-tools.
I'm mostly interested in if I can use virtualbox accelerated video with kvm because virgl3d seems well behind in that area.
I'm not even sure what your alternatives were for this before now, I guess everyone was just using samba.
Most of the VMs are encrypted, so I feel safe traveling with them. Various secrets are also encrypted, but the encryption of the VMs themselves mean that I don't have to worry about losing my device at an airport and someone else potentially getting access to things they shouldn't. There are schemes that make this work in virt-manager and KVM, but I didn't like any of them as much; I didn't want to rely on the host for filesystem-level encryption (see portability), and I have previously had a bit of trouble with full disk encryption, so I wasn't comfortable relying on that. VirtualBox essentially is also doing full disk encryption, but it's invisible to the guest and seems to be reliable.
For portability, I should be able to use https://www.vbox.me/ to install the VMs and a host onto a flash drive and be able to run any of my environments from any Windows host without additional installations. Haven't actually tried this yet (happily, I no longer have easy access to Windows machines!), but it was a big point in favor.
Most of my environments now get auto-configured through Vagrant: https://github.com/robsheldon/vagrantfiles, so I get some of the benefits of virt-manager that way.
I really don't love relying on Oracle for anything
I’m sure some people would make the assumption that it’s under the same license as the upstream package but in some environments absolutely clarity around licenses is really appreciated.
Every time I need to run a virtual machine, I choose libvirt because it's more performant and easy to deal with than Virtualbox (no kernel module, etc.), but the GUI choices are pretty terrible. The "best" libvirt GUI is virt-manager and it's very, very buggy and lacking features (i.e. doesn't play nice with HiDPI screens, no way of configuring IPv6, etc.)
Many times I have caved and chosen VirtualBox simply because at least it feels nice to use, even if not as performant as libvirt/kvm. Not anymore!
https://www.cyberus-technology.de/products/hypervisor (Don't mind the English, we are not native speakers. :)
We used to have support for Intel GVT-g GPU virtualization as well, which was more of a software solution. This doesn't work with modern Intel GPUs anymore.
I am surprised the open source community has not built better gui tools, and no project, closed or open has made configuring pcie passthrough easy.
I have always wanted to be able to run Windows in a virtualized session with my GPU for gaming, and use my onboard APU for the Linux host, but the configuration is daunting, and many of the games I play today don’t work on linux thanks to anticheat or DRM.
This is the thing that isn't surprising.
In order to make a better GUI tool, you have to understand how the internals work. Then you don't care to use a GUI tool because you know how to use the command line or edit the configuration files. The people who want better GUI tools are the people who don't know how to make them.
In order to change this, the people who want GUI tools but don't know how to make them have to provide some incentive (typically money) to the people who know how to make them but don't want them, e.g. via donations or some kind of commercial agreement. And if they stand around and wonder why nobody else has fixed their problem for them instead of doing the thing that causes it to be fixed, an object at rest tends to remain at rest.
"GPU passthrough with libvirt qemu kvm" https://wiki.gentoo.org/wiki/GPU_passthrough_with_libvirt_qe...
"PCI passthrough via OVMF" https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF :
> The Open Virtual Machine Firmware (OVMF) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the virtual machine native graphics performance which is useful for graphic-intensive tasks
KVM-GPU-Passthrough: https://github.com/BigAnteater/KVM-GPU-Passthrough
https://clayfreeman.github.io/gpu-passthrough/
Not saying I prefer one or the other, but it's worth bearing in mind where "the bar" is
E.g. virt-manager is built with glade XML and Python:
virt-manager/virt-manager//ui/createvm.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/cr...
virt-manager/virt-manager//ui/gfxdetails.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/gf...
virt-manager/virt-manager//ui/hoststorage.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/ho...
virtManager/createvm.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...
virtManager/device/addstorage.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...
virtManager/device/gfxdetails.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...
virtManager/addhardware.py:
https://github.com/virt-manager/virt-manager/blob/135cf17072... https://github.com/virt-manager/virt-manager/blob/135cf17072...iommu_groups.sh: https://github.com/drewmullen/pci-passthrough-ryzen/blob/mas... :
"PCI passthrough via OVMF > 2. Setting up IOMMU > 2.2 Ensuring that the groups are valid": https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#En... :> An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that any of them could be passed to a virtual machine without affecting the others.
"Exporting your ROM": https://github.com/BigAnteater/KVM-GPU-Passthrough?tab=readm... :
"Attaching the GPU" [with `virsh`] https://clayfreeman.github.io/gpu-passthrough/#attaching-the... : "Adding your GPU and USB devices to the VM" [with `virt-manager`]: https://github.com/BigAnteater/KVM-GPU-Passthrough?tab=readm...> 1. Add every PCI device which has to do with your graphics card to the VM.
It’s basically the opposite for both. I use virt-manager because the GUI is simpler (and setting up virtualbox is a nightmare anyway).
Regardless, this feature is a step in the right direction. I’m wondering if distributions will pick it up or if it will ever be integrated upstream.
I'd be curious to hear the specific reasoning behind it.
virt-manager v4.0.0 (Ubuntu 22.04 amd64)
"QEMU/KVM" > "Details" (via double-click or context menu) > "Virtual Networks" tab > "+" (Add) button > "Create Virtual Network"/"Details" tab > Expand "IPv6 configuration" > check "Enable IPv6" > enter "Network" prefix and/or check "Enable DHCPv6" > enter "Start" and "End" addresses
I'm honestly surprised there aren't more alternatives. Especially since SPICE has gstreamer instegration. Gstreamer can natively do webrtc video and data back channel. Someone could hack together a webrtc broweser-based VM viewer and I suspect there's a lot of interest, potentially money making things you could crate.
KVM also always has the hottest new (performance-relevant) features, because Intel and AMD will always build their hot stuff into KVM first.
If you use virtualbox with this new backend, you can use it concurrently with qemu (and a few other virtualisation tools).
KVM is also part of Linux itself, so there’s a lot less haste with setting it up.
IIRC they also disabled ASLR kernel wide.
Additionally, and perhaps less important: USB3 is a commercial feature of virtualbox, there are stories of companies getting C&D letters (or Audits/Invoices) from Oracle because a developer had installed the virtualbox extensions..
KVM is a userspace API, and kvm-intel/kvm-amd are the drivers for the hardware.
You will be using hardware features. That's also why it is in the kernel: nothing but the kernel should have full unlimited access to the CPU to set this up.
So you could say it must be in kernel to keep the kernel secure. And the performance benefit is "just" exposed hardware features.
The kernel does not provide additional things. As far as I understand: you set up a dedicated memory space and handle traps that halt the execution e.g. when the VM talks to the PCI bus. (It's been a while since I looked this up)
But you need the pieces, especially virtual PCI devices. That's where qemu or VirtualBox enter the scene (or minimalist systems like firecracker). They provide a repository of virtual hardware and all the auxiliary methods to boot a virtual machines. You also need to emulate something like a BIOS or UEFI.
You can think of it as your CPU removing the need to emulate the very same CPU (and a memory controller). You still need to emulate the rest though! But running on the same CPU removes most performance penalties. You run at native speed.
Newer generations can even nest this. Having virtual machines in virtual machines. That's mostly useful for cloud environments so that the cloud provider can run kvm based VMs and you are still able to run VMs inside that VM.
So they went from using hardware virtualization (provided by intel/amd) with their kernel module to the KVM one. I don't know which benefits it brings
[1] https://www.phoronix.com/news/OTk5Mw
in security research, you either run your samples in qemu without even kvm or you don't.
Congrats for the work!
From a performance perspective, it's a bit more complicated. KVM has support for modern virtualization features (Intel APICv, AMD AVIC, etc) that vanilla VBox lacks. You get these in the VirtualBox/KVM version. On the other hand, vanilla VBox emulates most devices in the kernel (see above). So SATA emulation in vanilla VBox is very fast compared to KVM/Qemu or KVM/VirtualBox for a bit unfair reasons. Modern devices, such as virtio or NVMe, are not as impacted by that.
tl;dr So the performance you get depends on your workload. If it's very interrupt heavy, VirtualBox/KVM will win. If it uses antiquated virtual devices (SATA), vanilla VirtualBox (with vboxdrv) will have an edge.
EDIT: That was a joke, but actually it is a thing https://www.youtube.com/watch?v=nTMls33dG8Q
As a pentester, I run use Linux on my laptop and I spend a lot of time working inside a Kali VM with VirtualBox.
How much performance improvement can we expect with the KVM backend ?
Which networking setup do you use for your pentesting VM? Ideally, I'd want a setup where the VM can access the internet (and therefore the HTB VPN), but not anything inside my local network. But I don't quite know how I could achieve that, at least in a way where I'd trust it to be reliable. Maybe the whole idea's a bit too paranoid to be practical in general, I don't know, so I'd love an expert opinion on this :)
Usually, I'd be using QEMU, but I'd be fine with using VirtualBox for this case if it includes something that makes this easier.
https://github.com/xjasonlyu/tun2socks
Thank you :)
Anyway, apart from this possibly impractical usecase, it just seems like an interesting problem.
I'll just go with a standard bridge + ip/nftables setup then. Thanks for the input.
Then you can be more confident that (barring something escaping the VM), you have control of what a particular VM may or may not access. Also allows you to have a couple of different netvms (maybe one does give some access to your internal network, one connects only over tor, one via some VPN...) that you can switch between without needing to change anything on the actual VM of concern.
How does this work in licensing terms? If VB foss enough?
Do you expect Oracle to merge this?
If oracle doesn’t merge this, will you keep on maintaining it, potentially forking VirtualBox?
It's as FOSS as the VirtualBox open source edition.
> Do you expect Oracle to merge this?
That would be nice, but I wouldn't hold my breath. Oracle gonna Oracle.
> If oracle doesn’t merge this, will you keep on maintaining it, potentially forking VirtualBox?
We don't intend to fork VirtualBox. VBox has a somewhat modular architecture where you can plug-in different hypervisor backends. That's what we did. It's not as modular, but our changes to core VirtualBox code is very small.
As far as our plans go, we are pretty open at this point. We are very interested to get to know people that find this useful!
I built and tried this, it worked great, so excellent work there. I found the processor CPU core counts being grayed out unpleasant, it's not clear to me how I am supposed to adjust my core and ram values now, maybe document this?
`VBoxManage modifyvm <vm_name> --cpus <number of cpus>`
`VBoxManage modifyvm <vm_name> --memory <amout of memory in MB>`
You can sill select these PV interfaces with VirtualBox-KVM, but the underlying hypervisor is different (kvm vs vboxdrv).
This new code is about using different virtualization technology on the host system.
AFAIK VirtualBox does not support PCI passthrough (like GPU), how is this case handled?
Since I've moved to QEMU/KVM on Linux I've never looked back at VirtualBox, but I use the latter on Windows and there I'm always remembered of how much nicer and friendlier the GUI is.
Edit: I just noticed that VirtualBox has experimental PCI passthrough via the extension package [0], could this be used with the KVM backend?
[0] https://docs.oracle.com/en/virtualization/virtualbox/6.0/adm...