What do you all use for quick one-command (non gui) new vm bringup?
I haven't had much luck with terraform with libvirt, and I miss Vagrant but don't want to lock myself into its syntax. Maybe I should just start using it again but I feel like there has got to be a good libvirt-based way.
Provisioning new test vms with userdata embedded into an kickstart-or-whatever iso that is provided to libvirt also seems clunky. docker-machine is no longer maintained so I can't use it as a vagrant replacement to just bring up sshable hosts. Multipass?
I have some extremely clunky 50 line shell scripts around virt-install presently but was hoping to find something a little more streamlined (like the usual Vagrant or docker-machine workflow).
Do you just manually run a huge virt-install command each time?
I was playing around with this a while ago, using cloud-init and virt-install. But I found it to be quite cumbersome if you don't have some kind of metadata service that can provide cloud-init with the user-data, ie. not using the nocloud provider.
That's the one I've had suboptimal experience with. Maybe I need to learn it better. What kind of storage are you using it with? I ran into issues using file-backed trying to keep everything in tf.
I’m copying qcow2 volumes with it to file based storage and some Ceph RBD as well, I mean AFAIK it just talks to the standard libvirt volumes interface so anything you can do with that you can do with the Terraform provider.
vagrant with the vagrant-libvirt plugin. The roboxes project provides pre-installed and configured vagrant boxes for allmost all distributions. Its really just a matter of seconds to bring up whatever distribution box you like. Even pre-configured and sysprepped windows images exist.
I use qemu and never really understood the need for any further abstraction such as libvirt over it's cli usage. Can anyone tell me what there is to be gained other than needing further setup, packages and systemd services?
I am not an expert but libvirt is not only for qemu but it supports various virtualization technologies such as Xen, LXC, VMware. So if you decide to stick with QEMU then I guess you don't need to use libvirt.
It's not just about abstraction. VirtIO is significantly more efficient than SATA emulation, and as it is built into the Linux kernel it just works. There is also a Windows driver package that adds VirtIO support, but it's a bit tricky to get it to work when porting in an existing Windows VM. VirtIO also makes it possible to do USB relay.
VirtIO is very usable via QEMU, without libvirt (naturally, because in the configuration described in the article, libvirt is just calling QEMU). It is usually as simple as `qemu-system-x86_64 -drive file=/path/to/my/disk,if=virtio`.
There's more cool stuff coming in this area too. For a long time there's been the virtio family of protocols for shuttling IO to something outside QEMU to handle. Originally that was always KVM and the implementation is called vhost. Then later it became clear that these same messages could be sent to another user space process to handle instead (called vhost-user). These work great for creating virtio devices in the guest. But operating systems like Windows don't have virtio device drivers in-box, so it's a little annoying.
Recently, a new protocol to replace virtio has been defined. It is modeled on vfio ioctls and currently only can forward to another user space process, so we're calling it vfio-user. With this protocol, it's possible to emulate any PCI device rather than only virtio devices. Projects like SPDK (what I work on) can now use this to present fully emulated NVMe devices into guests and back them with whatever actual storage is available (a file, something over the network, a real NVMe SSD, etc). This allows an OS, including Windows, to boot from the virtual disk using it's in-box NVMe driver. This hasn't quite made it into a QEMU release yet, but it's close!
I'm like you and use the qemu command line. But libvirt does make it easier to define cpu groups, thread affinities and the like and without needing to be root to do so. Something like this. There are other ways to accomplish this I think without libvirt, but it gets a little hairy.
When I started using virt-manager I started there because I was specifically looking to replace Virtual box due to having kernel modules for VB sometimes block kernel upgrades. So I guess I was looking for a GUI since I was on a desktop anyways. (Just have some dev VMs so I don't have to clutter my desktop with running MySQL etc for some projects and so I can match the distro of the production server)
So I guess, it worked, didn't look into raw qemu. I also use it infrequently enough that I would have to relearn the CLI all the time.
It's been probably a decade since I've used libvirt, but I can say why I like using ganeti:
- I can get a list of the host machines in my cluster and how much memory and storage they have available.
- I can easily move VMs between hosts if I want to evacuate a host for hardware/software/firmware maintenance.
- It has the ability to set up DRBD backed VMs and live migrate between the host nodes.
- List what machines are running and on what hosts.
- Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.
> - Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.
Shouldn't be the case. I put my VM args into a script, but it's only like 20 lines?
Probably my favorite feature of libvirt is the security and isolation features provided by sVirt. It applies a security policy via SELinux (or AppArmor) that ensures that in the event of a VM breakout exploit, the attacker can only access resources allocated to that VM. So it isolates VMs from the host and from each other. Really cool!
Browse with JS disabled by default. Then in uBlock Origin you can temporarily whitelist the page and browse it with JS enabled if it really requires it.
One thing is missing comparing to virtualbox, is the bridge-mode especially when host uses wireless which is very common these days.
virtualbox hacked bridge-mode and made it working well all the time, with libvirt/qemu/kvm you will need write a lot of scripts and click around to set this up.
This is the sole reason I had to stick with virtualbox these years.
This is the main virtualbox feature I sorely missed running kvm. Is there any easy way to configure bridge mode on kvm? I didn't dig too much because I also run docker and kubernetes on the same machine and I don't want to break their network config accidentally.
If anyone knows how to setup the bridge with wireless in libvirt/qemu/kvm following the steps with/without ebtables listed in Debian and Arch wiki, please do tell
I was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive dedicated to putting ISOs on, in addition to the OS install and the virtual machine storage. I was trying to set it up on a SFF machine and was limited in what I could put into it and I just kind of noped out of it. I couldn't install any VMs until I did this.
I don't recall this being an issue before.
I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up.
In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.
I snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.)
Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup.
8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox .
I went over to Untangle website for any data sheet but was unable to get one immediately (have to fork over everything but your youngest child, so to speak in order to get it.
I don’t want to sidetrack this in to an Untangle conversation, but yes: it’s fundamentally not free as in speech while also being supported as though it’s just a thin shell over Debian.
However, the YouTube breakdown by Battle(non)sense was conclusive (and is confirmed in my case in undeniable ways) so I switched.
(I think) I ran into the same or similar problem you did - there's a partition for the debian OS + iso images of a fixed size (df shows it as /dev/mapper/pce-root -> / ) and all the reallocatable storage used by the VMs comes out of the rest.
Thing is it's hard to re-allocate this partition, and that's where all the iso images, and the backups go.
I tried to migrate a vm and boom - it filled up.
I tried to figure out how to reallocate it, but in the end, I found this (sort of hacky) solution documented:
you migrate the stuff from /var/lib/vz to the new location and you have 200G for iso images/etc
Why this has to fight for storage from the root system, or why it's not easily expandable is a head-scratcher. I think it might be better to have a dedicated boot debian partition, a resizable/easily growable pve partition, then storage that can be allocated to vms/containers.
but yeah - that stopped me in my tracks, and when I'm doing it on a home machine you just don't have the time to chase this stuff down.
If you install Proxmox VE the next time and are not happy with the default layout, check out the parameters for LVM [0]. I personally prefer to install on ZFS as all ZFS datasets share the same free space unlike with LVM.
If you want full control of the disk layout and maybe want to do some more custom stuff, consider installing a base Debian and then install Proxmox VE on top of it [1].
The PVE installer makes it easy to get a system up and running quickly but is opinionated what the disk layout should be. That of course might not be working for everyone.
libvirt has too many gotchas to be as easy to use as virtualbox, vmware or proxmox. Proxmox also uses qemu+kvm, but is opinionated enough to pick the most appropriate option by default.
Example? For Networking, you probably want the virtio driver for maximum throughput if it is supported on the guest (even Windows has drivers now), using the emulated realtek device/e1000 is very slow. That means you should probably also use the virtio block devices, right? Nope, wrong, work on that was abandoned years ago, use SATA!
One annoyance I had with the defaults are the short dhcp lease time configured for dnsmasq, causing repeated dhcp lease logging in the logs. Turns out you can configure the lease time using virsh net-edit. The libvrit docs contain additional useful configuration options.
Another small thing but big quality of life improvement is exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that everything you execute uses the correct connection.
Do I guess correctly that you have cockpit running on the host and manage guests via the cockpit-machines plugin? If so, do you have any hints on where to find documentation about that? A cursory glance at the repo didn't produce much information.
I've been struggling to run Windows VMs on Linux for a while. Are any of these tools a good option for that? I've installed a Windows VM more times than I can count and it hangs after 20-30 minutes of waiting. If anyone has a suggestion for an all in one tool that can help with this, I'm all ears.
63 comments
[ 2.8 ms ] story [ 120 ms ] threadI haven't had much luck with terraform with libvirt, and I miss Vagrant but don't want to lock myself into its syntax. Maybe I should just start using it again but I feel like there has got to be a good libvirt-based way.
Provisioning new test vms with userdata embedded into an kickstart-or-whatever iso that is provided to libvirt also seems clunky. docker-machine is no longer maintained so I can't use it as a vagrant replacement to just bring up sshable hosts. Multipass?
Curious as to how others have solved this.
Do you just manually run a huge virt-install command each time?
https://github.com/weaveworks/ignite
I started looking at this mock EC2 metadata service, but never did anything with it: https://github.com/sjjf/md_server
Ninja-edit: It seems like virt-install rencently abstracted the whole NoCloud-provider, this changes a lot of things: https://blog.wikichoon.com/2020/09/virt-install-cloud-init.h...
The tricky part is intercepting the requests and redirecting them to your service
Other than a working Proxmox install, you'll need to create a cloud-init template, which is documented here: https://pve.proxmox.com/wiki/Cloud-Init_Support
https://multipass.run/
(For others, it's a simple-to-use CLI that provisions Ubuntu instances.)
virt-install -n centos -r 1024 --cpu=host -l http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ --disk path=/var/lib/libvirt/images/centos.img,size=10,bus=virtio --boot kernel_args="console=/dev/ttyS0" -w network=internal,model=virtio --initrd-inject=centos.ks -x "ks=file:/centos.ks console=ttyS0" --graphics none
Setting up the serial console requires some configuration on the host part, too, unfortunately, depending on the OS.
As for what it “gives”, I suppose I’ve never migrated a VM outside of libvirt, though I’m not 100% sure if that’s not possible with plain old qemu+kvm
https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u... (fixed now)
Recently, a new protocol to replace virtio has been defined. It is modeled on vfio ioctls and currently only can forward to another user space process, so we're calling it vfio-user. With this protocol, it's possible to emulate any PCI device rather than only virtio devices. Projects like SPDK (what I work on) can now use this to present fully emulated NVMe devices into guests and back them with whatever actual storage is available (a file, something over the network, a real NVMe SSD, etc). This allows an OS, including Windows, to boot from the virtual disk using it's in-box NVMe driver. This hasn't quite made it into a QEMU release yet, but it's close!
https://www.reddit.com/r/VFIO/comments/9iaj7x/smt_not_suppor...
So I guess, it worked, didn't look into raw qemu. I also use it infrequently enough that I would have to relearn the CLI all the time.
- I can get a list of the host machines in my cluster and how much memory and storage they have available.
- I can easily move VMs between hosts if I want to evacuate a host for hardware/software/firmware maintenance.
- It has the ability to set up DRBD backed VMs and live migrate between the host nodes.
- List what machines are running and on what hosts.
- Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.
Shouldn't be the case. I put my VM args into a script, but it's only like 20 lines?
qemu-system-x86_64 -enable-kvm -name hostname.domain -m 4096 -smp 2 -pidfile /var/run/ganeti/kvm-hypervisor/pid/hostname.domain-device virtio-balloon -daemonize -D /var/log/ganeti/kvm/hostname.domain.com.log -machine pc-i440fx-bionic -monitor unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.monitor,server,nowait -serial unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.serial,server,nowait -usb -usbdevicetablet -vnc 127.0.0.1:5156 -chroot /var/run/ganeti/kvm-hypervisor/chroot/hostname.domain.com -uuid 3d9ea1c9-33ce-4e06-81d3-a97005b89e4b -netdev type=tap,id=nic-19f31fbd-f5de-4b57,fd=9 -device virtio-net-pci,bus=pci.0,id=nic-19f31fbd-f5de-4b57,addr=0xd,netdev=nic-19f31fbd-f5de-4b57,mac=aa:00:00:XX:XX:XX -netdev type=tap,id=nic-0e7f0c3a-e19b-4ae2,fd=12 -device virtio-net-pci,bus=pci.0,id=nic-0e7f0c3a-e19b-4ae2,addr=0xe,netdev=nic-0e7f0c3a-e19b-4ae2,mac=aa:00:00:XX:XX:XX -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.qmp,server,nowait -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.kvmd,server,nowait -bootc -device virtio-blk-pci,bus=pci.0,id=disk-066d688d-9f5a-4ca5,addr=0xc,drive=disk-066d688d-9f5a-4ca5 -drivefile=/var/run/ganeti/instance-disks/hostname.domain.com:0,format=raw,if=none,id=disk-066d688d-9f5a-4ca5 -S -runas kvm37
In production you probably want the host-vm and vm-vm confinement provided by svirt https://libvirt.org/drvqemu.html#driver-security-architectur...
It makes my firewall pop up constantly.
virtualbox hacked bridge-mode and made it working well all the time, with libvirt/qemu/kvm you will need write a lot of scripts and click around to set this up.
This is the sole reason I had to stick with virtualbox these years.
- Simple OpenGL accelerated guest graphics (alternatives: intel gvt-g, card passthrough, virgl in qemu (not generally available), SPICE+qxl in qemu (not nearly as performant))
- bridged networking "just works"
- open-vm-tools enable quite an interesting array of comfortable guest integration
- disk image format can be handled by qemu-img for conversion purposes, so it's easy to migrate vmware <> virtualbox <> qemu+kvm.
The graphics part is why for desktop VMs, I prefer the vmware solution at the moment.
1. https://wiki.archlinux.org/index.php/QEMU#Network_sharing_be...
2. https://wiki.archlinux.org/index.php/Network_bridge#Wireless...
3. https://wiki.debian.org/BridgeNetworkConnections#Bridging_wi...
[1]: https://lxd.readthedocs.io/en/latest/virtual-machines/
It’s good to see what this article details into it and how some can pull all this together … for free (or for a small sum).
URL: https://proxmox.com/en/proxmox-ve/features
I don't recall this being an issue before.
I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up.
In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.
Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup.
8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox .
runs very quiet.
so now it’s my home datacenter.
One thing I had to do recently though is swap pfsense for Untangle because of drastically better FQ-CoDel implementation in Untangle.
Not sure why would one even do that.
However, the YouTube breakdown by Battle(non)sense was conclusive (and is confirmed in my case in undeniable ways) so I switched.
Thing is it's hard to re-allocate this partition, and that's where all the iso images, and the backups go.
I tried to migrate a vm and boom - it filled up.
I tried to figure out how to reallocate it, but in the end, I found this (sort of hacky) solution documented:
https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)
basically:
you migrate the stuff from /var/lib/vz to the new location and you have 200G for iso images/etcWhy this has to fight for storage from the root system, or why it's not easily expandable is a head-scratcher. I think it might be better to have a dedicated boot debian partition, a resizable/easily growable pve partition, then storage that can be allocated to vms/containers.
but yeah - that stopped me in my tracks, and when I'm doing it on a home machine you just don't have the time to chase this stuff down.
Made more sense to have a second drive or second but largest LVM to play with and store all things VM.
So confident that I disabled the Proxmox 'local' drive.
If you want full control of the disk layout and maybe want to do some more custom stuff, consider installing a base Debian and then install Proxmox VE on top of it [1].
The PVE installer makes it easy to get a system up and running quickly but is opinionated what the disk layout should be. That of course might not be working for everyone.
[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanc... [1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_insta...
Example? For Networking, you probably want the virtio driver for maximum throughput if it is supported on the guest (even Windows has drivers now), using the emulated realtek device/e1000 is very slow. That means you should probably also use the virtio block devices, right? Nope, wrong, work on that was abandoned years ago, use SATA!
However, I am not an expert and would love to learn more. Intuitively though I wouldn't expect emulated SATA to be better than paravirt of any kind.
Another small thing but big quality of life improvement is exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that everything you execute uses the correct connection.
https://cockpit-project.org/
On debian add backports then it's as simple as "apt-get install cockpit-machines" then point your browser to http://localhost:9090
- using Vagrant with the Libvirt plugin. Then there's vagrant-mutate for converting Virtualbox boxes to KVM images.
- VM snapshots with virsh