Proxmox is pretty cool, but it doesn't make sense unless you're running many machines and utilizing ceph. For a home user, Debian (or some other stable distro) is still going to be the best distro for doing hypervisor stuff.
I run it on a single node to run LXC guests for all sorts of non-production workloads. That it takes care of all the networking, storage, OS templates, backups, KVM access and has a comprehensive web UI is all very useful to me.
What else is there to take its place? oVirt can't do LXC.
virt-manager can do lxc. I rdesktop to the server and set up things with virt-manager.
You can also install pve on a debian system if you want the proxmox gui. If you want to do full disk encryption, that's the recommended approach, because proxmox's installer doesn't support it.
I'd disagree entirely.
If you want an extremely easy to use gui that gives you access to KVM and LXC virtualisations, along with easy access web remote consoles, snapshots and backups, then Proxmox has been a fantastic go-to for over 6 years.
My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.
> Proxmox is pretty cool, but it doesn't make sense unless you're running many machines and utilizing ceph. For a home user, Debian (or some other stable distro) is still going to be the best distro for doing hypervisor stuff.
> My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.
> If you want an extremely easy to use gui that gives you access to KVM and LXC virtualisations, along with easy access web remote consoles, snapshots and backups
My understanding is that alpine is a very light weight linux ditro and isn't a productised, easy to use hypervisor with full control via a gui.
It's a bit like me saying "I like music editing using xxx because it has a tonne of value added, makes it easy, but it requires an amount of hardware resource" and someone replying "pah, I just use a reel-to-reel, some sharp scissors and a roll of tape!" - unless I'm missing something, it's not really comparable.
edit: as tfigment pointed out along side my comment, I'm slightly annoyed that proxmox is a full-fat linux install for the hypervisor. The comparison I tried to make was with vmware ESXi which boots from an SD card or USB stick and is almost entirely held in memory after boot.
ESXi has a whole number of other issues attached to it, but the fact I can throw a £10 usb stick in a server and use that as a boot drive, rather than "wasting" two of my SATA/SAS bays to a pair of real disks with the hypervisor installed on is nice.
They mean proxmox requires full debian. Pretty sure its not a choice. In VMs you can run anything qemu runs. Also no Docker. The devs are opinionated, not that that is necessarily an issue.
Docker has been on the wishlist for a while but it seems like the developers refuse to integrate it. I just installed docker-ce manually and added portainer as a systemd service.
That way I can manage containers with a 2nd webui.
I've been wanting to do something like that, but I was afraid that the combination of LXC + Docker would blow up due to networking or cgroups or something.
Are you running Docker and LXC containers side by side?
What are your thoughts on this setup vs just docker+portainer on regular Ubuntu LTS, or even a dedicated Docker distro like Rancher?
I had a previous home server built around docker, and found it fiddly to get right— most of the services I wanted to deploy didn't already have containers, and there was a lot of outer config stuff to play with, getting the containers to restart properly on reboot, getting DNS to resolve them, getting the right data directories set up as binds.
But that was also in 2016, so it's likely things are a lot better now, and conversely, it's not necessarily clear that a VM-based approach would be a lot better in some of these areas.
Yeah, proxmox is debian. I think people don't realize you can install their fancy web UI on debian.
Proxmox is opinionated about how storage is allocated (I think lvm-thin is a bad idea), doesn't support docker, and has some quirks about getting the apt repos set up correctly if you're not a paying subscriber. It's fine if you want to use things in their opinionated way, but know that they're not doing any magic to make KVM work. KVM and LXC are the exact same as you'd get from debian out of the box.
Mostly agree but proxmox still has quite a few quirks. Its cloud-init support is difficult to use unless you just need hostname and single user and single NIC which does cover a lot of cases but not mine. Its web api is non-standard and cannot upload full cloud-init config files but requires files already on disk. Also no Docker, just LxC. I found ESXi free tier better for test lab for somethings like backup and snapshots but no creating vms from scratch via api was an issue which is why I used Proxmox in first place.
But you don't get that great UI. I'm the world's biggest Linux cheerleader; one thing you never hear is that almost every 'modern' Linux has a hypervisor built-in (KVM/QEMU) to it.
PVE is basically Debian with a wonderful web-UI wrapped around KVM/QEMU. You don't really need it, but it sure makes life easy!
It makes sense to use Proxmox at home or in a business on a lot of planes. Ceph usage is not required, and it comes with excellent ZFS support. There are a lot of other great features which Proxmox facilitates.
Yes. It's nice for LXC and Linux VMs and use cases that fit its feature set, but not nearly as versatile and integrated as commercial hypervisors.
Most enterprise users would eventually have to also have - or leave in place - an (enterprise) virtualization platform. For others, it may or may not work.
You can see it as Debian, with a web gui for managing virtualized resources. Ceph is not required, but it is something nice to have if you have many hypervisors and no central storage.
But it is meant for using the machine mainly as an hypervisor, not as a desktop with the ability to ocasionally run a few virtual machines (in that case kvm+virt_manager or virtualbox are more desktop friendly).
I installed Proxmox at a previous job, doing snapshots in seconds on a database container was nice for backups and the UI makes you feel informed and in control
I can't say enough good things about PVE! I started managing VMware in 2003 and ESXi is a wonderful product. I was running ESXi at home on the free license which only gives you eight vCPUs (if I remember correctly). After hitting that limit, I slowly decided to move to PVE. I was able to migrate my machines quite easily and have never looked back. For so many use cases, PVE is just as good as ESXi. If you're looking to virtualize and have relatively straightforward requirements, it's worth a look. Free is great and it's rock solid. I made that switch three years ago and haven't had a single issue.
Can't say enough about how I appreciate Proxmox. I had tried Xenserver but it didn't play well with my setup. Proxmox is robust from GUI and CLI. I run two small businesses from my home and the setup allows me to efficiently host pFsense, a mail server, multiple web and DB servers.
But even better for me, the ease and ability to pass through a GPU and run Windows in a VM while using modifiable resources makes it so flexible.
I've been a customer for over two years and they provide support with a super reasonable yearly subscription of just $99. The forum is active and the updates are recent and up to speed.
I'm looking through the CLI docs now and I can't even figure out how to launch a VM. At face value, it doesn't look like a good CLI (well actually they list a bunch of CLI tools for VM management). Am I just looking in the wrong place?
I also want the ability to run setup commands (think AWS user-data) at first boot. The docs indicate user-data support though cloud-init, but I see forum discussions that indicate that you can only set up basic things like IP addresses, SSH keys, etc. Is there a programmatic way to inject bash commands at first boot?
You might want to consider ganeti. It works great from the command-line. There is a GUI for it as a separate project, but I've never used it. I've been running ganeti in dev/stg for 6-7 years and production for around 4, and it's been extremely solid.
As far as running bash commands at first boot, in ganeti I have customized their "create" script to include a bash script that installs some packages, clones our Ansible setup, and then runs Ansible to set the machine up. We respin our entire staging cluster every other day (even machines one day, odd the next).
do you know if you can pass through raw disks as well? I.e. i see that it supports ZFS could i attach that pool within a vm and not directly to proxmox?
I have done PCI passthroughs and USB passthroughs. One of my PCI passthroughs is a MegaRaid card. Not sure how you are physically connecting the pool, but if the pool is running on dedicated hardware, there's a good chance you could passthrough the hardware...very good if PCI or USB based.
Proxmox's hosts controller does not require being on ZFS itself...you can have multiple drives and filesystems. If it were available to Proxmox and not passed through, it could be allocated only to a particular VM. In case that was your only end goal..
The Hades Canyon NUC actually has 2 RJ45 ports: https://simplynuc.co.uk/hades-canyon/
But it does cost you some serious money. I have a small i3 NUC and would rather plugin a USB3.0 to ethernet adapter.
Back several years ago I worked on a project to build a system for making demos of network solutions. We ran Proxmox inside of an ESX virtual machine and then used it to virtualise various pieces of the demo architecture. The ESX image could simply be coped to an external drive for the consultant to use.
Ultimately the company went for a central-lab demo environment instead but it was pretty cool anyway.
The demo itself was stored as a bunch of Ansible playbooks and assorted scrips so it was Git-friendly, to be applied to a generic ESX template.
We built cool things like a canvas-based Visio-like UI that could be used to define VMs / containers / networks and provision at the touch of a button.
Similarly the solution itself would be visualised using a diagram that supported dynamic components like links showing traffic, counters, graphs etc that would show traffic running between the VMs inside the demo.
We also had a central server that would store pre-created demo templates using Ceph that could checked-out for use via a web UI.
> We built cool things like a canvas-based Visio-like UI that could be used to define VMs / containers / networks and provision at the touch of a button. Similarly the solution itself would be visualised using a diagram that supported dynamic components like links showing traffic, counters, graphs etc that would show traffic running between the VMs inside the demo.
Nifty! What frontend/backend dev stack did you use? Does anything like that exist in OSS, for any virt platform?
I had a brief stint with proxmox a few weeks ago and didn't like it much. I didn't like the nag popup when you don't have a subscription (and I had installed the nosubscription package!). I migrated to it to see if it would be easier to manage than OpenStack, but I honestly found OpenStack easier to deal with, probably due to familiarity.
Yeah that nag is my only nitpick with Proxmox. It's a bit too much in the face everytime I login. You can easily disable it with a code snippet, but I think the devs should just advertise the non-subscription model differently.
I've been running proxmox since v3.0 with lots of success. I admit that back then you could break stuff easily if you did not know what you're doing. Over the time I've been running it continuously with every version in my homelab and it just got more awesome. I tried ESXI and Microsoft Server.. But in the end I always went back to Proxmox and the very stable debian that runs it. Now with the latest version ZFS encryption is also native and I have pretty much everything I could ask for. Can only recommend!
Last year I had applied for a job where we would have to manage Windows servers. Stuff with Active Directory and storage... I'm Linux guy but have extended experience in WinServer 2012. I just set up a server, boot ProxMox, spun up as many as Win Server 2016 instances I wanted to experiment with all the aspects of the new Win Server and guess what? I was second in the job interview. Thank you ProxMox. You is the best.
Proxmox is good with the large exception of high speed interconnects.
If I have Mellanox IB cards in my servers, proxmox fails to handle ipoib without a lot of legwork. Compare that to something like oVirt; that supports it out of the box.
There is very little incentive for me to recommend a proxmox subscription to any of my clients because having >= 40 gbit interconnects is far better than using lags on single gbit. High traffic internal applications, (and migration!) benefit so much from those interconnects.
I have run dual 100gb mellanox in it with no problems without the IPoIB. It seems a pretty specific problem that is probably related to the mellanox cards and not to proxmox, as similar bugs show up with mellanox in oVirt.
I've used Proxmox on my dedicated host and it was working really well, however one problem I've encountered is that it doesn't really play well with guest OpenBSD (guest randomly was hanging, with no apparent pattern). But Linux and Windows worked perfectly.
I agree, FreeBSD (FreeNAS) doesn’t play well with it and iSCSI did not work well according to a lot of forum postings and blogs. I wanted a robust NAS and doing it through a VM did not provide the peace of mind I was looking.
The only upside is now I can build a dedicated visualization machine or grab some old enterprise hardware from Craigslist. My wife is not so amused.
Probably one of the more notable current applications that still use a lot of Perl. The admin interface is mostly Perl, as are many of the command line utilities.
I gave proxmox a solid try for virtualizing a mattermost instance. But it was just way too slow compared to bare metal. Chats took seconds to load instead of ms. I'm sure there's some explanation for it but I couldn't be assed to figure it out and reverted.
I'm currently using Yunohost, but I'm finding the system a bit brittle for some use-cases, and some choices rather arbitrary (which was also the point, as I initially chose it to spend less time managing my server).
I would like something a bit more fleximble, and was seriously thinking about jumping to Guix or Nix. Does any of you have any input on this? Proxmox is also a candidate, as I could run my old Yunohost side-by-side with Nix, Guix, and others, I guess. I'm just afraid of overcomplexifying my nginx config.
Allso, if I run Guix/Nix on Proxmox, I'll lose the declarative config advantage... Is the opposite possible?
I am sure usecases are slightly different for these two. At least for me personally, "plug and play" offer was the key - so I could easily move stuff around and not think about it. And it has browser access so non-tech people can use email/Nextcloud/etc.
But perhaps Yunohost can be used to model what you have in mind, and once solidified, use it as a base to build VM with precise setup.
Ansible is perhaps the closest you can get to declarative config... See "sovereign" on github - I migrated from it to Yunohost, as things kept breaking and I was looking hands off setup (hence Yunohost), but given that Sovereign is ansible-based - perhaps you can leverage it.
One thing I've never understood with Proxmox is why networking can't be configured host-side. It seems you are expected to go into each new VM and manually set network settings. Why not have a DHCP server on the host that hands out prescribed addresses by VM id?
If Proxmox could do this in addition to maintaining port forwarding to these VMs I would be happy.
I've set up Proxmox a couple times. I do exactly what you described, using dnsmasq. There are a few blog posts out there, this is one [1] I used for inspiration.
I too wish it was a built-in Proxmox feature, or at least officially documented...
Nobody said it's wrong. It's perfectly OK. Proxmox is precisely a nice web UI for that and a lot of stuff integrated, such as LXC, CEPH, HA, Live Migration...
Would be cool to see an addon for NAS services in Proxmox... I'm setting up a home server when I get the chance and just want it to run a VM, a handful of docker containers and provide NAS storage at home. In the end, will probably add Docker support, run the NAS in a container or vm and leave it at that.
I run a Synology full of spinning rust for backups, archives, and shared file storage. It works reliably and is stupid simple to maintain.
Adjacent to that is a server running Proxmox, and I'm currently working to add Kubernetes/Rancher as a layer on top of that. The Proxmox server has a local SSD array so the VMs have fast low-latency storage. VMs that touch larger files (media server, etc) have an NFS mount to the NAS.
IMHO the financial/maintenance overhead of a second unit is totally outweighed by the easy opportunity to have two performance and reliability tiers (fast homelab vs reliable slow storage), ability to upgrade/change the two units independently of each other, and ability for cohabitants to use the NAS without having to understand/deal with the virtualization stuff.
I have an older Synology NAS, it's really slow, and TBH need more compute for a few things at home... but don't want to spend on yet another box for this. The main role of the new server is NAS stuff, but if I can do a few other things in the background, so much the better.
I'll be running proxmox and VMs off of an nvme drive in a 4x pcie adapter, and the NAS drives using the hotswap bays in the front.. it's a used R710 (iirc), and has 6 3.5" bays (came with 2TB drives, but replacing the raid controller with one that will support larger drives and upgrading to 8TB as time/cost permits).
I could totally drop the home server compute requirement before the NAS requirement as my desktop could handle those chores.
TBH FreeNAS is a great NAS/SAN solution. I'm running a 4x4 RAIDZ2 (8TB effective) with a 10 year old quad core AMD and 16GB DDR2 RAM.
If you are okay with the limited VM hosting capabilities of Freenas (headless VMs and containers on the same network as the file server) then you can use it for that as well.
77 comments
[ 3.1 ms ] story [ 147 ms ] threadWhat else is there to take its place? oVirt can't do LXC.
You can also install pve on a debian system if you want the proxmox gui. If you want to do full disk encryption, that's the recommended approach, because proxmox's installer doesn't support it.
My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.
> My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.
Why not Alpine Linux? I use that in diskless mode https://wiki.alpinelinux.org/wiki/Installation#diskless_mode
My understanding is that alpine is a very light weight linux ditro and isn't a productised, easy to use hypervisor with full control via a gui.
It's a bit like me saying "I like music editing using xxx because it has a tonne of value added, makes it easy, but it requires an amount of hardware resource" and someone replying "pah, I just use a reel-to-reel, some sharp scissors and a roll of tape!" - unless I'm missing something, it's not really comparable.
edit: as tfigment pointed out along side my comment, I'm slightly annoyed that proxmox is a full-fat linux install for the hypervisor. The comparison I tried to make was with vmware ESXi which boots from an SD card or USB stick and is almost entirely held in memory after boot. ESXi has a whole number of other issues attached to it, but the fact I can throw a £10 usb stick in a server and use that as a boot drive, rather than "wasting" two of my SATA/SAS bays to a pair of real disks with the hypervisor installed on is nice.
I've been wanting to do something like that, but I was afraid that the combination of LXC + Docker would blow up due to networking or cgroups or something.
Are you running Docker and LXC containers side by side?
Any documentation on how you did it?
I had a previous home server built around docker, and found it fiddly to get right— most of the services I wanted to deploy didn't already have containers, and there was a lot of outer config stuff to play with, getting the containers to restart properly on reboot, getting DNS to resolve them, getting the right data directories set up as binds.
But that was also in 2016, so it's likely things are a lot better now, and conversely, it's not necessarily clear that a VM-based approach would be a lot better in some of these areas.
Proxmox is opinionated about how storage is allocated (I think lvm-thin is a bad idea), doesn't support docker, and has some quirks about getting the apt repos set up correctly if you're not a paying subscriber. It's fine if you want to use things in their opinionated way, but know that they're not doing any magic to make KVM work. KVM and LXC are the exact same as you'd get from debian out of the box.
PVE is basically Debian with a wonderful web-UI wrapped around KVM/QEMU. You don't really need it, but it sure makes life easy!
It makes sense to use Proxmox at home or in a business on a lot of planes. Ceph usage is not required, and it comes with excellent ZFS support. There are a lot of other great features which Proxmox facilitates.
Most enterprise users would eventually have to also have - or leave in place - an (enterprise) virtualization platform. For others, it may or may not work.
But it is meant for using the machine mainly as an hypervisor, not as a desktop with the ability to ocasionally run a few virtual machines (in that case kvm+virt_manager or virtualbox are more desktop friendly).
I would recommend it for most businesses honestly
Recently I tried it again when attempting to install Untangle on a system with EFI. Untangle didn’t support EFI so I needed to virtualize it.
I found it too difficult to get software network adapters to work with different physical network cards.
Switched over to VMware ESXi and had vswitch configured within minutes.
YMMV and I’d still recommend Proxmox for simple virtualization projects.
But even better for me, the ease and ability to pass through a GPU and run Windows in a VM while using modifiable resources makes it so flexible.
I've been a customer for over two years and they provide support with a super reasonable yearly subscription of just $99. The forum is active and the updates are recent and up to speed.
But also in the cases when the VMs are never running at the same time. Sometimes the "shop" closes and gaming starts...
I'm looking through the CLI docs now and I can't even figure out how to launch a VM. At face value, it doesn't look like a good CLI (well actually they list a bunch of CLI tools for VM management). Am I just looking in the wrong place?
I also want the ability to run setup commands (think AWS user-data) at first boot. The docs indicate user-data support though cloud-init, but I see forum discussions that indicate that you can only set up basic things like IP addresses, SSH keys, etc. Is there a programmatic way to inject bash commands at first boot?
As far as running bash commands at first boot, in ganeti I have customized their "create" script to include a bash script that installs some packages, clones our Ansible setup, and then runs Ansible to set the machine up. We respin our entire staging cluster every other day (even machines one day, odd the next).
http://www.ganeti.org/
The "qm" page shows:
pct - controls containers
qm - controls vms
Proxmox's hosts controller does not require being on ZFS itself...you can have multiple drives and filesystems. If it were available to Proxmox and not passed through, it could be allocated only to a particular VM. In case that was your only end goal..
Back several years ago I worked on a project to build a system for making demos of network solutions. We ran Proxmox inside of an ESX virtual machine and then used it to virtualise various pieces of the demo architecture. The ESX image could simply be coped to an external drive for the consultant to use.
Ultimately the company went for a central-lab demo environment instead but it was pretty cool anyway.
The demo itself was stored as a bunch of Ansible playbooks and assorted scrips so it was Git-friendly, to be applied to a generic ESX template.
We built cool things like a canvas-based Visio-like UI that could be used to define VMs / containers / networks and provision at the touch of a button.
Similarly the solution itself would be visualised using a diagram that supported dynamic components like links showing traffic, counters, graphs etc that would show traffic running between the VMs inside the demo.
We also had a central server that would store pre-created demo templates using Ceph that could checked-out for use via a web UI.
Fun times :)
Nifty! What frontend/backend dev stack did you use? Does anything like that exist in OSS, for any virt platform?
The canvas diagram was plain JS and for logos we used SVGs from company official PowerPoint icon pack to make it look consistent.
The backend was Node.js that ran Ansible playbooks that provisioned things via Proxmox API and then configured the VMs as needed.
If I have Mellanox IB cards in my servers, proxmox fails to handle ipoib without a lot of legwork. Compare that to something like oVirt; that supports it out of the box.
There is very little incentive for me to recommend a proxmox subscription to any of my clients because having >= 40 gbit interconnects is far better than using lags on single gbit. High traffic internal applications, (and migration!) benefit so much from those interconnects.
The only thing I miss is a truly compatible Terraform module that lets you create accounts, machines and infrastructure. That would be cool
The only upside is now I can build a dedicated visualization machine or grab some old enterprise hardware from Craigslist. My wife is not so amused.
I would like something a bit more fleximble, and was seriously thinking about jumping to Guix or Nix. Does any of you have any input on this? Proxmox is also a candidate, as I could run my old Yunohost side-by-side with Nix, Guix, and others, I guess. I'm just afraid of overcomplexifying my nginx config.
Allso, if I run Guix/Nix on Proxmox, I'll lose the declarative config advantage... Is the opposite possible?
But perhaps Yunohost can be used to model what you have in mind, and once solidified, use it as a base to build VM with precise setup.
Ansible is perhaps the closest you can get to declarative config... See "sovereign" on github - I migrated from it to Yunohost, as things kept breaking and I was looking hands off setup (hence Yunohost), but given that Sovereign is ansible-based - perhaps you can leverage it.
If Proxmox could do this in addition to maintaining port forwarding to these VMs I would be happy.
I too wish it was a built-in Proxmox feature, or at least officially documented...
[1]: https://blog.jenningsga.com/private-network-with-proxmox/
XCP-NG is also a nightmare due to how half baked Xen Orchestra is.
VMWare ESXi is the only type 1 hypervisor that I would ever recommend to a fellow hacker. You can get a free license for personal use.
I run a Synology full of spinning rust for backups, archives, and shared file storage. It works reliably and is stupid simple to maintain.
Adjacent to that is a server running Proxmox, and I'm currently working to add Kubernetes/Rancher as a layer on top of that. The Proxmox server has a local SSD array so the VMs have fast low-latency storage. VMs that touch larger files (media server, etc) have an NFS mount to the NAS.
IMHO the financial/maintenance overhead of a second unit is totally outweighed by the easy opportunity to have two performance and reliability tiers (fast homelab vs reliable slow storage), ability to upgrade/change the two units independently of each other, and ability for cohabitants to use the NAS without having to understand/deal with the virtualization stuff.
I'll be running proxmox and VMs off of an nvme drive in a 4x pcie adapter, and the NAS drives using the hotswap bays in the front.. it's a used R710 (iirc), and has 6 3.5" bays (came with 2TB drives, but replacing the raid controller with one that will support larger drives and upgrading to 8TB as time/cost permits).
I could totally drop the home server compute requirement before the NAS requirement as my desktop could handle those chores.
If you are okay with the limited VM hosting capabilities of Freenas (headless VMs and containers on the same network as the file server) then you can use it for that as well.
https://unraid.net/product
downside is it's closed source and not free