Anyone using it with Kubernetes and Ceph in production? What are the advantages to other open source? What's worse in Proxmox than in Hyper-V or VMSphere?
LXD is similar to proxmox if you only need containers, and pretty slick. They use their own distributed version of sqlite to propagate state, so it's much simpler than proxmox.
It does not come with a GUI, though there are a few third party ones. It has API endpoints for everything, a fully featured CLI client, and is extremely stable.
Googles marketing department does not want you to know this, but Ganeti runs most of its internal infrastructure (not public facing stuff).
Interesting...they don't seem to market it well, I hadn't heard of it. Looks like it started in 2007, where Proxmox started in 2008. It doesn't seem to support containers though. Just Xen and KVM.
Indeed the similarity with Ganeti is only on the surface: being a scalable infrastructure management tool, DRBD as a first-class supported disk backend, etc.
The cluster replication and consensus model is very different: Ganeti has a single "master" node that makes all the decisions and replicates cluster state to other nodes.
Proxmox is awesome - VMs and containers with an API that just works. Network file system with a UI that just works.
We once built an internal system to make network solution demos in it - a VM image running Proxmox that would be configured with any number of other VMs and containers all inside of that image, so you could literally take the while demo environment with you on a USB stick when you presented it, work on it while on a plane etc.
I also built a mini cloud on a Proxmox cluster using Ceph that would instantiate these images from a template using a web UI - a bit of Proxmox recursion fun.
I love Proxmox but the absence of incremental backups is very annoying. If you have a large vm whose contents don't change much your options are limited as storage (edited: and bandwidth) is not infinite.
I also don't want to use aryufan patches to enable incremental backup. I feel like they could probably partner with duplicacy to ship that feature. Duplicacy already released a paid version specifically designed to backup VMware hosts so it should be possible to adapt it to backup Proxmox hosts.
(Thank you all for your comments ! They gave me some ideas)
I've been wondering about the very same problem (and stumbled across aryufan's patches as well), and discovered that they're now actually working on an expanded backup solution of their own:
Use a CoW/snapshotting backing file system for your backup (btrfs/zfs), or one of the modern de-duplicating backup systems (bup/borg/restic), and the space taken will be proportional to the changed data, while still providing random-access to any snapshot -- Not much more you can expect in terms of space usage.
snapshotting filesystems will be much more efficient in terms of time/effort compared to de-duplicating later, but other than that, all of these solutions are mostly equivalent.
I don't argue that it would work but when you do that you're still sending all the data over some data link.
For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution
That's not at all true for bup and borg, almost certainly not true for restic (Haven't used it myself and can't tell for sure).
bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average;
You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead.
> I don't argue that it would work but when you do that you're still sending all the data over some data link.
That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
Yes, I mentioned that you need a file system that can do that (afaik, zfs, btrfs on Linux, and hammer2 on dragonfly) Or that you’ll need to read it all.
I've been using proxmox in my homelab for ~2y, and was doing everything bare metal for the ~3y prior to that (which is when I started my homelab).
At this point, I'd never use anything else. Proxmox "just works" in the best sense of the phrase. Never had uptime issues or anything (knock on wood). The only bare-metal install at this point is my plex server, which has a special level of importance in my close and extended family, so I baby it a little bit. A lotta bit.
I use it for a lot of things - a windows VM for running the mega client (so it's always on + isn't on a personal machine to snoop around), a postgres server that I use for the dev/qa versions of my personal stuff, a jenkins+wekan server, a vm just for minecraft servers, etc etc. It's so easy to just spin up a new VM when I need a clean slate for something random.
If I at some point actually start my own business, we'd be a proxmox shop for any non-cloud infra. And the cost-effectiveness of even a handful of 1-gen-old Dell servers on-prem (for things that don't need super high availability) is really hard to overstate when your budget isn't attached to VC funding. I would not run prod infra on-prem at this point, though: DigitalOcean makes the cloud far too affordable for that. But I'd probably have all CI, data analysis, etc etc local for a good long while.
One gripe I have is their subscription/license nag. It's annoying and tacky. I would gripe less if they had an option that made it feasible for me, a guy running it in his basement with 0 profit motive, to buy a license. Right now, it'd be (~$85 * 2 sockets * 4 servers) for me. Sorry, that's not happening, I could actually use DigitalOcean cheaper than that (and maybe even AWS). And it would be a huge % of my house's tech-related OpEx (compare that to ~80/yr for domains, $5/mo for my one droplet, $100/mo for internet, etc). If I had to name a price, I'd say $25/yr would be about the most I'd drop (as a guy with a basement server room, of course). Contrast this with Plex which I got a lifetime license for for like 150 bucks or something, and plex doesn't even really have an alternative. I could live without proxmox (libvirt is really annoying, but not that annoying), but I couldn't live without plex.
The pricing seems fine for businesses, though. I'd argue it's probably good RoI.
I really like Proxmox, except for one glaring issue: dealign with storage. It is all command line for anything else but local storage. Trying to set up iSCSI can be frustrating. I hope 7.0 adds that feature.
I used Proxmox in production on a decently sized ecommerce site, and it worked well otherwise.
33 comments
[ 2.3 ms ] story [ 78.6 ms ] threadI don't think there is another notable, proven, open source clustered manager of both actual VMs and containers. OpenNebula might be the closest: https://opennebula.io/get-your-hands-on-v-5-8-edge/
LXD is similar to proxmox if you only need containers, and pretty slick. They use their own distributed version of sqlite to propagate state, so it's much simpler than proxmox.
Googles Ganeti is strikingly similar to Proxmox, to the point where I suspect Proxmox started as a reimplementation:
http://www.ganeti.org/ (developed at https://github.com/ganeti/ganeti)
It does not come with a GUI, though there are a few third party ones. It has API endpoints for everything, a fully featured CLI client, and is extremely stable.
Googles marketing department does not want you to know this, but Ganeti runs most of its internal infrastructure (not public facing stuff).
No affiliation, just a happy user.
Thanks for checking the dates, I thought Proxmox was newer.
The basis of Proxmox VE is the real time replicated clustered configuration file system: https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html
I do not see anything similar for ganeti, also not much similar in general besides that it manages VMs?
Indeed the similarity with Ganeti is only on the surface: being a scalable infrastructure management tool, DRBD as a first-class supported disk backend, etc.
The cluster replication and consensus model is very different: Ganeti has a single "master" node that makes all the decisions and replicates cluster state to other nodes.
We once built an internal system to make network solution demos in it - a VM image running Proxmox that would be configured with any number of other VMs and containers all inside of that image, so you could literally take the while demo environment with you on a USB stick when you presented it, work on it while on a plane etc.
I also built a mini cloud on a Proxmox cluster using Ceph that would instantiate these images from a template using a web UI - a bit of Proxmox recursion fun.
That was a cool project to work on. :)
I also don't want to use aryufan patches to enable incremental backup. I feel like they could probably partner with duplicacy to ship that feature. Duplicacy already released a paid version specifically designed to backup VMware hosts so it should be possible to adapt it to backup Proxmox hosts.
(Thank you all for your comments ! They gave me some ideas)
https://git.proxmox.com/?p=proxmox-backup.git;a=tree
https://forum.proxmox.com/threads/proxmox-backup-server-and-...
snapshotting filesystems will be much more efficient in terms of time/effort compared to de-duplicating later, but other than that, all of these solutions are mostly equivalent.
For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution
bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average;
You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead.
> I don't argue that it would work but when you do that you're still sending all the data over some data link.
That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
At this point, I'd never use anything else. Proxmox "just works" in the best sense of the phrase. Never had uptime issues or anything (knock on wood). The only bare-metal install at this point is my plex server, which has a special level of importance in my close and extended family, so I baby it a little bit. A lotta bit.
I use it for a lot of things - a windows VM for running the mega client (so it's always on + isn't on a personal machine to snoop around), a postgres server that I use for the dev/qa versions of my personal stuff, a jenkins+wekan server, a vm just for minecraft servers, etc etc. It's so easy to just spin up a new VM when I need a clean slate for something random.
If I at some point actually start my own business, we'd be a proxmox shop for any non-cloud infra. And the cost-effectiveness of even a handful of 1-gen-old Dell servers on-prem (for things that don't need super high availability) is really hard to overstate when your budget isn't attached to VC funding. I would not run prod infra on-prem at this point, though: DigitalOcean makes the cloud far too affordable for that. But I'd probably have all CI, data analysis, etc etc local for a good long while.
One gripe I have is their subscription/license nag. It's annoying and tacky. I would gripe less if they had an option that made it feasible for me, a guy running it in his basement with 0 profit motive, to buy a license. Right now, it'd be (~$85 * 2 sockets * 4 servers) for me. Sorry, that's not happening, I could actually use DigitalOcean cheaper than that (and maybe even AWS). And it would be a huge % of my house's tech-related OpEx (compare that to ~80/yr for domains, $5/mo for my one droplet, $100/mo for internet, etc). If I had to name a price, I'd say $25/yr would be about the most I'd drop (as a guy with a basement server room, of course). Contrast this with Plex which I got a lifetime license for for like 150 bucks or something, and plex doesn't even really have an alternative. I could live without proxmox (libvirt is really annoying, but not that annoying), but I couldn't live without plex.
The pricing seems fine for businesses, though. I'd argue it's probably good RoI.
root@pve01:~# cd /etc/cron.daily/
root@pve01:/etc/cron.daily# cat proxmox-nosub
#!/bin/sh sed -i "s|if (data.status !== 'Active')|if (false \&\& data.status !== 'Active')|g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
I put it in daily in case I forget to run it during an update.
I used Proxmox in production on a decently sized ecommerce site, and it worked well otherwise.