Having standardized components like these is pretty valuable and reusable.
One could definitely write them on their own.
It’s also another way to discover other packages in these categories.
Some of the LXC container settings can be a little specific and these scripts manage those we’ll most of the time with a standard install, or an advanced one is available too.
It’s very myopic to not be able to imagine others are not like you, and that they may be better and wanting to help create beginners more easily.
The “I can do it in x” period of time reviews are baffling to me too.
I like to be able to rely on smart folks who have fought scripts like these into being polished, so I can built on top of them, even though I have plenty of experience to build these myself. It makes no sense unless it becomes much more important to me.
I see software as a form of literacy - and I think this is a good point of view for your issue here.
The bash commands and virtualisation host are words or perhaps "cliches" - short pieces of literate work that are available to everyone.
And yes you don't have to remember them any more than you need to know every word in the OED.
But people can then string those words together to make their own unique literate work - to set up your virtualisation host in the right way. It's possible to do that in some fill in the variables way but it is waaaay better for you to shape what you need.
Anyway, that's my take. Yes it's sensible to put the weird ass params to X in a single script and forget about it, I came across one of my own not touched for 18 months. But then I was able to read it, remember the important bits and adjust it.
100% agree. I have been running my own NAS the hard way: FreeBSD, ZFS, manual samba/NFS shares, dhcp+gateway+firewall, PXE Boot, Jellyfin (can't remember if it's in a jail or not). All this stuff is a huge, utter PITA and never works quite right.
- I have to manually check the zpool periodically to see if there are faults. I went about 30d with a DEGRADED disk before I realized it, turns out it was a bad cable but if another disk failed I would have been hosed.
- I have to manually run zpool scrub (yeah I know about cron but chances are if I just add the cron entry it will fail, probably some permission thing or a 1-character bug in the cmd line. Nothing like this is ever easy, it's a 1-hour rodeo, at minimum. And yeah, I'm lazy too, not gonna deny it)
- I can only get NFS v3 working, not v4, no idea why. IIRC v4 isn't technically supported by either FreeBSD yet. But rather than just say "not supported" it gives you some cryptic bs that you waste minimum 20m on googling and trying things. Everything you try, you have to triple check: restart rpciod? Or was it nfsd? exportfs -ra? Or was it some encryption version in /etc/exports?
- Samba kinda works, no idea how the permissions are supposed to work, but turns out SMB is more resilient to network interrupts than NFS which just hangs the client indefinitely. So for listening to music or reading a book its fine.
- I wanted to set up VPN on a 2nd network (not VLAN, just a 2nd subnet that routes out the tun device. I thought since I control DHCP, I could force certain devices to only ever have access via VPN), turns out this isn't possible or "very hard" doing some hack to the NAT pf rules, way above my skill level. I wasted hours on this, got the VPN and tun to authenticate, but got stuck at the routing layer. Oh also I think dhcpd doesn't like routing to different adapters? Can't remember.
- PXE was a disaster, as it usually was. I got it barely working enough to boot my tape backup computer off it and run backup scripts.
- I want simple logfile aggregation. I dont want to have to ssh into the NAS and check the logs/dmesg. I admit I haven't done a lot of research (10 years ago there was nagios, thats all I know), but it bugs me profusely that I cannot easily monitor the logs of all the machines on my network, and I rarely ever hear about anyone else doing it.
so probably 40% success, 60% (frustration|half-assery|disappointment). I guess it's just masochism, like the weightlifters who love the pain of adding 10 extra pounds to their set. At least they get some results eventually. :/
Overall, I don't regret doing it the hard way, but I am often times envious of the Proxmox users who (presumably) just plug and play.
I did all this with TrueNAS and nothing ever worked right so I just bought a Synology and I haven’t had issues since.
There is real benefit I think in the manual approach, you’ll learn so much. But at some point I just want to watch stuff on TV or want my Mom in another state to watch too.
> We have a complexity crisis, and few people see it
I’m not saying memorize this stuff - put it in a big proxmox.readme.md file. Then you can incorporate the actual command into your script. Otherwise you end up incorporating this wrapper into other stuff, ad infinitum, and that’s how you get a complexity crisis.
Do you know how to join copper pipes? The difference between PEX and PVC? The various pipe fittings and their benefits/costs/tradeoffs? Do you know how high a pipe can rise before you need to add additional pumping?
I'm pretty sure you turn the faucet on, and expect drinkable water to come out, without thinking of what's going on between the source of the water and your sink.
Yet, anyone, given the right motivation, can use and learn all those pieces and parts, and make use of them at whatever competency level they have. You can even make a soccer goal net with PVC.
There is room (and need) in the world for all people to explore, geek out, and use these details. And the people who are passionate about these details, and the intermediate pieces should be cherished, because they are the ones that will understand and fix the intermediary pieces when they break.
I've been a "Just Work" person for a long time, but over the last couple years I've shifted to DIYing more instead of buying turn-key products. I replaced my off-the-shelf NAS with a custom TrueNAS box, and swapped my router+firewall+switch+WiFi all-in-one with a low-energy device running pfSense and a dedicated WiFi AP.
I have been pleasantly surprised to find that, after the initial research & setup, things still "just work!" In some cases, more reliably (and way more quietly!) than before.
I've had the same experience in reverse. I'm a tinkerer and I always have been. I bought a server expecting to tinker with it all the time. What fun!
But no, after the first couple hours of tinkering it Just Works.
This past week I settled in for a good solid weekend of getting a gitlab server stood up. I was looking forward to it! Imagine my disappointment when it was a single command that took half an hour to run.
Oh well, I guess it just means more time for my other hobbies.
I think the balance is closer to 80% shiny, 20% working code.
There's definitely some cool stuff here (disabling the nag screen, thank you!) but I really dislike the way it's presented.
I don't like the curl|bash very much, but mostly I think this would be far more interesting as a collection of one-liners to accomplish each task vs the current form of a curl|bash with dialogs and prompts. I think that's better for security and better for learning you should know what commands you're executing!
I really wish proxmox had sort of first-class docker/podman integration.
With docker it seems a Dockerfile is a self-contained recipe. You build it, then you run the container.
with proxmox, you sort of have to be a sysadmin. The proxmox UI helps you define the characteristics of the container, but then you have to put everything inside the container yourself without help or reproducibility.
If anyone is looking at proxmox, running these scripts from the host will speed up experimenting.
It’s great to have something up and running in minutes to only redo it slightly differently.
If you are considering two proxmox hosts ensure the second one added to the cluster is an empty proxmox host, and the proxmox cluster from the first box will assign unique id’s to all containers on any host.
Also, for homelabbing, there's a magic incantation that allows you to configure Proxmox away from the default High Availability cluster mode.
By default, the HA mode insists that all nodes are up or others won't come up, making this tweak (it may be contained in the OP scripts, I don't recall) allows you to retain many of the benefits of having Proxmox machines "connected", without requiring you treat them like a HA cluster.
In my case, I have a node I shutdown when it's not used, I use it for specific occasional tasks, and then I have nodes I want up all of the time.
With the non-HA tweaks, you can still do things like centrally manage, migrate VMs and containers between nodes etc, without the limitations of it wanting them all up and available all of the time.
Would this work for having proxmox modes across multiple datacenters? IIRC it was sensitive to latency
I don't really use the HA features of proxmox that often, mostly just for migrating vms across hosts. But I had been thinking of re-installing some of my remote servers with proxmox and hooking them up to the same cluster so that I could potentially shut off all of the servers in my basement and run everything from a remote location, then migrate the vms back later.
I've not noticed anything latency-sensitivty, perhaps the Ceph feature? I'm not using that though.
There's also the issue of quorum of it's only a 2-node cluster, though I've personally not had any issues with it, despite repeatedly restarting the different nodes.
I see no reason why you couldn't do this to allow you to move your VMs from home to elsewhere and back again as you just mentioned.
Its not just one script that you're curling and execing. That scripts curls and execs some other scripts. And those others. You're pretty much auditing a whole codebase at that point, for better or worse.
Slightly off topic: I’m setting up a home server on a Mini PC that has Windows 11 Pro pre-installed. I want to attach it to my TV and play retro games as well as run home automation tasks (periodic scripts, UniFi controller, Pihole etc)
Is anyone using Proxmox on their homelabs? Would you recommend blowing away Windows and installing Proxmox and then install Windows with PCIE passthrough?
Maybe running a firewall on dedicated hardware so the internet doesn't drop if you reboot the hypervisor.. but even then I just live with that and run pfSense in proxmox.
Excellent point! I do run my main PiHole in a VM, but I have a second PiHole running on an RPi just in case I'm rebooting the hypervisor. DNS is the only real dependency that can cause problems for me when the hypervisor reboots for updates. I'm just running Fedora on a somewhat recent Dell rack server.
Firewall is a great example of what not to run on a vm, at least for me! I consider gateways as appliances though, and I haven't run my own router on Linux since I first got Cable internet in 2002. I remember how awesome it was compared to the weak routers available at the time!
I have run proxmox for several years, rely on it for many bits of house and network infrastructure, and recommend proxmox overall. My desktop also runs proxmox with PCIe passthrough for my "actual" desktop (but this is a different proxmox server from the primary VM and container host for infrastructure).
That said, I wouldn't mix the two use cases either initially nor over the long-term. House/network infrastructure should be on a more stable host than the retro-game console connected to your TV (IMO).
In your case, I'd recommend buying another PC (even an ancient Haswell would be fine to start) and getting experience with vanilla proxmox usage there before jumping straight into trying to run infra and MAME/retro gaming on PCIe passthrough on the same singleton box.
I did this for a while where I ran multiple VMs, some of which had PCIE passthrough for some GPUs on both Windows and Linux. Luckily my motherboard separated out IOMMU groupings to make this work for me. While you _could_ do this, you may run into issues if your IOMMU groups aren't separated enough. The biggest issue I always had was drivers always causing issues with Windows. I eventually blew the entire instance away and just run Windows.
I'd recommend a separate device if you need any access to a GPU. But I do recommend Proxmox as a homelab. I still have it running on a separate 2012 Mac Mini.
You can, but it disconnects it from the host, so you'll be headless. Which may be fine for a lot of people if you are able to ssh in and manage it that way.
This is exactly how I have mine set up aye. Proxmox on hardware, Main PC as a VM - once I got to the point where Proxmox had its web interface (and ssh) up and running I had no real reason to have a monitor plugged into the hardware OS anyway. Passed the GPU and all USB ports to the PC guest from there.
At the time I had no idea how popular it was to run this setup, I thought I was being all weird and experimental. Was surprised how smoothly everything ran (and still runs, a year and change later!)
Bonus, I was able to just move the PC to another disk when the SSD it was on was getting a bit full. Moved PC's storage onto a spinny HDD to make room to shuffle some other stuff around, then moved it to another SSD. Didn't even need to reboot the PC VM, haha.
Proxmox backup server running on my NAS handles deduplicated backups for it and other VMs too which is great.
I’m in the middle of this. Got a Bee Link miniPC. It came with windows, licensed oddly. I’m configuring it as a home server. Current plan is to migrate my Unraid install over from the vintage server it’s currently on. Most services are run in docker. We’ll see how performance is.
ProxMox is on my list to try out. So far I’m very happy with Unraid. It makes it easy to set up network shares, find and deploy containerized services, and handles VMs if you need them. I try to avoid the VM and focus on containers because it’s more flexible resource wise.
If the pc is beefy enough (win 11 pro runs smoothly) just go with the included hyper-v. Imho you don’t get any benefits installing proxmox on bare metal in this scenario. YMMV of course
My use-case is slightly different, but I use Proxmox for my home server and would recommend it. Especially if you're familiar with linux systems or want to learn about them which I've done through the years I've been using this setup.
My server was originally a single debian installation set up to host local services for things like git. That grew into hosting a site, vpn, then some multiplayer game servers. When I reached a point where too many things were installed on single machine, I looked at vm options. I've used VMWare/VSphere professionally, but settled on Proxmox for these main reasons: easy to set up and update, easy to build/copy vms, simple way to split physical resources, monitoring of each vm, and simple backup and restores. All without any weird licensing.
That server houses 4 vms right now. That might be a bit much for your mini pc but you could do a couple. The multiplayer servers are the main hog so I isolate resources for that. The windows machine is only for development which isn't your exact use case. I can say however that I've never had issue when I need it. Only thing I can't speak for is the need for graphics passthrough.
You want to use Hyper-v, you can use GPU-P(Gpu Partitoning) where hyper-v will pass through the GPU to the VM and share it, it's not some emulated adapter, it's genuinely the real GPU and runs natively and you can share it across multiple VM's and host. Linux has NOTHING that can compete with the feature.
But a Windows user, so I could be completely off base, but isn't GPU-P just VFIO under the hood? I don't know about Proxmox, but this is completely supported by OpenStack and KVM.
No, it uses the hypervisor to implement a scheduler for the GPU between the different vm's and gives them access to it. VFIO on linux requires passing the entire GPU to only a single VM and making it so the host doesn't have access.
Kind of, i'm not to familiar with the enterprise gpu sharing, but I believe that requires hardware support for it on the GPU to split up contexts for the OS. Whereas the GPU-P solution works on any GPU and is vendor agnostic as it's done on the OS/Processor level. It's really cool and I have no idea how they can ship this since it does trample on the enterprise tech pretty substantially. But I belive it's a byproduct of Microsoft wanting to compete on the datacenter level and it gives them a killer feature for Azure, and as a byproduct it entered as a semi undocumented feature into consumer hyper-v.
MIG does need hardware suppoet, and from what I understand really expensive licensing. I don't know if AMD has a parallel technology.
That is pretty cool that it is vendor agnostic. I've found a few docs from a few years ago talking about stuff like it on Linux, but development of it seems to have stopped or just not progressed at all.
But she is they would use it in Azure, I imagine there they are using enterprise GPUs. But in Andheri scale datacenters it definitely sounds like an advantage!
It's probably overkill for what he wants though. A headless Linux server frees up the GPU for a gaming VM, and he can run containers natively for everything else.
If its just a few simple things you list, I might stick with HyperV. If you care about more sophisticated VLAN'ed networking setups I would probably go proxmox. But hardware passthru is a can of worms so understand there will be a tradeoff.
I actually use Proxmox on my main PC. Ryzen 5950x, 64GB RAM, RTX 4070, AMD 6500XT. The two GPU's are each passed to a Windows and Debian VM respectively, and each also gets a USB card passed for convenience. I run half a dozen other VM's off of it hosting various portions of the standard homelab media/automation stacks.
Anecdotally, it's a very effective setup when combined with a solid KVM. I like keeping my main Debian desktop and the hypervisor separate because it keeps me from borking my whole lab with an accidental rm -rf.
It is possible to pass all of a systems GPU's to VM's, using exclusively the web interface/shell for administration, but it can cause some headaches when there are issues unrelated to the system itself. For example, if I lose access to the hypervisor over the network, getting the system back online can be a bit of a PITA as you can no longer just plug it into a screen to update any static network configuration. My current solution to this is enabling DHCP on Proxmox and managing the IP with static mappings at the router level.
There are a few other caveats to passing all of the GPU's that I could detail further, but as a low impact setup (like running emulators on a TV) its should work fairly well. I have also found that Proxmox plays well with mini PC's. Besides the desktop, I run it on an Intel NUC as well as a Topton mini PC with a bunch of high-speed NICS as a router. I cluster them without enabling the high availability features in order to unify the control plane for the three systems into one interface. It all comes together into a pretty slick system
Can someone explain why it is useful to do do virtualization at all when you just want to run a small amount of things like this?
I have a ubuntu server install running on an old laptop to do very basic background jobs, backups, automation, run some containers etc. – am I missing something by not using a hypervisor? What are the benefits?
Some software really prefers to control the whole host, usually highly integrated stuff. Some examples:
- Unifi Controller installs like a half dozen dependencies to run (Mongo, Redis, etc last time I used it), much easier to isolate all that in a VM
- Home Assistant's preferable and most blessed install method is Home Assistant OS, which is an entire distribution. I've run HA in Docker myself before but the experience is like 10x better if you just let it control the OS itself
- I have Plex,Sonarr,Radarr, etc running for media - there is software called Saltbox which integrates all of these things for you so that you don't need to configure 10 different things. Makes it a breeze, but requires a specific version of Ubuntu or you're in unsupported territory (kinda defeating the purpose)
Lots of stuff you can be totally fine just using Docker or installing directly onto the host. But having the bare metal system running Proxmox from that start gives you a ton of flexibility to handle other scenarios.
Worst case you just setup a single VM & run your stuff on it if you have no need for other types of installs. Nothing lost, but you gain flexibility in the future as well as easy backups via snapshotting, etc
Easy backups via snapshotting is quick to say, but has an outsized benefit IME. My go-to approach for keeping many of my machines up to date is now scheduled apt get update; apt get upgrade and relying on scheduled backups in the unlikely event that goes awry. I don't have to worry about package interdependencies across machines.
For major upgrades, I may go a step further and do a manual snapshot before upgrading and then decide whether or not to commit (usually) or rollback (easy, when needed).
The (emotional) security provided by this is nice, as is the time-savings (after initial time expense to learn and setup the base proxmox infrastructure).
HomeAssistant also does some voodoo with Bluetooth, wifi, ipv6 and mDns for IOT devices. For this reason it seems best suited to a host machine instead of a docker container.
Something like HomeAssistant, it would be good if it had an agent to handle all the low-level networking stuff that could be run directly on the host, and then all the other stuff which doesn’t directly require that can run inside an unprivileged Docker container.
Me either, although I am using it for my home. It does allow configuring VMs, containers, logical volumes, and such. I think you can have multiple proxmox in a cluster. I might not have reached the point where I am doing enough with it to understand the benefit.
For me who uses it as a NAS/VM platform, it’s basically Debian with some virtualization relevant packages and an opt in kernel that are more reasonably up to date, with solid ZFS support out of the box so I don't care about the "update kernel, reboot, oops version mismatch I need to revert" nonsense.
While things like TrueNAS exist with ZFS support, in the past iX systems has made some overeager patches to ZFS, to bring in new features that weren't appropriate to include yet IMO. I hate to dig up old news, but that's why I've avoided them so far.
Basically I've got Proxmox "figured out", which lets me fuck around with the things I want, rather than the things I don’t want. This reduces my very subjective mental overhead.
All the high availability stuff is wasted on me though, and I just disable the related services.
Note to new users experimenting with proxmox
The most common "gotcha" with proxmox, is that it does NOT automatically adjust network stuff after install.
If you switch out a network card or change what physical port it's connected with, you will lose network access, and need to manually edit `/etc/network/interfaces` to make sure your using a correct port ID and/or assigned IP are valid.
iface vmbr0 inet static
...
bridge-ports <The port ID, like "enp130s0f0">
...
> While things like TrueNAS exist with ZFS support, in the past iX systems has made some overeager patches to ZFS, to bring in new features that weren't appropriate to include yet IMO. I hate to dig up old news, but that's why I've avoided them so far.
I've also not enjoyed reading their forums at all, while information about Proxmox tends to be actually helpful rather than berating people over their hardware choice or spreading misinformation about ZFS.
It's a product instead of a box of parts. My brain only has so much cognitive power. If I'm spinning up a VM, I'm trying to accomplish some other work.
I was in a similar boat and finally got around to trying out Proxmox in the last year or so. I just wanted share my own experience here, since it's a bit different than the "standard" uses I've seen.
I had been running a k3s cluster (k8s flavor) on some Raspberry Pis, but decided I needed some non-ARM nodes, and "invested" in a few low power "1L" AMD64 PCs (6-8 core + hyper-threading). I was initially going to just install Ubuntu and base my setup off my existing Ansible automation to make things less inefficient. But I figured I'd play around with Proxmox first and see if there was any benefit to using that as a base layer since I'd heard a lot about it.
I'm so glad I did. I ended up learning quite a bit in the process. Some quick highlights about using Proxmox for VMs in general:
* Proxmox supports creating a "cluster", so you can login though one machine to administer them all. You can conveniently "move" VMs between machines pretty seamlessly.
* If you install the para-virtualization drivers for e.g. Windows or Ubuntu VMs, you can do pretty fast remote KVM. E.g. I could run Youtube on a Windows VM in my basement over "Spice" and it almost looks like it's running locally. (Not that it's a use case I care about, mostly just shows the fact it's performant.)
In terms of actually getting around to deploying k3s on top of the infra:
* I ended up learning HPC-Packer, and HPC-Terraform, which integrated nicely with my existing Ansible experience.
* Packer turns an Ubuntu ISO + my "base" Ansible setup playbooks into a pre-baked machine template directly in Proxmox. (My local machine's Packer binary just orchestrates the process.)
* Terraform deploys the machine template into the Proxmox cluster. Basically a config file of machine names + IPs + mac adresses, and a few other params and initial setup.
* Ansible then installs any final dependencies (anything not in the base template), setups up the first k3s master, grabs the join token, and adds in 2 more master nodes for a proper `etcd` backend.
* Ansible then installs my base Kubernetes services (cert-manager, Rancher, Longhron storage, etc) via running helm commands on one of the nodes.
* This is where I'm at now; the next step is for me to deploy my existing Flux.cd-automated "Gitops" apps (built for ARM64+AMD64 via Gitlab runner, also in Proxmox). These _had_ been running on my now-quite-crusty-seeming Pi cluster.
I can run a single command to delete all the VMs, and rebuild + setup everything (full HA cluster + apps deployed and running) from scratch in ~6 minutes without any manual input required from me, just a few secrets/params in a config file.
This has made exploring the horizon of possibilities _so much easier_ without getting locked in; I can try to weird Longhorn storage configs, or try out k8s monitoring stacks without worrying about needing to "back out" my changes if I picked bad settings. (Just blow it up and try again!) I can change how VLANs are configured in early steps, or try adding a library to the base Ubuntu install cluster-wide super easily, etc.
I am primarily a software-engineer, so it has been really nice to delve into the operational side of things, and get a proper reproducible setup. It really has transformed how I think about the cluster in that it's no longer a "thing to carefully maintain", but instead a great sandbox to explore AND deploy my own k8s applications on top of without playing cloud bills.
My Proxmox journey in the past few months definitely turned into more than a rabbit hole than I'd expected.
Could I please ask you to share some of your Terraform files? I'm looking to build a similar setup on my Proxmox cluster, and it would really help to have a template to go from!
yeah you can write weird shell script and re-implement hot vm migration (moving a physical machine from one physical host to another without shutting it down) but what's the point, really? you might as well use proxmox.
if you ever actually need to see how it's done you could just see what's proxmox doing under the hood (it's open source after all)
It's a sandbox that lets you play without stepping in the mud or throwing rocks. See my other post in this thread for what happens when you try to do it the hard way.
With proxmox, I click the plus sign, pick an iso image, cpu/mem, and i'm good to go and I have a remote console viewer and tons of other neat mgmt stuff.
In the same amount of time, if I were diving in to DIYing it, I'd probably be about 5 minutes into reading the arch linux wiki page for KVM/qemu.
Wondering the same. Also a libvirt and virsh user here, and pretty happy using commands such as `virsh snapshot-create ...`. I've used VMWare ESXi in the distant past, and from the screenshots it looks like Proxmox is maybe inspired by VMWare? So maybe a more polished and integrated GUI based experience than libvirt? Also looks like Proxmox supports containers in addition to virtual machines. Also I see from other comments that there's some features to migrate VMs between hosts, which would be a much more manual effort with libvirt.
Yeah, that’s basically it, it’s an ESXi style management UI that just makes everything really easy. In our little setup I used libvirt on an old server, Proxmox on the new one, and now I’ll probably shift the old server too to Proxmox due to the clustering (for example, you log onto one machine and can manage them all).
ZFS support that just works is really nice too. They also have a backup solution I’d like to check out but haven’t yet.
I've had like 75% success on throwing XML files at ChatGPT and telling it what I want to do for weird things like passthrough or pinning. It gets me most of the way there. That is not a compelling user interface as is. :)
I've used this and It's been the easiest way to set up proxmox, configure the updates, and add HAOS to it. I have this bookmarked so I can recreate it later.
Although its supported, its not well documented enough to be a good way to learn about cloud-init in my experience. I tried configuring a K3's cluster across three proxmox nodes via cloud-init to get some exposure to it and eventually gave up and just configured them manually
I had the same issue at the beginning, and unfortunately stopped on it for over a year: then I went back and got it and now I cannot think of provisioning a VM by hand anymore
I try to do everything without Ansible: cloud-init is pretty powerful, too, but I am reaching the 64k limit, so sooner then later will go "back" to Ansible.
If you're already familiar with Ansible, what about doing some stuff with cloud-init and then having cloud-init trigger a local ansible run? Then you don't have to worry about running into the limit.
It works, but anytime you have custom networking or more complex cloud-init configs, you'll have to go into "snippets" territory or referring to files in a local filesystem of the host. They don't have API support for making snippets the last time I checked. Where I ran into this was when trying to set up hosts with Terraform on Proxmox VE (which works well itself too).
The best way to handle this (which is really terrible, honestly, but it works) is to make configdrive2 ISOs locally, and upload those using the API. That is, don't give Proxmox a cloud-init snippet, but rather a fully-built cloud-init ISO. This is basically bypassing/reimplementing Proxmox's cloud-init features yourself, which is terrible. But it works really well.
Proxmox is awesome. I use it for my router and a bunch of servers. [1] it is so nice to be able to take a snapshot, update opnsense and go back if something fails.
It's working so well I now have too many vms and need to get a bigger disk.
112 comments
[ 3.4 ms ] story [ 178 ms ] threadAfter poking at a couple of these, they seem like they're 50% shiny packaging and 50% one-liner bash commands.
One could definitely write them on their own.
It’s also another way to discover other packages in these categories.
Some of the LXC container settings can be a little specific and these scripts manage those we’ll most of the time with a standard install, or an advanced one is available too.
The “I can do it in x” period of time reviews are baffling to me too.
I like to be able to rely on smart folks who have fought scripts like these into being polished, so I can built on top of them, even though I have plenty of experience to build these myself. It makes no sense unless it becomes much more important to me.
The bash commands and virtualisation host are words or perhaps "cliches" - short pieces of literate work that are available to everyone.
And yes you don't have to remember them any more than you need to know every word in the OED.
But people can then string those words together to make their own unique literate work - to set up your virtualisation host in the right way. It's possible to do that in some fill in the variables way but it is waaaay better for you to shape what you need.
Anyway, that's my take. Yes it's sensible to put the weird ass params to X in a single script and forget about it, I came across one of my own not touched for 18 months. But then I was able to read it, remember the important bits and adjust it.
I agree with you - I think
- I have to manually check the zpool periodically to see if there are faults. I went about 30d with a DEGRADED disk before I realized it, turns out it was a bad cable but if another disk failed I would have been hosed.
- I have to manually run zpool scrub (yeah I know about cron but chances are if I just add the cron entry it will fail, probably some permission thing or a 1-character bug in the cmd line. Nothing like this is ever easy, it's a 1-hour rodeo, at minimum. And yeah, I'm lazy too, not gonna deny it)
- I can only get NFS v3 working, not v4, no idea why. IIRC v4 isn't technically supported by either FreeBSD yet. But rather than just say "not supported" it gives you some cryptic bs that you waste minimum 20m on googling and trying things. Everything you try, you have to triple check: restart rpciod? Or was it nfsd? exportfs -ra? Or was it some encryption version in /etc/exports?
- Samba kinda works, no idea how the permissions are supposed to work, but turns out SMB is more resilient to network interrupts than NFS which just hangs the client indefinitely. So for listening to music or reading a book its fine.
- I wanted to set up VPN on a 2nd network (not VLAN, just a 2nd subnet that routes out the tun device. I thought since I control DHCP, I could force certain devices to only ever have access via VPN), turns out this isn't possible or "very hard" doing some hack to the NAT pf rules, way above my skill level. I wasted hours on this, got the VPN and tun to authenticate, but got stuck at the routing layer. Oh also I think dhcpd doesn't like routing to different adapters? Can't remember.
- PXE was a disaster, as it usually was. I got it barely working enough to boot my tape backup computer off it and run backup scripts.
- I want simple logfile aggregation. I dont want to have to ssh into the NAS and check the logs/dmesg. I admit I haven't done a lot of research (10 years ago there was nagios, thats all I know), but it bugs me profusely that I cannot easily monitor the logs of all the machines on my network, and I rarely ever hear about anyone else doing it.
so probably 40% success, 60% (frustration|half-assery|disappointment). I guess it's just masochism, like the weightlifters who love the pain of adding 10 extra pounds to their set. At least they get some results eventually. :/
Overall, I don't regret doing it the hard way, but I am often times envious of the Proxmox users who (presumably) just plug and play.
There is real benefit I think in the manual approach, you’ll learn so much. But at some point I just want to watch stuff on TV or want my Mom in another state to watch too.
But that's where the fun is ;)
Skill issue. You lack monitoring. The basic prometheus node exporter and alertmanager would be sufficient:
You can easily alert on such metrics.I’m not saying memorize this stuff - put it in a big proxmox.readme.md file. Then you can incorporate the actual command into your script. Otherwise you end up incorporating this wrapper into other stuff, ad infinitum, and that’s how you get a complexity crisis.
I'm pretty sure you turn the faucet on, and expect drinkable water to come out, without thinking of what's going on between the source of the water and your sink.
Yet, anyone, given the right motivation, can use and learn all those pieces and parts, and make use of them at whatever competency level they have. You can even make a soccer goal net with PVC.
There is room (and need) in the world for all people to explore, geek out, and use these details. And the people who are passionate about these details, and the intermediate pieces should be cherished, because they are the ones that will understand and fix the intermediary pieces when they break.
Sometimes, making money is not the goal.
And more broadly, a lot of software could be summarised as a pretty packaging of a basic tool.
If you want things to Just Work, this is probably not for you
I have been pleasantly surprised to find that, after the initial research & setup, things still "just work!" In some cases, more reliably (and way more quietly!) than before.
But no, after the first couple hours of tinkering it Just Works.
This past week I settled in for a good solid weekend of getting a gitlab server stood up. I was looking forward to it! Imagine my disappointment when it was a single command that took half an hour to run.
Oh well, I guess it just means more time for my other hobbies.
There's definitely some cool stuff here (disabling the nag screen, thank you!) but I really dislike the way it's presented.
I don't like the curl|bash very much, but mostly I think this would be far more interesting as a collection of one-liners to accomplish each task vs the current form of a curl|bash with dialogs and prompts. I think that's better for security and better for learning you should know what commands you're executing!
I do a quick look over of the source to make sure it’s ok. It saves me so much time and pain.
With docker it seems a Dockerfile is a self-contained recipe. You build it, then you run the container.
with proxmox, you sort of have to be a sysadmin. The proxmox UI helps you define the characteristics of the container, but then you have to put everything inside the container yourself without help or reproducibility.
It’s great to have something up and running in minutes to only redo it slightly differently.
If you are considering two proxmox hosts ensure the second one added to the cluster is an empty proxmox host, and the proxmox cluster from the first box will assign unique id’s to all containers on any host.
By default, the HA mode insists that all nodes are up or others won't come up, making this tweak (it may be contained in the OP scripts, I don't recall) allows you to retain many of the benefits of having Proxmox machines "connected", without requiring you treat them like a HA cluster.
In my case, I have a node I shutdown when it's not used, I use it for specific occasional tasks, and then I have nodes I want up all of the time.
With the non-HA tweaks, you can still do things like centrally manage, migrate VMs and containers between nodes etc, without the limitations of it wanting them all up and available all of the time.
I don't really use the HA features of proxmox that often, mostly just for migrating vms across hosts. But I had been thinking of re-installing some of my remote servers with proxmox and hooking them up to the same cluster so that I could potentially shut off all of the servers in my basement and run everything from a remote location, then migrate the vms back later.
There's also the issue of quorum of it's only a 2-node cluster, though I've personally not had any issues with it, despite repeatedly restarting the different nodes.
I see no reason why you couldn't do this to allow you to move your VMs from home to elsewhere and back again as you just mentioned.
I thought quorum feature of proxmox was also latency sensitive, but this is the only thing I found:
https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_n...
Maybe this doesn't matter if not using HA? I'll read more about it and maybe test it out
Just my experience though, YMMV.
https://web.archive.org/web/20231128113854/https://www.idont...
Is anyone using Proxmox on their homelabs? Would you recommend blowing away Windows and installing Proxmox and then install Windows with PCIE passthrough?
Firewall is a great example of what not to run on a vm, at least for me! I consider gateways as appliances though, and I haven't run my own router on Linux since I first got Cable internet in 2002. I remember how awesome it was compared to the weak routers available at the time!
Now I just buy a UDM Pro and forget about it!
That said, I wouldn't mix the two use cases either initially nor over the long-term. House/network infrastructure should be on a more stable host than the retro-game console connected to your TV (IMO).
In your case, I'd recommend buying another PC (even an ancient Haswell would be fine to start) and getting experience with vanilla proxmox usage there before jumping straight into trying to run infra and MAME/retro gaming on PCIe passthrough on the same singleton box.
I'd recommend a separate device if you need any access to a GPU. But I do recommend Proxmox as a homelab. I still have it running on a separate 2012 Mac Mini.
But i'd be happy to be proven wrong.
At the time I had no idea how popular it was to run this setup, I thought I was being all weird and experimental. Was surprised how smoothly everything ran (and still runs, a year and change later!)
Bonus, I was able to just move the PC to another disk when the SSD it was on was getting a bit full. Moved PC's storage onto a spinny HDD to make room to shuffle some other stuff around, then moved it to another SSD. Didn't even need to reboot the PC VM, haha.
Proxmox backup server running on my NAS handles deduplicated backups for it and other VMs too which is great.
[1] https://jellyfin.org/docs/general/administration/hardware-ac...
ProxMox is on my list to try out. So far I’m very happy with Unraid. It makes it easy to set up network shares, find and deploy containerized services, and handles VMs if you need them. I try to avoid the VM and focus on containers because it’s more flexible resource wise.
My server was originally a single debian installation set up to host local services for things like git. That grew into hosting a site, vpn, then some multiplayer game servers. When I reached a point where too many things were installed on single machine, I looked at vm options. I've used VMWare/VSphere professionally, but settled on Proxmox for these main reasons: easy to set up and update, easy to build/copy vms, simple way to split physical resources, monitoring of each vm, and simple backup and restores. All without any weird licensing.
That server houses 4 vms right now. That might be a bit much for your mini pc but you could do a couple. The multiplayer servers are the main hog so I isolate resources for that. The windows machine is only for development which isn't your exact use case. I can say however that I've never had issue when I need it. Only thing I can't speak for is the need for graphics passthrough.
MIG does need hardware suppoet, and from what I understand really expensive licensing. I don't know if AMD has a parallel technology.
That is pretty cool that it is vendor agnostic. I've found a few docs from a few years ago talking about stuff like it on Linux, but development of it seems to have stopped or just not progressed at all.
But she is they would use it in Azure, I imagine there they are using enterprise GPUs. But in Andheri scale datacenters it definitely sounds like an advantage!
Anecdotally, it's a very effective setup when combined with a solid KVM. I like keeping my main Debian desktop and the hypervisor separate because it keeps me from borking my whole lab with an accidental rm -rf.
It is possible to pass all of a systems GPU's to VM's, using exclusively the web interface/shell for administration, but it can cause some headaches when there are issues unrelated to the system itself. For example, if I lose access to the hypervisor over the network, getting the system back online can be a bit of a PITA as you can no longer just plug it into a screen to update any static network configuration. My current solution to this is enabling DHCP on Proxmox and managing the IP with static mappings at the router level.
There are a few other caveats to passing all of the GPU's that I could detail further, but as a low impact setup (like running emulators on a TV) its should work fairly well. I have also found that Proxmox plays well with mini PC's. Besides the desktop, I run it on an Intel NUC as well as a Topton mini PC with a bunch of high-speed NICS as a router. I cluster them without enabling the high availability features in order to unify the control plane for the three systems into one interface. It all comes together into a pretty slick system
I have a ubuntu server install running on an old laptop to do very basic background jobs, backups, automation, run some containers etc. – am I missing something by not using a hypervisor? What are the benefits?
- Unifi Controller installs like a half dozen dependencies to run (Mongo, Redis, etc last time I used it), much easier to isolate all that in a VM
- Home Assistant's preferable and most blessed install method is Home Assistant OS, which is an entire distribution. I've run HA in Docker myself before but the experience is like 10x better if you just let it control the OS itself
- I have Plex,Sonarr,Radarr, etc running for media - there is software called Saltbox which integrates all of these things for you so that you don't need to configure 10 different things. Makes it a breeze, but requires a specific version of Ubuntu or you're in unsupported territory (kinda defeating the purpose)
Lots of stuff you can be totally fine just using Docker or installing directly onto the host. But having the bare metal system running Proxmox from that start gives you a ton of flexibility to handle other scenarios.
Worst case you just setup a single VM & run your stuff on it if you have no need for other types of installs. Nothing lost, but you gain flexibility in the future as well as easy backups via snapshotting, etc
For major upgrades, I may go a step further and do a manual snapshot before upgrading and then decide whether or not to commit (usually) or rollback (easy, when needed).
The (emotional) security provided by this is nice, as is the time-savings (after initial time expense to learn and setup the base proxmox infrastructure).
I'm always reading stuff about people using proxmox but don't really understand its 'edge'
While things like TrueNAS exist with ZFS support, in the past iX systems has made some overeager patches to ZFS, to bring in new features that weren't appropriate to include yet IMO. I hate to dig up old news, but that's why I've avoided them so far.
Basically I've got Proxmox "figured out", which lets me fuck around with the things I want, rather than the things I don’t want. This reduces my very subjective mental overhead.
All the high availability stuff is wasted on me though, and I just disable the related services.
Note to new users experimenting with proxmox The most common "gotcha" with proxmox, is that it does NOT automatically adjust network stuff after install. If you switch out a network card or change what physical port it's connected with, you will lose network access, and need to manually edit `/etc/network/interfaces` to make sure your using a correct port ID and/or assigned IP are valid.
I've also not enjoyed reading their forums at all, while information about Proxmox tends to be actually helpful rather than berating people over their hardware choice or spreading misinformation about ZFS.
I had been running a k3s cluster (k8s flavor) on some Raspberry Pis, but decided I needed some non-ARM nodes, and "invested" in a few low power "1L" AMD64 PCs (6-8 core + hyper-threading). I was initially going to just install Ubuntu and base my setup off my existing Ansible automation to make things less inefficient. But I figured I'd play around with Proxmox first and see if there was any benefit to using that as a base layer since I'd heard a lot about it.
I'm so glad I did. I ended up learning quite a bit in the process. Some quick highlights about using Proxmox for VMs in general:
* Proxmox supports creating a "cluster", so you can login though one machine to administer them all. You can conveniently "move" VMs between machines pretty seamlessly.
* If you install the para-virtualization drivers for e.g. Windows or Ubuntu VMs, you can do pretty fast remote KVM. E.g. I could run Youtube on a Windows VM in my basement over "Spice" and it almost looks like it's running locally. (Not that it's a use case I care about, mostly just shows the fact it's performant.)
In terms of actually getting around to deploying k3s on top of the infra:
* I ended up learning HPC-Packer, and HPC-Terraform, which integrated nicely with my existing Ansible experience.
* Packer turns an Ubuntu ISO + my "base" Ansible setup playbooks into a pre-baked machine template directly in Proxmox. (My local machine's Packer binary just orchestrates the process.)
* Terraform deploys the machine template into the Proxmox cluster. Basically a config file of machine names + IPs + mac adresses, and a few other params and initial setup.
* Ansible then installs any final dependencies (anything not in the base template), setups up the first k3s master, grabs the join token, and adds in 2 more master nodes for a proper `etcd` backend.
* Ansible then installs my base Kubernetes services (cert-manager, Rancher, Longhron storage, etc) via running helm commands on one of the nodes.
* This is where I'm at now; the next step is for me to deploy my existing Flux.cd-automated "Gitops" apps (built for ARM64+AMD64 via Gitlab runner, also in Proxmox). These _had_ been running on my now-quite-crusty-seeming Pi cluster.
I can run a single command to delete all the VMs, and rebuild + setup everything (full HA cluster + apps deployed and running) from scratch in ~6 minutes without any manual input required from me, just a few secrets/params in a config file.
This has made exploring the horizon of possibilities _so much easier_ without getting locked in; I can try to weird Longhorn storage configs, or try out k8s monitoring stacks without worrying about needing to "back out" my changes if I picked bad settings. (Just blow it up and try again!) I can change how VLANs are configured in early steps, or try adding a library to the base Ubuntu install cluster-wide super easily, etc.
I am primarily a software-engineer, so it has been really nice to delve into the operational side of things, and get a proper reproducible setup. It really has transformed how I think about the cluster in that it's no longer a "thing to carefully maintain", but instead a great sandbox to explore AND deploy my own k8s applications on top of without playing cloud bills.
My Proxmox journey in the past few months definitely turned into more than a rabbit hole than I'd expected.
yeah you can write weird shell script and re-implement hot vm migration (moving a physical machine from one physical host to another without shutting it down) but what's the point, really? you might as well use proxmox.
if you ever actually need to see how it's done you could just see what's proxmox doing under the hood (it's open source after all)
It has a webui with a bunch of features.
If you don't like or want those things, that's fine. This question feels like a "no true power user would...".
In the same amount of time, if I were diving in to DIYing it, I'd probably be about 5 minutes into reading the arch linux wiki page for KVM/qemu.
Like any distro, it has some quirks, but, those are worth dealing with because the day to day is almost always frictionless.
ZFS support that just works is really nice too. They also have a backup solution I’d like to check out but haven’t yet.
I'm a virsh/kvm user myself, but I admit I'd probably leverage more features of the platform if the interface was easier to use.
Have a look at this script for example https://github.com/francescor/swarm/blob/main/create_swarm_v... which is the "proxmox" side of cloud-init, that then load an ordinary cloud-init https://github.com/francescor/swarm/blob/main/cloud-init/clo...
I try to do everything without Ansible: cloud-init is pretty powerful, too, but I am reaching the 64k limit, so sooner then later will go "back" to Ansible.
There's a pretty neat Proxmox API library written in Go that can do this all for you: https://github.com/luthermonson/go-proxmox
There's a Terraform plugin planned, as well. Not sure what the status on it is, currently.
I also am slowly working on my own Proxmox CLI, consuming the go-proxmox library: https://github.com/perchnet/gomox
But unfortunately I don't have much software engineering experience, so it's a very slow process... :)
It's working so well I now have too many vms and need to get a bigger disk.
[1] https://sschueller.github.io/posts/wiring-a-home-with-fiber/