I work in the refurb division of an ewaste recycling company[0]. To prepare a machine for sale, the drive needs to be wiped, and (optionally) an OS loaded. Wiping happens in WipeOS[1], which loads when you PXE boot on the internal company network. To install an OS, I have a separate network on my desk that will load iVentoy[2] when PXE booted, where I can further boot from ISOs I have on my server, but I almost always install Linux Mint. With those 2 things, I can largely do my job without fumbling with and losing USB drives.
I have 2 16 port switches on my desk, with over a dozen ethernet cables plugged into each. The yellow cables will PXE boot WipeOS, and the black ones PXE boot iVentoy.
Ah I did something similar at the university I worked at as a student. Everything was already set to network boot as the first step, so I set up a PXE server that loaded up DBAN. When we needed to wipe a lab before decommissioning, we'd flip their network to the PXE DBAN network, tell them all to reboot, and decom them in the morning.
Saved us a bunch of hours we then used to play Minecraft haha
I've set up PXE booting at two previous companies for very different use cases.
The first was to automate server deployment; we ran bare metal servers, and even though we had managed hosting in our data centre the installation, configuration, and deployment of a server could potentially take days since it was just me doing it and I had other things to do.
So one day I set to work. I installed an Ubuntu server the same way I always did and then captured the debconf configuration to turn into a preseed file. I set up the disk partitioning, etc., and configured the OS to boot from DHCP. Then I configured the DHCP server with MAC addresses for every server we got and an associated IP address so that a given physical server would always get the same IP address.
Then I set up an internal apt repository; that's where I put custom packages, backports I had to recompile, third-party packages (e.g. perconadb) and so on.
Lastly, I set up salt (the management orchestration tool, like puppet or chef or ansible) with a nice simple (detailed) configuration.
The machines would be configured to boot via PXE. They'd load the kernel and initrd, which contained the preseed file that answered all of the installation/configuration questions. Lastly it ran the post-install shell script which started salt and ran the initial configuration, much of which was based on hostname. This would turn the current DHCP-provided IP address into a static networking configuration so that the server wasn't reliant on DHCP anymore; it would ensure that SSH keys were installed, and that the right services were enabled or disabled, install some packages based on the hostname (which represented the role, e.g. db02.blah.blah got percona installed). I also had some custom data sources (whatever you would call them) so that I could install the right RAID controller software based on which PCI devices were present; after all that, it would reboot. Once it rebooted from the local disk, salt would pick back up again and do the rest of the configuration (now that it wasn't running from a chroot and had all the required systemd services running). What used to take me several days to do for two servers turned into something one of our co-ops could do in an hour.
Second was another company that wanted to standardize the version of Linux its developers were running. Again, I set up an Ubuntu installer and configured it to boot iPXE and then fetch the kernel and the root image via HTTPS. The Ubuntu installer at that point was a Snap, and the default 'source' was a squashfs file that it unpacked to the new root filesystem before proceeding with package installation. I set up some scripts and configurations to take the default squashfs filesystem, unpack it, install new packages via apt in a chroot, and then repack it again. This let me do things like ensure Firefox, Thunderbird, and Chrome were installed and configured not from snaps; update to the latest packages; make sure Gnome was installed, etc. A lot of that was stuff the installer would do, of course, but given we were on gigabit ethernet it was significantly faster to download a 2 GB squashfs file than to download a 512M squashfs file and then download new or updated packages. One again what used to start with "Here's a USB, I think it has the latest Ubuntu on it" and take most of a day turned into "Do a one-off boot from the network via UEFI, choose a hostname, username, and password, and then just wait for twenty minutes while you get a coffee or meet your coworkers". I even found a "bug" (misbehaviour) in the installer where it would mount the squashfs and then rsync the files, which seemed significantly slower because the kernel was only using one thread for decompressing; using `unsquashfs` could use all cores and was dramatically faster, so I got to patch that (which I'm not sure ever made it into the installer anyway).
The one thing I couldn't make work was the OEM installation, where you put everything down on...
The fun thing about learning to boot from PXE, is that you have to learn it every time you onboard a new type of hardware... or a new VM hypervisor... or new NIC firmware... or new BIOS firmware.
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
I've not found this at all -- PXE "just works" on legacy boot or UEFI for me. I've used it for years to install hosts via Foreman (https://theforeman.org/), as well as for personal stuff on my home network, and it's so much better than getting people to use USB sticks or whatever else!
Agreed, PXE seems ideal for provisioning things, but it's just too hard to use, especially when you're not on a network you fully control.
I just want to start the computer, and have it download an immutable OS image from somewhere I decide (and supply a checksum for, etc). I don't want to set up TFTP or any of this other stuff. It feels like I should be able to just specify an IP (let's say) a checksum (maybe supply that information to the NIC directly somehow), and be off to the races after a reboot.
TFTP is crazy slow, even with RFC 7740 (buffering), but the payloads are usually small so few people care.
Thankfully modern BIOSes tend to implement HTTP boot option, where you can point to any HTTP or HTTPS URL (as long as the URL ends with ".efi", which is a pretty dumb limitation if you ask me).
You can also do things like boot with PXE (Legacy or UEFI PXE boot) to get a small image like iPXE, and then have iPXE do the http boot part. This means that you have an extra shim but you can pull larger images than TFTP is any good for.
TFTP is also UDP and I don't think it is pipelined, so it's all req->ack->req->ack, so any additional latency hits it hard too.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
PXE is awesome, especially if you combine it with systemd's UKI mechanism and its EFI stub. You can load a single file via TFTP or HTTP(S) and boot into a read-only (or ramdisk-only) full Linux system. Most off the shelf distributions can be made to work in this way, with a small bit of effort. A very usable Debian system is a few hundred MB.
You can extend this with secure boot (using your own keys) to sign the entire UKI file, so your firmware will authenticate the full "disk" image that it boots into.
I've used PXE (not even iPXE, just DHCP/TFTP without HTTP) mainly in environments where a LAN client-server game would need to be launched on many systems at once. Nothing quite like rolling out a hand-tailored distro for a single game to 16 computers and seeing them all boot and load straight into the game, one after the other, entirely unattended, from one broadcast boot-over-Ethernet trigger.
I think at one point we were even using distcc to use the clients to speed up rebuilds while iterating on the game. I should revisit that with iPXE and icecream.
I’m glad a lot of server stuff has redfish. But something better still needs to be there for non-server stuff for sure. Raspberry pi style bootloaders would be amazing, ones we could configure to use a certain image before powering on for first boot would be even more amazinger.
Having done lots of network booting over the years, here are a few of my lessons learned:
PXE is a big improvement over the boot EPROMs that we needed to install on our NICs back in the day. Those would get an address via DHCP and then TFTP the boot image, and boot it.
I've had some trouble with PXE boot that's been caused by STP. If your PXE boot server has, or is behind a bridge with STP turned on, it can prevent the client from booting. I think this has something to do with the STP "learning state", but turning off STP on the bridge can solve the problem, as long as you're sure that you will not be creating any network loops on the affected interfaces.
There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.
18 comments
[ 4.9 ms ] story [ 40.4 ms ] threadI work in the refurb division of an ewaste recycling company[0]. To prepare a machine for sale, the drive needs to be wiped, and (optionally) an OS loaded. Wiping happens in WipeOS[1], which loads when you PXE boot on the internal company network. To install an OS, I have a separate network on my desk that will load iVentoy[2] when PXE booted, where I can further boot from ISOs I have on my server, but I almost always install Linux Mint. With those 2 things, I can largely do my job without fumbling with and losing USB drives.
I have 2 16 port switches on my desk, with over a dozen ethernet cables plugged into each. The yellow cables will PXE boot WipeOS, and the black ones PXE boot iVentoy.
[0] https://www.ebay.com/str/evolutionecycling
[1] https://www.wipeos.com/
[2] https://www.iventoy.com/en/index.html
Saved us a bunch of hours we then used to play Minecraft haha
The first was to automate server deployment; we ran bare metal servers, and even though we had managed hosting in our data centre the installation, configuration, and deployment of a server could potentially take days since it was just me doing it and I had other things to do.
So one day I set to work. I installed an Ubuntu server the same way I always did and then captured the debconf configuration to turn into a preseed file. I set up the disk partitioning, etc., and configured the OS to boot from DHCP. Then I configured the DHCP server with MAC addresses for every server we got and an associated IP address so that a given physical server would always get the same IP address.
Then I set up an internal apt repository; that's where I put custom packages, backports I had to recompile, third-party packages (e.g. perconadb) and so on.
Lastly, I set up salt (the management orchestration tool, like puppet or chef or ansible) with a nice simple (detailed) configuration.
The machines would be configured to boot via PXE. They'd load the kernel and initrd, which contained the preseed file that answered all of the installation/configuration questions. Lastly it ran the post-install shell script which started salt and ran the initial configuration, much of which was based on hostname. This would turn the current DHCP-provided IP address into a static networking configuration so that the server wasn't reliant on DHCP anymore; it would ensure that SSH keys were installed, and that the right services were enabled or disabled, install some packages based on the hostname (which represented the role, e.g. db02.blah.blah got percona installed). I also had some custom data sources (whatever you would call them) so that I could install the right RAID controller software based on which PCI devices were present; after all that, it would reboot. Once it rebooted from the local disk, salt would pick back up again and do the rest of the configuration (now that it wasn't running from a chroot and had all the required systemd services running). What used to take me several days to do for two servers turned into something one of our co-ops could do in an hour.
Second was another company that wanted to standardize the version of Linux its developers were running. Again, I set up an Ubuntu installer and configured it to boot iPXE and then fetch the kernel and the root image via HTTPS. The Ubuntu installer at that point was a Snap, and the default 'source' was a squashfs file that it unpacked to the new root filesystem before proceeding with package installation. I set up some scripts and configurations to take the default squashfs filesystem, unpack it, install new packages via apt in a chroot, and then repack it again. This let me do things like ensure Firefox, Thunderbird, and Chrome were installed and configured not from snaps; update to the latest packages; make sure Gnome was installed, etc. A lot of that was stuff the installer would do, of course, but given we were on gigabit ethernet it was significantly faster to download a 2 GB squashfs file than to download a 512M squashfs file and then download new or updated packages. One again what used to start with "Here's a USB, I think it has the latest Ubuntu on it" and take most of a day turned into "Do a one-off boot from the network via UEFI, choose a hostname, username, and password, and then just wait for twenty minutes while you get a coffee or meet your coworkers". I even found a "bug" (misbehaviour) in the installer where it would mount the squashfs and then rsync the files, which seemed significantly slower because the kernel was only using one thread for decompressing; using `unsquashfs` could use all cores and was dramatically faster, so I got to patch that (which I'm not sure ever made it into the installer anyway).
The one thing I couldn't make work was the OEM installation, where you put everything down on...
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
I just want to start the computer, and have it download an immutable OS image from somewhere I decide (and supply a checksum for, etc). I don't want to set up TFTP or any of this other stuff. It feels like I should be able to just specify an IP (let's say) a checksum (maybe supply that information to the NIC directly somehow), and be off to the races after a reboot.
Thankfully modern BIOSes tend to implement HTTP boot option, where you can point to any HTTP or HTTPS URL (as long as the URL ends with ".efi", which is a pretty dumb limitation if you ask me).
TFTP is also UDP and I don't think it is pipelined, so it's all req->ack->req->ack, so any additional latency hits it hard too.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
You can extend this with secure boot (using your own keys) to sign the entire UKI file, so your firmware will authenticate the full "disk" image that it boots into.
I think at one point we were even using distcc to use the clients to speed up rebuilds while iterating on the game. I should revisit that with iPXE and icecream.
PXE is a big improvement over the boot EPROMs that we needed to install on our NICs back in the day. Those would get an address via DHCP and then TFTP the boot image, and boot it.
I've had some trouble with PXE boot that's been caused by STP. If your PXE boot server has, or is behind a bridge with STP turned on, it can prevent the client from booting. I think this has something to do with the STP "learning state", but turning off STP on the bridge can solve the problem, as long as you're sure that you will not be creating any network loops on the affected interfaces.
There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.