I run K3s on NixOS as the central piece of my homelab and it was actually even too much easy to setup (although now that I think about it there was some gotcha I had to manually tweak in K3s config). This "Kubernetes on NixOS the hard way" seems very interesting and I will have a look at it via the QEMU image at some point. Thanks for sharing!
First off, amazing post. I learned a lot about networking, Linux and Kubernetes.
As a learning project, this is absolutely awesome.
I run Kubernetes via Kind on Docker on NixOS.
There's a ton of other ways to get a development environment on your NixOS developer PC.
I don't pretend this one is very good, I just copy what my colleagues have come up with (+ NixOS).
For production workloads, I wouldn't run the kubelet using this much custom wiring.
I'd run Talos. It's vastly simpler, you can run them in NixOS VMs, it's declarative and lowers the surface area of things that need interaction, no SSH'ing in.
It seems like the author is torn between where to put control: In NixOS, or in Kubernetes?
You can move stuff, e.g. CoreDNS, out of Kubernetes for a "simpler" setup.
But the point of running workloads inside Kubernetes is that you get redundancy between nodes.
So if a single node dies, your services don't die.
Embracing Kubernetes, I certainly haven't let go of NixOS. My personal servers still just run NixOS.
It's much simpler, much cheaper, and resilient in its own way.
Selling Kubernetes and Cloud Native users on using NixOS, I'd probably go another way, e.g. via dev environments.
I wonder what the author’s development process for all this is like. It’s fascinating seeing the end result, and I certainly learned a ton. However, I’d love to see some of the trial and error process. My own process is starting from a configuration.nix on a node and iterate with `nixos-rebuild switch`, but it still feels like there’s better methodologies out there.
Author here. Yeah, unfortunately, that's kinda it: just rebuild a lot. At work we have a custom setup with a build server and agents for provisioning, which is nice for multiple nodes, but also even slower. The QEMU setup in the attached repo was added later and also handy for testing multiple nodes. QEMU is also nice because you can just trash the disk images to get a clean start.
5 comments
[ 2.5 ms ] story [ 19.8 ms ] threadAs a learning project, this is absolutely awesome.
I run Kubernetes via Kind on Docker on NixOS.
There's a ton of other ways to get a development environment on your NixOS developer PC.
I don't pretend this one is very good, I just copy what my colleagues have come up with (+ NixOS).
For production workloads, I wouldn't run the kubelet using this much custom wiring.
I'd run Talos. It's vastly simpler, you can run them in NixOS VMs, it's declarative and lowers the surface area of things that need interaction, no SSH'ing in.
It seems like the author is torn between where to put control: In NixOS, or in Kubernetes?
You can move stuff, e.g. CoreDNS, out of Kubernetes for a "simpler" setup.
But the point of running workloads inside Kubernetes is that you get redundancy between nodes.
So if a single node dies, your services don't die.
Embracing Kubernetes, I certainly haven't let go of NixOS. My personal servers still just run NixOS.
It's much simpler, much cheaper, and resilient in its own way.
Selling Kubernetes and Cloud Native users on using NixOS, I'd probably go another way, e.g. via dev environments.