Seeing the final picture made me think of something: one of my "life hacks" is to not accept cables that are too long. I used to think the longer the better, and just coil it up or something. But long cables just get in the way and collect dust etc.
If something is going to be considered permanent, cut that cable down to length. Either buy shorter moulded cables, or learn to wire cables yourself. Too often have I left a cable long "just in case" only for it to get in the way for years to come.
For patch cables it's easiest and best to buy moulded cables that fit your rack. For things like power cables (extension leads etc.) it's easiest to wire them yourself (at least, in the UK where our plugs are designed to be wired by anyone).
On the other hand, I did this, cut my cables, and after I needed to reorganise things slightly, it was very difficult, even a centimetre was at luxury. Also, when I need to move a computer for some reason, there’s no room, at all. These days, I’m trying to leave at least some extra cm (usually that’s an inch or two, depending on their location) for that. I’d do very tight cable cut only for situations when I’m super sure nothing will ever move. Again, even then, I’d rather leave some extra inch, just in case.
I am very much on board with this line of thinking. Because things are still somewhat in flux, it was much easier to plan for excess cabling and have a place for that cabling to live in the rack so that things can be moved if needed. I'll probably re-cable it with cut-to-length cables in the future.
One thing that I haven't found a solution for though: I have a lot of USB and HDMI cable coiled up behind the Beelink boxes (for KVM connectivity). I've found the normal length cables (1', 3', 6', etc), but I haven't been able to find custom length cables for those specific connections. Do you happen to know anywhere I can find those?
I'm currently sitting at < 200W, but I expect that to go up with a higher workload. The SER9s idle at 5-7W, but they can run at 50-60W sustained without thermal throttling. Some reviewers have claimed that they can run at 75-80W sustained for 10-15 minutes, but I think that's pretty unlikely.
It’s always fun to see how people build their home labs but I went exactly the opposite route.
I focused on energy consumption, because of cost and - gasp - wanting to be mindful about it given the current predicament.
Anything that needs to be on 24/7 is on a Pi, and anything that consumes more power is turned off by default (remote poweron possible).
For me at home there is zero need for redundancy and I use a cluster of four tini-mini-micro 1L PCs for my lab work. There are also turned off by default and are also low-power.
It's funny, I started with rPis for the same reason, but I'm about to replace them. I bought 20 rPi 4Bs for my homelab, and I just couldn't get them to do what I needed. I was looking to run a home k8s cluster and the Pis were just not suited to it at all (don't use sd storage for k8s 'cause it'll burn out the card w/writes, booting off usb was unstable even with powered usb hubs, netboot turned into an enormous pain in the neck).
Yea, but, like owning a car you soup up in your garage, it isn't about what you _need_, it's about what's fun and what's enough to give you something to do in your free time.
With the way Ubiquity has treated their software stack on the network side in the past years (major bugs, regressions, and updates that had to be reissued multiple times), I wouldn't trust them with all my data. Ubiquiti's QA was outsourced to the customers and a NAS is the last place where I want to risk bad updates, no matter how many backups I have.
This is definitely one of the purchasing decisions that I regret. My backups are robust and trustworthy enough that I don't have data loss concerns, but the software is atrocious and the customizability is extremely limited.
e.g. I wanted to serve tftp directly from the NAS. I can log in and `apt install tftpd-hpa`, but that package has to be reinstalled every time the NAS updates.
I'll be replacing this in the medium term, but I'm not buying more hardware for a little while lol
It now only consists of a Intel n100 with a big SSD and 32GB RAM running Proxmox. These China TopTon-boxed with their 5x Intel i226-IV network cards are great and can be passively cooled.
Every night the Proxmox makes a backup onto a Raspberry Pi which runs the Proxmox Backup Server.
I've read that PBS requires fast (NVME-fast) storage and a decent CPU to handle incremental backups efficiently. What speeds do you get when restoring backups?
Space and energy costs are a factor for me too. My current Mac Mini SMB is really good but DAS consumes lot of power. Ideally, I would really love Mikrotik's RB5009UPr+S+IN next iteration to have antennas and a 4-Bay Rose Data Server merged. Gateway, Switch, Access Point, PoE for CCTV and NAS - all in one.
Yup. Mine rates 80W on idle, but it has _two_ NAS boxes, a pair of N150s, a Ryzen APU Steam server, and a beefy i7 with 128GB RAM and a 3060 (which of course is the one burning the most watts in use). Most of it is running Proxmox and backing up VMs to the Synology (which I've demoted to dumb storage with a few Docker containers since they started burning bridges with their customers).
This is awesome and I wish more of this were happening. Hardware home labs are the best way to learn. I gained most of my Linux/FreeBSD skills at home.
It feels with cloud computing a generation of computer scientists kind of missed out on the experience.
Enjoyed this. Lots of large homelabs like this are built in order to stream video or local llm models and that usually leaves me feeling a bit left out because I have been building my own, but have no interest in either of those things.
Some services I am interested in are hosting my own RSS feed reader, an ebook library, and a password manager. But I'm always looking for more if there are any suggestions.
For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
It might be fun to have so many machines, but in reality it's simpler and cheaper to virtualize everything on two or three powerful hosts. Considering that you're using a soundproof rack already you might as well go with used rack servers with lots of memory and compute. Those also come with goodies like BMCs, dual PSUs, and ECC.
Personally I have two Xeon rack servers running in a Proxmox cluster with a SBC Qdevice. It has more than enough memory and compute for my needs and it also serves as my virtualized router and NAS. The whole setup only takes up 4U of space (servers + switch + modem/qdevice) with a single UPS on the floor, and idle power is around 150W.
I'm rebuilding my homelab [1] too, actually, but deprioritizing it while I stave off a wee spot o' burnout.
I'm excited to see that you're building on Talos; I am too! I used to use CoreOS back in the day, like 8-9 years ago or smth, on PXE-booted VMWare VMs, and I've always missed that cleanliness.
That's a large part of why I'm rebuilding right now - I based everything around Ansible + Terraform, and that's workable of course but working iteratively on a homelab leaves so much cruft around, can lead to incidental complexity, etc etc etc.
I too am preparing for an entire rebuild of my home lab so the posting of this information is greatly appreciated at this time.
My home lab has been rebuilt 3 times in the last 20 years and the continuous mention of energy usage within this post is a main driver and one of several primary objectives towards my effort. My 36U rack as configured now with Dell Servers having Intel Xeons E chips has served me well for all the tech stuff that interests me both in hobby and profession. My entire setup now with all devices running draws 200 watts continuously and my existing home energy renewables generation and battery storage system will greatly benefit from this home lab rebuild as I am targeting energy use and floor area recovery outright while learning many new things as well. This rebuild is also a lead into my fourth tech company I'm starting so my motivation is strongly tied to many factors not discussed.
Looking forward to the future posts, Stay Healthy!
I haven't had a huge issue with power draw, but I live in a place where electricity is really cheap. I had 3x Dell R720s, an R210ii, and an Aruba 48-port PoE switch and felt like I had way more power than I knew what to do with, but it still didn't seem to have a substantial impact on my power bill. Maybe I lucked out with some components. Or maybe the overall power draw in my house is so pants-on-head stupid that I just don't have realistic expectations anymore.
This year I redid my whole lab as well and prioritized declarative everything so it'd be easy to go from scratch if needed. So I've got nix to create RKE2 images with cloud-init for XCP-ng templates, Terraform with XenOrchestra provider, and ArgoCD for the K8s cluster.
I've got two R640's so I can live migrate, and an R720XD with TrueNAS (democratic-csi for K8s persistence). QSFP (40Gb) for TrueNAS / R720XD, and SFP+ (10Gb) for R640's linked to a Brocade ICX 6610.
So I can update the hosts, and K8 nodes with 0 downtime. Do I need it? No, but I learned a lot and had / have fun deploying and maintaining it.
40 comments
[ 3.3 ms ] story [ 56.7 ms ] threadYet. :)
If something is going to be considered permanent, cut that cable down to length. Either buy shorter moulded cables, or learn to wire cables yourself. Too often have I left a cable long "just in case" only for it to get in the way for years to come.
For patch cables it's easiest and best to buy moulded cables that fit your rack. For things like power cables (extension leads etc.) it's easiest to wire them yourself (at least, in the UK where our plugs are designed to be wired by anyone).
One thing that I haven't found a solution for though: I have a lot of USB and HDMI cable coiled up behind the Beelink boxes (for KVM connectivity). I've found the normal length cables (1', 3', 6', etc), but I haven't been able to find custom length cables for those specific connections. Do you happen to know anywhere I can find those?
Peak draw could probably be 2kW for a beefy system so electricity costs could really skyrocket depending on usage patterns.
We have pretty reasonable power rates here (https://www.idahopower.com/accounts-service/understand-your-...), so ~$12-20 per month depending on tier.
I’m in the middle of my first homelab journey with just a single mini PC with 8GB of RAM. Excited to one day get to this level.
I focused on energy consumption, because of cost and - gasp - wanting to be mindful about it given the current predicament.
Anything that needs to be on 24/7 is on a Pi, and anything that consumes more power is turned off by default (remote poweron possible).
For me at home there is zero need for redundancy and I use a cluster of four tini-mini-micro 1L PCs for my lab work. There are also turned off by default and are also low-power.
Cool, but nothing a single compute machine wouldn't get done with a bunch of VM's if learning and home workload are the focus.
This thing probably idles at a couple hundred watts.
With the way Ubiquity has treated their software stack on the network side in the past years (major bugs, regressions, and updates that had to be reissued multiple times), I wouldn't trust them with all my data. Ubiquiti's QA was outsourced to the customers and a NAS is the last place where I want to risk bad updates, no matter how many backups I have.
e.g. I wanted to serve tftp directly from the NAS. I can log in and `apt install tftpd-hpa`, but that package has to be reinstalled every time the NAS updates.
I'll be replacing this in the medium term, but I'm not buying more hardware for a little while lol
It now only consists of a Intel n100 with a big SSD and 32GB RAM running Proxmox. These China TopTon-boxed with their 5x Intel i226-IV network cards are great and can be passively cooled.
Every night the Proxmox makes a backup onto a Raspberry Pi which runs the Proxmox Backup Server.
It feels with cloud computing a generation of computer scientists kind of missed out on the experience.
Some services I am interested in are hosting my own RSS feed reader, an ebook library, and a password manager. But I'm always looking for more if there are any suggestions.
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
Confused as to why 10x nodes but running single control node and no HA?
Control nodes can be pretty light - even down to raspberry 4s
Personally I have two Xeon rack servers running in a Proxmox cluster with a SBC Qdevice. It has more than enough memory and compute for my needs and it also serves as my virtualized router and NAS. The whole setup only takes up 4U of space (servers + switch + modem/qdevice) with a single UPS on the floor, and idle power is around 150W.
I run 4 thin clients and only one of them is a worker. The rest are untainted control plane nodes in HA.
I'm rebuilding my homelab [1] too, actually, but deprioritizing it while I stave off a wee spot o' burnout.
I'm excited to see that you're building on Talos; I am too! I used to use CoreOS back in the day, like 8-9 years ago or smth, on PXE-booted VMWare VMs, and I've always missed that cleanliness.
That's a large part of why I'm rebuilding right now - I based everything around Ansible + Terraform, and that's workable of course but working iteratively on a homelab leaves so much cruft around, can lead to incidental complexity, etc etc etc.
Anyway, I'm pumped to keep reading!
[1] https://clog.goldentooth.net/
My home lab has been rebuilt 3 times in the last 20 years and the continuous mention of energy usage within this post is a main driver and one of several primary objectives towards my effort. My 36U rack as configured now with Dell Servers having Intel Xeons E chips has served me well for all the tech stuff that interests me both in hobby and profession. My entire setup now with all devices running draws 200 watts continuously and my existing home energy renewables generation and battery storage system will greatly benefit from this home lab rebuild as I am targeting energy use and floor area recovery outright while learning many new things as well. This rebuild is also a lead into my fourth tech company I'm starting so my motivation is strongly tied to many factors not discussed.
Looking forward to the future posts, Stay Healthy!
I've got two R640's so I can live migrate, and an R720XD with TrueNAS (democratic-csi for K8s persistence). QSFP (40Gb) for TrueNAS / R720XD, and SFP+ (10Gb) for R640's linked to a Brocade ICX 6610.
So I can update the hosts, and K8 nodes with 0 downtime. Do I need it? No, but I learned a lot and had / have fun deploying and maintaining it.