Ask HN: Show Your Homelab and Home Server Setup for Inspiration

The 4U sits on the floor in a "closet", I think you can picture it

Like someone already mentioned /r/homelab for more ideas.

The motherboard was a huge pain to work with, and I had to return two (!) units; I made the third one work. The first two would not boot in the same configuration.

I set up the box with NixOS. Here is its flake: https://github.com/fnune/bilbo

The NixOS experience was fun!

https://www.servethehome.com/introducing-project-tinyminimic...

Corporate thin clients with Pentium J5005 or similar also make decent raspberry pi replacements, usually just not as good price-power ratio as proper minipc. Certain models have pcie slot, which makes them ideal DIY routers when fitted with pcie NIC & pfsense/opnsense as OS. If you're using a consumer router, such project can make lots of sense.

For NAS, I recommend building your own. Personally I use Asrock J5040 mini-itx board with Node 304 case and Unraid as OS, houses 4x 3.5" drives, so far happy with it. For maximum reliability you might want a mothebroard that supports EEC memory though.

Need to get a new router though. I have a Mikrotik hAP ac and it seems like it's hanging onto dear life.

  * 3x Dell R620 running K3os on clustered Proxmox
  * 1x X11 Supermicro 2U with ZFS handling the disk array
  * 1x X9 Supermicro 2U, also with ZFS, that is a backup target for the other one (WoL to ingest, then shutdown)
  * UniFi Dream Machine Pro
  * UniFi 24 port PoE 1G switch
  * 2x UniFi UAP-AC-PRO APs (not in the cabinet, obvs)
  * APC UPS with battery extension

The Supermicros have some cheap ADATA NVMe SSDs for boot (the X9 has a modded BIOS to make this possible).

In general I like it, but the current in-progress work is rewriting all of the Ansible playbook that generates QEMU images for Proxmox, and setting up PXE so I can grab IPs on boot for new images. The latter will enable me to shift off of K3os in favor of Talos. Technically I can do so now, but it requires manual input to set the IP addresses.

Beyond that, I want to get a 10G switch so a. RBD-backed devices are faster b. ZFS send/recv is faster c. Just because.

It survived the HN front page and even bigger traffic spikes. Did blip out when Elon Musk tweeted a link to one of my blog posts, but only momentarily.

Now that server's been relegated to run a test environment and to perform various odd jobs.

I do think everyone interested in programming should have some sort of server in the house. Being able to run processing jobs for a few days really does radically expand what you're able to do.

I've got an OPNSense appliance router running FreeBSD. I made no attempt to modify it. I tried to build a Linux router but realized I can't make anything that takes up this little space and the PCIE-mounted NICs to get the number of ethernet ports costs a lot more than buying an appliance. I built the NAS server myself, using ASRock Rack mini-ITX motherboard with AMD CPUs that have graphics integrated onto the chip. It's got a 1 TB SSD cache and 8 spinning drives. The machines I use as servers are six Minisforum small form-factor PCs, similar to NUCs but fanless, cheaper, and with AMD 6-core processors. I don't think they outperform or anything, but having more cores makes it easier to pin many VMs. These plus the NAS server and television are plugged into two Cisco switches that support 10 GbE and two UPSes that typically give me about 30-40 minutes in the event of the frequent power outages we get in Texas.

I've been tempted forever to try building a "real" server, but they're power-hungry and loud and way more than I need. The small form-factor PCs have done the job and I can run them in a closed cabinet that looks identical to all my other cabinets. The only modification I needed to do was install USB fans in the cabinets themselves to ventilate the heat, but they don't make any level of noise I can perceive.

I've got Aruba Instant-On WiFi access points, one for each floor of the house. They run five separate networks, one for work devices, one for IoT, one for televisions that aren't hard-wired, one for guests, and one for a main non-work WiFi network. Everything except the main network is forbidden from sending packets to local IPs. I don't know there's much benefit outside of that, but it allows me to set the television network to be optimized for streaming, low QoS on the IoT and guest networks, and make the main network WiFi 6. It's also pretty funny to ban porn on the guest network and see which houseguests notice and complain about it.

Being in a townhouse, the only walls I have running floor to ceiling are either shared or external, which means they're very tightly insulated, and running cable between floors was a pretty serious challenge. If you're ever going to do it, I would recommend doing it immediately upon moving in, before you do anything else, before you even move in furniture. I'm pretty serious about cable management and keeping things neat, so I run everything I can through walls and/or floors. No cable is loose except at the last mile.

As for the self-hosted services, I don't use any sort of on-prem management layer like vSphere or Proxmox or anything. It's all Arch Linux with libvirt running on hardware-accelerated KVM. I at least automated the Arch builds by putting provisioning scripts on USB drives with a "cidata" label since the Arch installer comes with cloud-init and you can use this for unattended installs by just in two drives at first boot instead of one. Most everything else runs on Kubernetes, with Longhorn as a storage provider. I use Ansible playbooks to install Kubernetes and the applications are installed and configured with GitOps. So the external services are a Git server and Minio on the NAS acting as a backup target for anything that will backup to S3 as well as a p...

most important thing for my homelab i ever did was starting to mess around with VMs. first with virt-manager to have a gui, then just plain kvm (actually probably qemu) on the commandline using scripts to auto-install vms, and now a combination of Proxmox and Ansible to make a vm with just a few keystrokes. The freedom of messing around as much as i want without danger of breaking my system is the best.

next most important was setting up dns and dhcp automation - whenever the ansible playbook makes a vm, it immediately reserves a static ip for that mac and makes a DNS entry for that vm's name pointing to it's ip - so i can easily reach all my vm's by name without having to go through the effort of setting that up. great for when a test vm becomes permanent and needs to be accessible. this combined with using ansible to automatically create my user and put my ssh pubkey into it's authorized-hosts file makes everything SO smooth, no friction at all to work on different vm's and experiment.

also, when the ssd proxmox was installed on died, it was super easy to restore all the vm's.

next step is probably automating public dns, port forwarding and reverse proxy settings - so far i am doing that manually, but i just got a new phone and refuse to log into google on it, so i've had to set up a bunch of services, kinda running my own cloud, and they all need port forwards and reverse proxy entries. speaking of reverse proxy... it's a wonderful tool, set up ssl/https ONCE on it and use certbot to manage certs in one place, and you can test any service by just forwarding it to the vm without ssl, no hassle. for critical things i guess i'd still recommend setting up ssl for your internal network too, defense in depth like.

Totally understand that it's a bit more expensive than second hand hardware and smaller elements so I'd only suggest going down that path once you know you'll get the utility out of it.

I've used it to replace my reliance on Google photos/docs and lean more on self hosting, though that means that backups, disk mirrors, and run books for restoring everything is double important!

Current iteration is: a skylake era CPU, 16 gb ddr4, 4x 14TB WD Reds, 2x 512 GB SSD, and a m.2 boot drive

Used primarily for shared storage, backups, media serving, and lightweight docker containers (home assistant, git tea, etc) in an Ubuntu VM.

It's been super stable (paired with a UPS) and was surprisingly economical for what it is. I've also learned a lot in the process.

I do wish my CPU had a few more cores though.

- Mid-'90s 42U IBM server cabinet

- Two 1500VA APC UPSes (non-rackmount)

- 1U Dell R610 (PFSense firewall)

- 1U Dell R620 (ESXi host)

- 2U Dell R720XD (60TB NFS/CIFS storage server)

- 4U rackmount case containing a Windows gaming/workstation PC

- 4U rackmount case containing a Linux workstation (the previous iteration of the gaming/workstation PC)

- Black Box KVM for the two workstations

- TP-Link unmanaged 10GB switch

- Cisco managed 1GB switch

- Cable modem

- Two 1U PDUs plugged into the UPSes - half-ass dual power path for the machines that support it.

It's a pretty nice setup; I have a bunch of system service VMs on the ESXi host (IDS, Splunk, etc) and can spin up a new one in a few minutes whenever I have a new project or want to try something out. The cables from the KVM go through the floor to my office on the main floor so I don't have to listen to fans and can switch with a key combination. And of course I have plenty of storage for whatever - I back up all of my VMs and my workstations to it, I can download pretty much anything I need to, etc.

And since it's been asked before, yes, electricity is oddly inexpensive where I live.

Server: Dell Precision Rack 7910 with a single E5-2690v4 (14C/28T), 64GB DDR4 ECC (4x16GB), 4x 3.5" HDDs (a hack) with 30TB net storage in RAID1 and 2x small SSDs (OS on RAID1, and temporary data/caches without RAID). I put in the 2x GbE + 2x SFP+ NIC. It's connected with 10GbE and 1GbE.

My old system was nice, but upgrading meant I had to get an expensive CPU; luckily the Dell fell in my hands. With this one, I could double the core count cheaply (it's a 2S system and the CPU is really cheap) and iirc have 192 or 256GB memory per socket (with the cheap 16GB modules).

The above hardware plus all the PoE stuff (5x UniFi AP, a switch behind the TV, DECT-VoIP Gateway), DSL modem and a few ESP draw 110W. The server itself is in the 80W ballpark (that's 15€/month worth of electricity). This could be reduced with more modern or less powerful hardware. What I did was replace my old, big array of 12 disks with four much bigger disks. Since OS and cashes are on SSD, the disks can spin down a lot. I only insert one PSU, since the second one adds 17W idle power draw.

Compared to other home servers that can be pushed much lower it's still okay, since it replaces a bunch of services I'd have to pay for & allows much faster transfers to my workstation (for backups etc) than the 32MBit/s DSL. The old server was running out of cores... The only external service I still spend money on is mail.

Server runs baremetal Arch Linux (it's not cattle), and two VMs (qemu): OpnSense and Home Assistant. Samba serves files from the raid to the network. Services are partially native, some in podman. There are: cockpit, mealy, foundry, step-ca, scanserv, vaultwarden, mosquito, zigbee2mqtt, pihole, tasmoadmin, uptime kuma, UniFi manager, samba, lancache, heimdall, plex. Plus two custom services. It also interfaces with the PV using RS485. A 20m USB cable connects it to the zigbee stick on the second floor. I'm looking forward to adding influxdb/graphana for long term monitoring of our heat pump and BEV power consumptions.

The OpnSense does DHCP, wireguard and local DNS.

First step was to setup cockpit, since I like it to configure IPs and access the VMs. Then OpnSense for inter-VLAN routing and firewall, and to the outside world.

Since I wanted to encrypt the traffic even locally, that was the second step: I have a step-ca that serves certs using ACME. My nginx acts as a reverse proxy for most services and gets certs from the step-ca. The CA is limited to the .lan so it can't be used to intercept other traffic. Also, DHCP puts hosts on .dhcp.lan, so random hosts can't just try to pick any domain name inside the network and get certs for that.

With that done, I looked at services I use or that sounded useful, and spun them up.

I run proxmox for my router (opnsense) and several servers on it like gitlab, Mastodon, matrix chat, lemmy and some others.

I run it as a “pi-vpn” tap server and it has Samba installed.

Using Tunnelblick on my MBP, I can access my LAN from wherever I am. TimeMachine works too.

However, I mostly use the VPN to get to UK TV when I’m abroad. Sky Go, Netflix and iPlayer mostly.

I’m planing to use my second Pi 4 as a tap client and the WiFi as an access point. I’ll take it with me next time.

I’m going to try to get a Sky Q mini working with the system to get UK TV in an extended holiday in Portugal. Both ends have symmetric 1Gbit connections so it might work.

Edit: I’ve also had a Pi 4 running N64/PSX/Arcade emulators (Emulation Station) over-clocked to get a good frame rate. PS Bluetooth controller.

Raspberry pi 4 8GB with a 22TB external hdd and 1tb external ssd. For uhhhh media. Runs *arr stack and samba. Raspberry pi os (debian) with everything defined in a docker compose file. Also runs tailscale. Having my entire media library anywhere I have one of my devices is pretty sweet and setup was stupid easy. No plex.

An i3 NUC for hosting a game server for some friends. Think it runs ubuntu server. Runs Valheim. Also in docker compose.

A mitac board with dual GbE and a low power intel cpu N3-something. Runs pfsense.

For learning and tinkering I spin up VMs in hetzner and delete them when I’m done. We use AWS via terraform at work and I’m not likely to work somewhere where physical servers is something I’d deal with so using the most likely interface I’m going to be put in front of makes more sense for me. Having everything in terraform really is lovely, from networks to machines it’s all in my editor.

I like the Supermicro Xeon D boards because I can power 6 of them off a single power supply (the GPU cables can be converted to a 4 pin cpu).

I also use systemd-nspawn (w/ dnf --installroot or debootstrap) or docker to attach instances to the network, where each has it's own layer 3 address distributed by frr.