If anyone is looking to get started with a homelab at a good price, I can highly recommend checking ebay for a Dell Wyse 5070. They flooded the market for $50 and are likely powerful enough for many needs. They have a M.2 slot that support SATA. The 'extended' version also has space for a small pcie card and has a parallel and 2 serial ports for a blast to the past.
This was exactly a use case I had in mind when building https://canine.sh -- also uses k3s as a provider, and provides a Heroku-like devex.
How to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying
The encryption is interesting but I wouldn’t call this over engineered at all, in fact it’s rather basic compared to a lot of homelabs I see people building particularly where people are doing K8s or similar over multiple machines.
Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.
> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this.
A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies.
Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
I’ve spend a decent amount on EcoFlow units (Delta 2) that I use as online UPS for my servers and networking. They work great and I also recently installed dual 220watt EcoFlow solar panels on my roof that pump in solar during the day. Works nicely though the ROI admittedly is not there at all, just a cool thing.
This is such a great post. I have a small collection of posts for inspiration in creating my homelab and this is getting added to it. Current have a Pi 4 with PiHole and a Beelink. Going to add one or two more machines.
Your VPS provider likely uses servers with ECC RAM, this home server doesn't. For most people it doesn't seem to matter but for me it does - a home server where I store my data needs to have ECC RAM.
As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case.
When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.
Building a homelab is an awesome way to learn a lot of things.
I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people.
I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.
While i get the whole homelab thing is exiting and a great learning experience, it's simply not worth the time and effort for the majority of people.
You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime).
In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever.
A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period).
It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks.
Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money.
Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.
The price and effort is practically irrelevant. My homelab is mine, local, and answers only to myself and a wall outlet. Also, where I live, the internet is simply not dependable enough to consider otherwise.
> Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you.
Agreed, but it doesn't have to take time from your family. I'm on a small team that self-hosts internal services to lower costs/risks. It takes very little time to maintain, and maintenance windows happen on our terms. Our uptime this year is better than "Github Actions", the latency is incredible, and we've had no known security issues.
There are two keys to doing this successfully: (1) don't deploy anything you don't understand (so it won't take you long to fix), and (2) even then, aggressively avoid complexity (so it doesn't break in the first place.)
For example, despite significant network expertise, we stuck to a basic COTS router and a simple IPv4 subnet for our servers. And the services we run are typically self-contained golang binaries that we can deploy with bash onto baremetal. No docker, kvm, ansible, or k8s.
This DIY setup saves us considerably more than it costs. Not for everyone, but with proper scoping, many readers of hacker news could pull this off without losing time with their loved ones.
I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments.
One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door.
At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me.
I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along.
I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks.
I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.
I love infrastructure. I run my own services at home too, and have for many years. But to be honest, the older I get the less fun it is to deal with issues that come up from time to time.
At some point you'll need to upgrade hardware and software, you get to do the exercise over again. There will always be lessons learned and its get better each time. Its still work.
This is cool - I have a similar home lab on a Mac Mini [1].
The encryption question is interesting. I don't have disk encryption turned on, because I want the computer to recover from power failure. If power turns off then on, the server would be offline until I decrypt it.
How does your "Wake on LAN" work with the encryption?
I really don't get why people like the Minisforum stuff over alternatives. I've unfortunately been given one, and honestly I'm really unimpressed between crap firmware, no real expandability and just all of the other compromises that come with buying Aliexpress hardware. For the same money you can either pick up a used entry model Dell/HP/Lenovo server (and if they're E3/W/other entry level Xeon, they're usually not terrible on power) or get a good ATX chassis and power it with some off lease Supermicro hardware. Then you don't need to compromise on things like OOB management, hot swap bays, a real SAS card, real 10G nics, ECC ram, etc. etc. Maybe people are just afraid of doing a little bit of putting hardware together? I've seen and have systems that use the above gear that have been going for well over a decade now, with basically no hiccups, and even the old Sandy Bridge era E3 stuff both punches above probably even RPi5 and N100 and doesn't draw more than 30-40W when you don't have spinning disks in there. I'm sure if you avoid AMD and go find a newer T variant Intel chip, you can both have your cake and eat it.
N100 is faster and more efficienct than any Ivy bridge E3. At idle the Xeon draws roughly 20W more, which works out to $30USD/year at the national average electricity prices. That gap widens as the load increases.
I can totally see why someone who doesnt need expandability would choose the cheap mini PC.
I just purchased a Minisforum mobo BD795i SE with a Ryzen 9 7945HX (16 core, 32 thread). Can’t beat the price to performance. Building a NAS / VM server with 5x 14TB Seagate Exos drives, 2TB NVME, 500GB boot SSD, and 96GB of DDR5 memory. I was able to buy all components including a 3u hotswap 5x drive caddy for less than $1,200 all in. Can’t really beat that.
I used to really like the minis but I had to basically e-waste two of them because the ethernet went bad (lightning strike i think) and there was really no way to replace them and the OS would crash from hardware issues from it.
Why install proxmox on top of debian? Proxmox distributes an iso that basically does the same as you with preseeding.
Although recently had to install proxmox as an apt pkg on top of debian happened because the proxmox iso wouldn’t install properly. That actually happened twice. I think I’ll just install debian from now on…
After some research, it seems much easier to just back up the Proxmox config (and VM disk images, if they're needed) than to define or deploy Proxmox VMs with OpenTofu or ansible.
29 comments
[ 3.2 ms ] story [ 52.7 ms ] threadHow to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying
Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.
A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies.
Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case.
When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.
I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people.
I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.
You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime).
In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever.
A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period).
It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks.
Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money.
Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.
Every homelab I have come across is a hobby project and time sink that is more like a backyard garden or classic car.
Agreed, but it doesn't have to take time from your family. I'm on a small team that self-hosts internal services to lower costs/risks. It takes very little time to maintain, and maintenance windows happen on our terms. Our uptime this year is better than "Github Actions", the latency is incredible, and we've had no known security issues.
There are two keys to doing this successfully: (1) don't deploy anything you don't understand (so it won't take you long to fix), and (2) even then, aggressively avoid complexity (so it doesn't break in the first place.)
For example, despite significant network expertise, we stuck to a basic COTS router and a simple IPv4 subnet for our servers. And the services we run are typically self-contained golang binaries that we can deploy with bash onto baremetal. No docker, kvm, ansible, or k8s.
This DIY setup saves us considerably more than it costs. Not for everyone, but with proper scoping, many readers of hacker news could pull this off without losing time with their loved ones.
I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments.
One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door.
At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me.
I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along.
I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks.
I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.
Having a uptime SLA for your "hobby" is a huge pain in the ass and absolutely sucks the fun out of it.
For me it was the constant requests for new media or midnight complaints about jellyfin being down.
If you want to learn infra ops just get a job in the field and get paid for it.
At some point you'll need to upgrade hardware and software, you get to do the exercise over again. There will always be lessons learned and its get better each time. Its still work.
The encryption question is interesting. I don't have disk encryption turned on, because I want the computer to recover from power failure. If power turns off then on, the server would be offline until I decrypt it.
How does your "Wake on LAN" work with the encryption?
[1] https://github.com/contraptionco/toolbox
I know we're in the AI hype cycle but I bet you meant LVM there >:-)
I can totally see why someone who doesnt need expandability would choose the cheap mini PC.
https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pm...
- Lightning protection
- 2 2U UPSes with 2 extra 2U battery packs each, temperature and humidity monitoring, and remote management
- 1x vSphere 7u3w Enterprise Plus + vCenter, 512 GiB ECC RAM, 16 TiB of RAID10 SSD, 96 thread EPYC server with 4 10 Gb optical NICs and 4x SAS3 ports
- 4U JBOD external NAS 330 TiB usable shared mostly over Samba with Time Machine support
- 2x Ryzen boxes with 128 GiB of ECC RAM, 100 Gb links to each other, 4 TiB RAID1 SSD, also used for distributed builds (also vSphere)
- Additional non-ECC 96 GiB tiny ITX Ryzen Windows lab machine
- Misc. non-ECC 128 GiB micro ITX Ryzen for additional distributed build capacity, currently Fedora w/ Podman and Docker
- Deciso OPNsense (Business license) router with 10 Gb optical ports, WireGuard, NTP-DHCP-DNS
- PoE 4x RPi 5 + SSD Ceph cluster
- Ubiquiti U7 Pro XGS APs
- Eufy security cameras with Home base
- PKI (TLS CA), TOTP 2FA ssh, YK gpg/ssh agent, RustDesk
- All boxes except lab, Fedora, and RPis are lights-out manageable and so don't need a KVM
To Do: NFS 4.2, LDAP, Krb5, TACACS+, SAML/OpenID (authelia), SNMP, Nagios or Grafana/Prometheus, K8s
Cost: $150/month in electricity