I keep meaning to produce a post like this. My setup is not nearly as pretty though. Love to see the cannabis grow operation exhaust fan, haha. Very well done!
I didn’t see a filter on the exhaust fan. He is so thorough I’m sure he thought of it but I think he needs a filter to keep the fan running long term unless he is filtering the air into the room and that is good enough?
Thank you for encouraging others. You’ve earned a loyal fan.
And thanks for sharing your knowledge. Infrastructure has been an off and on hobby, but it’s hard to know how to take it to the next level. Or absurd levels (in the best possible way), in your case.
I just bought a Ubiquiti Dream Machine SE along with a U6 Enterprise a few weeks ago and so far it’s my favorite tech purchase ever. The management interface is years if not decades ahead of everything else I’ve used before.
My only complaint is that there’s no public API and thus no official Terraform providers.
The Unifi software is pretty incredible. I am not using their router though (well... an ER-4 but it does not share the same management mechanism) so I am missing out on a lot of the goodies.
I actually moved away from ubiquiti stuff to OPNsense + TP link AP. The firewalls rules on OPNsense makes a ton more sense and the plugins are pretty awesome.
You can run a UniFi hardware controller through a plugin on OPNsense. Worked pretty well minus mongoDB conflicts with zenarmor. But I switched to omada APs because they had better range and speed.
I would take Ruckus Unleashed over unifi all day long. Ubiquiti, unfortunately, feels like everything is constantly in beta, both hardware and software. Wait until they release a UDM SE v2 and abandon firmware on the UDM SE.
The UDM in particular is a masterclass in how to upset all your customers. (coming from a previous all-in UBNT customer that had a first gen UDM Pro).
Ebay Ruckus + OPNsense and my network has never been more stable and performant.
Yeah I've heard a lot of mixed things about their software support but I did a lot of research and people seem to say it's gotten a lot better over the past couple years. I certainly wouldn't say any of it feels like a beta, it's currently been up since I bought it two weeks ago with no issues.
I love my Dream Machine too. I bought it about six months ago and it's been solid and was really easy for me, a novice with advanced networking, to get going with vlans and firewall rules.
I don't need this setup at all, like I don't need cat6 ports in every room, there's definitely an element of it being a hobby. Maybe I just like flashing LEDs.
Sonos gear still using STP was an unwanted pain, all part of the fun I guess.
I wanted to replace my EdgeRouter Lite with one of these but sadly Unifi hasn’t reached feature parity with their Edge line. The Dream Machines for example do not support multiple VLANs on the WAN port, when many ISPs use that configuration to deliver internet, VoIP and TV.
I also read some bad stories of the Dream Machines unable to restart after a power loss.
Wow, and my wife says my network is complicated! :)
In all seriousness, thanks for sharing, this is really incredible. I see a few similarities (fellow Harbor Freight shopper, ADS-B receiver)... but I took the mostly lazy way out and just use the TP Link Omada router, controller and access points. Works great for well over 50 wireless clients that we had at a recent BBQ. I particularly love your note about encrypted LoRa networks at the bottom there, I'll be interested in a follow-up on that topic. Thanks again!
Exactly. Those 50 wireless clients must be fed Internet while their 50 subjugated humans eat BBQ and prioritize device interactions over human ones. (I’m exaggerating of course! At least you are having gatherings — better than many of us!)
I've been very interested in the new TP-Link stuff, they have really come a long way. I don't know if I've really ever heard of much wrong with the TP-Link stuff
Should hopefully change in the next few years, about 54% of houses have fibre to the property and the plan is over the next five years to expand that to nearly 100%. I have had 100/1000 for a while now but symmetric is still a rare product that only smaller competing fibre companies are rolling out.
I'm in the UK (Oxfordshire) and have a 1gb symmetrical connection at home. It's provided by Gigaclear - there's a handful of other similar operators that do fibre in more "rural" areas. It costs £79 a month, so it's not cheap to be honest, but I love it.
Ring em up and tell them you’re leaving, they’ll drop you down to the new customer pricing. I’ve got the 1G up and down for something like £38 a month now?
I had 1g symmetric in the bay area (thanks at&t) and it was nice, but am now around your speeds (85m/13m) and it's clearly worse, but not really terrible. Certainly not terrible enough to pay $50k+ install to get munifiber, even though I'd enjoy it a lot. Maybe if one of the ISPs on munifiber starts offering 10g to residences. Not that I need it, but it'd be fun.
I’m in Spain and get 10Gbit symmetric which comes down to about 7000mbps up and down on speedtests to my airbnb (in a small town 30min away from a touristic hotspot) for 30 euro’s. My home in Barcelona gets 1gbit symmetric for 25 euro.
Competition here has gotten pretty good with lots of virtual carrier being allowed by law to use telefonicas ftth which has driven prices down.
I’m the opposite to you where I’m always surprised when I hear people in fairly big towns/cities in US/UK etc are still on less then 100mbit symmetric!
If you don't mind me asking, does your energy bill take a huge blow because of this? I had a modest homelab set up and had to start shutting things off due to how much it costs to keep it running
That would definitely be a big help, lol. I also imagine part of my problem is my house wiring- the previous owner had the place reno'd in '96 and clearly got the landlord special
Having a rack alone doesn't consume anything, it's what you put on the rack.
I have a rack setup and most of the time it consumes around 200 watts during the daytime and 100 watts at night, but can spike upto 600 watts if I put a heavy CPU+GPU load on it.
I also put my desktop into suspend at night, something which I think a lot more people with desktops could do. Don't run 24/7 services (e.g. Home Assistant) on your massive desktop with an i7/i9 and a GPU. Run that stuff on a NUC or Pi4 or anything that has low power consumption. Then turn your desktop on only when you're actually using it.
I get that but the blog discussed a pretty beefy setup. My whole rack still used less power than my gaming PC at load but after doing the math I ultimately saved more money by going serverless for my apps and dumping (non-critical) data into a B2 bucket on paper. In reality I just started shutting things off and only turn them on when I need them.
I'm curious as to what others are doing to save costs if anything. I love the hobby but we're in a recession, lol!
MiniITX boards are low idle power, 17 watts for idle with a Ryzen 5600 compared to a full size boards 50 watts. I would no longer buy anything larger, even for a gaming machine.
Amen on that. The past few rigs I built for myself were all ITX machines. I went through and undervolted my CPU, RAM, and GPU. Once my current gaming rig dies I'm just going to stick with the consoles for gaming until they goes to shit as well.
In an era of abundance we seem to be trying to use more power to support what we already do, rather than rethinking what we do to use less power. It's a shame!
Honestly I could I could easily solar power all of my compute needs (and probably a lot more) from my apartment's roof.
But at last, I'm a renter and my property management isn't going to allow that.
We're literally wasting clean energy because of a bunch of rich turds hogging real estate with shitty rules about what you can't do. When my property changed management the new management even banned EV charging on the property and locked up all the outlets in the parking lot. These kind of management are not welcome in Silicon Valley but somehow they are here.
It jumbles my jimmies hearing about outright rejection of modern tech by industries that would best benefit from said tech.
I'd almost be willing to bet money that your landlord would charge a "renewable energy" fee even if they're just dumping power back to the grid for $$$ even if they had the panels
I really wish more Mini-ITX boards came out that allowed GPUs to be horizontal so it could fit in a 2U case. Unfortunately making it vertical makes it 4U, negating the size advantages of Mini-ITX.
So I actually "repurposed" a riser cable from an old ITX build to slap it into my primary proxmox server to play around with home-rolled VDI. Surprisingly, it worked. If you're feeling bold you could try that route
If you don’t require ECC RAM a common setup these days is a Proxmox cluster set up on refurbished USFF PCs like the Lenovo m720. They sip power at idle.
My personal approach is separate archived storage from working storage and keep archive storage offline until I need it. Keeping hard drives spinning is costly.
I had bought an used enterprise server (Dell something), but it just used too much power and made some much noise. It's not as professional or reliable as op's homelab, but I'm now hosting most of my stuff on an Intel NUC. It's a much better fit for my use case and budget
Yeah, even with Comcast/Xfinity I run a HE.net tunnel for IPv6, just so I can have truly static allocations and not goof around with their DHCPv6 prefix request system.
It's been around for a long time! If you do the IPv6 training certification, they do eventually send the t-shirt :P
EDIT: They also provide free DNS for up to 50 domains at dns.he.net which is also worth looking at if you don't want to run your own for every little thing.
I haven’t used it in a few years, but one downside of an HE tunnel was that Netflix blocked traffic from it (since people were using it like a VPN to avoid geoblocks).
I had to run a DNS proxy to filter out AAAA records for any Netflix domains.
That is true, and a bunch of folks (including me) were experiencing issues with Google-related sites blocking with a 403 FORBIDDEN on the initial /64 static allocation from HE.net. Oddly enough, any subnet out of the optional routed /48 you can enable worked fine.
For sites that use HE.net and have more than one IPv6 subnet, I use the initial /64 for the DMZ, so that didn't matter on most of them. Small sites had just been using the /64 though, and I had to renumber those. Total pain.
HE and Cogent are still feuding over IPv6 peering, too. Using HE for regular web browsing will work fine, but if you want to host through HE your services might not be reachable to visitors from a Cogent network, e.g. office buildings.
The norm seems to be for hosting services to contract with both HE and Cogent if they decide to go with one of them. This is why for typical client browsing you wouldn't notice--the other side has ensured you can reach them. HE seems to be the cheaper option, so people will route most of their traffic through HE, and then separately contract with Cogent for the smaller amount of traffic originating from/to Cogent networks. Alternatively, you contract with another network provider who handles that idiocy for you.
To say "couldn't live without it" would be an exaggeration, but we do use it extensively for work, and having it at home makes WFH much easier. End-to-end connectivity, like in the old days, is very nice!
Probably not supplied, I’ve got no IPv6 on any connection I have access to other than a few Azure VMs (and it’s a PITA to set up so we haven’t bothered)
Looks like they've got dual-wan (at&t fiber + verizon 5G backup). It's pretty easy to do ipv4 failover with NAT as long as you've got an indication that it needs doing; ipv6 failover is a lot harder; unless you're going to NAT the v6 traffic too. I don't know about Apple devices, but I couldn't get Android or Windows to play nice with two v6 ranges in radvd. They wouldn't listen to priority, and they would do derpy things like pick an address from one router's advertisement and then send the traffic to the other router.
In some magic happy land where you can use your IPv6 prefix over both WANs, then yeah, things can work, but that doesn't work for residences, generally.
It’s a black hole for your wallet. Once you start. You can’t stop.
But I also now have a bunch of electrician tools for running wires, some private cameras that aren’t google or Amazon spying on everyone that walks by, and a network that isn’t bogged down by a couple people watching Netflix.
If you're crafty you can work around this- three of my servers (2 NAS boxes and a Proxmox box) were custom built from commodity hardware. I literally bought plain ol uATX and ITX decommed business gear from my local uni surplus store and a P-Link chassis for each used from eBay. I spent more on drives than I did machines (Y'ALL WANNA KNOW HOW MUCH IT COSTS TO BUY
16TB WORTH OF SSDS FOR ZFS MIRRORS? TOO. DAMN. MUCH.)
You really gotta do some due diligence to make sure you're not buying lemon parts but it's very possible to get a beefy homelab at fair prices!
But, its all used and all over a number of years, so not as much as you'd think. If you went out and tried to get it all at once, it would be crazy. Most things I spent a good while waiting for the right deal to come up
Like the UPS alone, I saved probably close to $3000 off MSRP
I'm sorry- when did they start doing this? I've been mulling making the switch to OPNSense. If pfSense is phoning home aside from the update check I might have a busy weekend ahead of me
I dunno if this is entirely true- my current pfSense router gets as close to line speed as I'll ever be able to get with my setup (~900 both ways to a test server in Chicago and on LAN) and I'm running an (overkill) i5 in my generic Chinese minipc. It's possible that I don't have Intel NICs but I swear I do
Got anything more I can read? Dunno if I'm just blind but I couldn't find anything on this. I'd like to learn more!
We had no problem routing gigabit on an Intel Atom D525 years and years ago at a previous job. Consumer Mini-ITX board, onboard gigabit NIC, and an Intel gigabit NIC in the one expansion slot. It did require minor tuning, but nothing that couldn't be done thru the web UI.
Everything I have to manage runs plain OpenBSD managed with Ansible now, so I don't know what the current state of pfSense/OPNsense throughput is.
Nice setup! I have a very similar Homelab minus the Generac (I regret not getting one before inflation kicked in, especially since I already have LNG to the home.)
My only recommendation would be switching your virtualization over to Proxmox (LXC / KVM) and setting up an HA cluster with Ceph and MLAG. It's relatively easy and free and will give you a lot more features than plain ESXi and even free vSphere/vCenter.
Yeah, the price on this genset I think has gone up around $4000 since I bought it, not including the install
I've been meaning to try Proxmox, but my day job heavily relies on ESXi, so its nice having something to mess with at home. I am also running vSphere with an Enterprise licence, so I get all the fancy stuff
The Pis are using GPS receivers to serve as NTP servers on the local network. The specific appear to use GPIO pins. You would not be able to wire these to the ESXI machines as readily. You typically also want time-sensitive workloads running on metal, as there are all sorts of potential complications with virtualized time.
I am sure there are PCIe solutions that could easily go into a more standard form-factor machine, but perhaps these are more expensive or less readily available; this is just conjecture. The ESXi machines have minimal PCIe connectivity. Perhaps such a card could be put into one of the Supermicro chassis.
GPS modules use SPI which you can get a standard USB serial adapter for $2 on eBay. As the title suggests, it's overkill. Basically it's using the Rpi as SPI adapter.
Edit: USB to TTL costs like $3.5, it's like a standard RS232 serial port but much simpler and faster.
If he's using the timing pin on a GPS module, you want it on GPIO or an actual built-in serial port, because SPI or USB or other adapters can't be configured to trigger a CPU interrupt within a few CPU cycles.
I'll have to go over the whole post, I'm terrible at writing! Quite a few errors have been pointed out. But I can just edit them and pretend they never happened
Not coming in with proof but I have personally seen this- a buddy and I did some testing and the Ookla tests were the only ones we did that were much higher than the others.
I'm sorry for not contributing with hard evidence but I've seen this behavior (though I never heard about it from anyone else before now)
Not surprised. I have long noted that when my my browser is strugeling to load tabs all i have to do is try connecting to speedtest.net for all open tabs to suddenly load and i have decent speed for the next 15-30 minutes.
Eh kinda... priority traffic shows the maximum potential speed of your link. The max speed of your link has nothing to do with the speed you'll get on any particular node on the internet. For example if the ISP has too little peering bandwidth to some other speed test.net host you will still be capped by that link.
These days I rarely need large bandwidth to any single host, but have a lot of bandwidth use between many hosts including streaming applications. Bufferbloat is far more apt to be problematic under this workload than ISP priority, unless there is a bandwidth shortage on the local links.
fast.com is the only browser based speedtest that shows the accurate results for me. The results from there resembles the download speeds I get when downloading files from different sites or installing upgrades through my package manager.
Cloudflare's speedtest often performs worse for me.
iperf3 - https://iperf.fr
Is the most accurate CLI speed test I've found (depending on which server you choose).
What's "accurate" is going to be different for every website / destination network anyways. Unless you suspect there's some issue with the speedtest software running on the browser / target, it's showing you the accurate speed you get to Cloudflare or whatever server you're trying to pull from. If you just want to see what your ISP is limiting your bandwidth to, sure, you need to find a server that can hit that.
The best speed test is to download some large linux ISOs with 100+ peers, so you max out your connection to tons of networks.
"accurate" is for me the speed I have on average throughout the day no matter what endpoint and not the max througput.
If I'm doing a speedtest, most of the time I just want to know if I'm now downloading file $x from $y what download duration can I expect.
Downloading packages with $PACKAGE_MANAGER, downloading files from $CLOUD_STORAGE, downloading videos from youtube, downloading linux ISOs, receiving files through $INSTANT_MESSENGER, ... they all are very close to the speed I get with fast.com.
Is there a post that covers the "why?" of all this? As someone researching maybe running some Ethernet for more stable WiFi and faster Internet access for general stuff, maybe some fiber for infotainment other purposes since it can lengthen various cables, and hooking up some security and nature cameras, I can't fathom why all of this is needed outside of it just being a hobby or running some sort of business (trading?) at home. For example, isn't $600 for redundant Internet for like the minutes the Internet is out every year worth it? Why is that level of connectivity needed?
Speaking personally, it's just another form of tinkering. Nobody HAS to do it, much like nobody has to buy and maintain their own computer, car, home, etc... We do it because we can and it makes us happy.
Extra note- I would NEVER recommend a business use their home resources for work unless you know what you're doing (and why you're doing it). Even my homelab has vulnerabilities and I'm a security professional- more moving parts = much larger attack surface. There's a reason why corporations pay big buck$ for managed security services. Most homelabs I've seen are mostly for fun and personal comfort.
"What good is knowledge that is never applied" is what drives me to stuff like this. I can't speak for others but I'm sure I'm not the minority here.
Gotcha. Thanks for the reply! To be clear, it wasn't a judgment but just a question to understand there weren't some unapparent reasons why. I think it makes total sense as a hobby and learning, but perhaps still overkill as the post title mentions. Haha.
I apologize if it seemed like I was irate- I'm not and your question is 100% the question we all ask ourselves before doing things like this (or for some of us, it's the first question on the wife acceptance factor audit). If anyone is running a data center in their home for a serious reason they either have small loads to justify the power consumption (rpi k8s cluster says hi), stacks to blow, regulatory pressure, or isn't factoring the costs in and is in for a rude awakening. I don't think (but also don't know) these labbers are the majority, and us homelabbers are already a rarity.
If it makes you feel better, megacorps are getting out of the "self-managed data center" industry and embracing the cloud, to exemplify your very point.
Lol. I was under zero impression that you were irate or anywhere close. My question was a little judgemental perhaps but not necessarily meant in any way. I was also curious if there was some interesting need that had come up. My primary thing that I want is a bunch of PoE powered nature cameras, buy I'm still figuring that out. It will affect whatever comes up though. Oh, and stable WiFi coverage.
I certainly have hobbies that go above any need or reasonable collection, namely synths and books. Haha.
It's interesting that you mention PoE nature cams- I designed a PoE home surveillance system for a friend that involved setting up a solar panel on a 30' pole that fed a box with a shitty camera system in it at ground level. From there he set the cams around his property- particularly where the foxes and coyotes would travel to get to his hens. The whole project was apparently about $800 aside from the solar panel (I just gave him the idea- I didn't help him build it).
He eventually got rid of the cameras because, well... They were shitty and only told him the critters were near AFTER he popped 'em. I think he's got a for-purpose system (in his own words "the new cameras didn't fall off a truck") now but it was a fun project!
I still think your question is reasonable. Even when someone is doing something for joy, there's usually a spark - an essence to it. There's usually an initial motivation that sent them down the rabbit hole, and it's interesting to hear what it was.
I mean, “need” is a pretty strong word. No one needs a MacBook Pro, you can do everything on a Raspberry Pi just slower (exaggerating but you get the point). Personally, having reliable internet has been a significant increase in quality of life but admittedly, mesh routers are getting closer and closer. The biggest draw are VLANs. When even the FBI recommends separate networks, it shows how prevalent these issues are. I know a bunch of people that got affected by cryptolocker. While having good practice is probably what helps me a lot, a hardened network helps tremendously.
Most of the time it's the IT Infrastructure, Operations, Security guys that love to do this for (WHY) learning, testing and as a hobby. Check out r/homelab and r/selfhosted for more on this.
There's a reason other than "because it's fun to tinker." I do it because I love the capabilities it provides. I don't love managing Ceph & Proxmox & whatnot, but I love being able to deploy whatever I want into a beefy cluster with 10gb without having to worry about cloudspend. If I wanted to replicate what I run in my home infra (I don't consider it a lab), it'd cost 200-300 in compute and easily another 300 in storage a month. Instead I spent ~2k on hardware and ~35/month in electricity.
It's the same thing with 3d printers, some like them because they want to install clipper and tune the best/fastest benchy. I do it because they love being able to CAD something and have it in my hands as soon as possible.
That said, nobody goes as far as this guy without really enjoying the tinkering
No, but that'd be neat! I don't have GPUs in my cluster though.
My big services are a Ceph cluster, a VPN, Borg (backups) and a K8s cluster (using kubespray).
In K8s I have the main stuff: Plex, Gitlab, Gitlab runner (for CI), vaultwarden (passwords), miniflux (RSS, might be moving to freshrss though), rust desk (remote-desktop), home assistant (smarthome).
My next project is to stream metrics into grafana. I have soil moisture sensors in my garden connected to stm32 boards, I just need to setup the receiving side and I can control my drip lines (they're using opensprinkler) with soil moisture information along with weather info.
> I have soil moisture sensors in my garden connected to stm32 boards, I just need to setup the receiving side and I can control my drip lines (they're using opensprinkler) with soil moisture information along with weather info.
This is a use case close to my interests. I want a network of environmental things monitoring and doing some detection stuff.
Wifi is rock solid, networking is fast, backups good and outages are almost nonexistent and are always due to me messing with things.
I have Netflix, Apple TV+ (is that what it’s called?) and a few others.
Particularly with Netflix, the quality is junk so I watch off Plex.
Other benefits are pretty neat too. POE just works. You can power cycle things remotely if you want (have never needed to) and the abundant local storage makes everything easy.
Enterprise stuff that is getting old is also dirt cheap. Converting sections of the network to 10gbe was very inexpensive.
I built a pretty serious "homelab" over a few years, in the sense of building out 24/7 services as robustly as possible. There is an element of practicality for very long-term maintenance if you run i.e. VMs off a different storage layer.
But mostly? It felt exactly like exploring computers in my childhood for the first time (I'm 43 now). I was bumping into things I had never bumped into it before, and it was really satisfying to figure all that stuff out.
I'm a game developer, too, and I realized awhile ago a big part of that is I like technical puzzles, and game development is very fertile soil. But there is a whole new world of enterprise networking/storage/virtualization/etc things that you wouldn't normally bump into in the course of software development.
And as a bonus, I have set up a lot of build systems for friends; being infrastructure-savvy is like the digital equivalent of owning a truck and helping people move all the time.
> There is an element of practicality for very long-term maintenance if you run i.e. VMs off a different storage layer.
That is very good advice. I know about a datacenter outage/degradation in a very large company, which all of you know.
A couple of network switches got overloaded. The switches connected the SAN to the VMs, so it caused all kinds of weird problems in different applications.
Why not? I had a pretty elaborate setup in the house I sold during covid (fortunately the purchaser was enthusiastic about it) with a wiring closet in each wing and a fibre spine running between them. Not at all as elaborate an external connection as the author but I had a lot of machines and wanted to keep wireless bandwidth for things that moved around.
One handy trick: I had at least one drop in each room, often one for each wall of the room. Behind the wallplate was a NEMA box with a conduit running straight down to the crawlspace (6' high in my house so hardly "crawl"). That made it easy to pull cable, not only initially but if I found I needed an extra run, which hardly ever happened. Instead of trying to run through a rat's nest of conduit the conduit was a straight shot and then the cables could easily be managed.
I have been working remotely since about 2005-2006. I have always love a good Internet connection with backups (wherever possible). I remember befriending the local cable guy so I can get 1Mbps in 2000.
Now, I have three Internet connections bonded and balanced. It is not about the minutes of disconnection but more about the disconnection when I needed it most. I have ample non-internet time and personal/family downtime. However, when the time comes, I’m happy that the Internet is never a bottleneck in my work and play. I have had this setup since around the early days of the Pandemic and our home, per se, “never had an Internet Outage” since.
I'm not into servers, devops, networking or anything of that sorts, but I love tinkering with them. I would love to have an "overkill home network" one day.
Just wondering:
1. How often do you actually need the third? As in, both primary and secondary are down.
2. Is it that important that you always have an internet connection?
After the 2nd, it was more of a fun and why-not! It is cheap, and comes built-in with a free Netflix subscription and quite a few others. I think it even has an unlimited voice call if I want to plug in a phone and use it.
It’s a good question, as long as the why is not a burden to be met.
First it’s a useful skill set that helps troubleshoot problems in code and apps that run on networks by knowing how such things work.
Second, try to think of it as a private cloud instead of a network. Because it uses a lot of the same types of software cloud provider do. Proxmox is one example that is a self hosted vps provider that is tremendous. So, it’s a private hybrid cloud that can push from your cloud to the other voids (or back). Build using a private hybrid cloud and you can push to,or between manu clouds
Wifi is for convenience, wired is for reliability. When wifi gets jammed, spotty, interfered with which can happen more often than imagined, especially during break ins, wired wins. Transferring files? Wired wins. 4K streaming ther doesn’t cut out when multiple devices are doing it at once? Wired wins.
Self-hosting is much easier than it was 5,10,15 years ago. Tools like proxmox loaded with yunohost running on a 1L usff/mini/tiny pc with 64 gb of ram and mirrored ssds ther maybe mirror to another i de oval box can sip power but power production grade apps for you personally. If you buy big beefy servers it will cost a bit more electricity.
Still, hosting locally can quickly pile up the savings on saas not spent.
Imagine being able to keep up all your test projects you might spin up a paid VPS for and run for way too long. They might not be production grade, but there is something valuable about having them around.
Data backups - the cloud is just someone else’s computer sold as convenient but not secure. Having your own local copy when the internet goes down means not as much of your life or work goes down. Remember it’s not a backup if there is only one copy of it.
Backing up your computers in the cloud are only so helpful when they are down, a lot of time to download them. Local backups win again.
Multiple connections can matter for people who wfh and can’t be down, or have spotty internet that is up.
If you like smart home tech, it’s a ticking clock until the cloud based supports for it that are usually free dissapear leaving perfectly good gear unusable. Instead you can run a local instance of home assistant, etc.
I used to host morethan I wanted ina. Data centre much like this rack. I was hesitant to come back to self hosting or homelabs but j have realized a home server that runs like an appliance (in between that wiring) capturing the sum of all my data and worn as services come and was pretty much unavoidable. Luckily it’s getting easier and easier to do.
Although not as elaborate as this, I have what one would call a "homelab" and it's for, well, testing and experimenting -- I write a lot of high-performance server/network-centric software (e.g, saturating a 100G link with 64B frames is a common test) and virtual machines just aren't suitable/capable most of the time.
If, say, you use AT&T and you need Internet for your job (WFH?), redundant Internet is crucial. It's not exactly "minutes" they are out. (Their outages states are often "meh, we'll get around to it in a day or so", figuratively)
If you live in a rural area, even more so. You're not high priority for fixes in the first place. (And AIUI, the author lives somewhere in rural Texas)
Do you need to run it as high tech as that? Probably not. My alternate Internet is my cell, tethered. (And if that breaks e.g. during travel, yes, I have a second cell with a separate provider, because heaven forbid the US had consistent cell coverage)
And most of us probably don't give a damn about e.g. 1,000 year photo retention. Or many of the other things he's doing. But it sure is fun if it appeals to your personality, and you can afford to keep it running.
Very nice setup, thanks for documenting it! I used to have a much more involved at-home network, but since moving the business out of the garage, I've moved most of the network stuff with it. Still think I'm going to have to pay pole attach fees and string my own fiber if I ever want it at the house...
If only I could find a way to get someone else to pay for it :P I tried to convince one of the local electrical co-ops to do it, and use my day-job's building (we own it) as the CO for the area. They actually sounded interested but then went radio silent.
Brightspeed did basically the same thing a few months ago, but they kinda suck.
> The generator is a 27kw Generator which powers everything in the house. This means the UPS's only need at most 10 seconds of runtime, as that's how long it takes for the generator to start and switch
That Generac generator though: it powers everything but only in case there's a power outage right?
P.S: there's a slight typo in TFA: it's Mellanox, not Mollonox.
Some Ryzen chips (I think- might be TR) often support ECC- see if you can find a compatible mobo that supports ECC modules and Ryzen sand! When I last checked there were a few products out there
Without looking at the listing I can't earnestly say what you're seeing isn't fake but I know back when I had the same question you're asking here I saw similar offerings. Ultimately I bought a used ITX supermicro mobo+CPU combo unit from eBay for like $200
I'd start with something like this[0]. The only search term I used was "xeon-d". You can easily find other form factors and combinations. This line of CPUs is lower power (the linked one is rated at 45W), so should be trivially cool-able with a very quiet system. They support ECC RAM. You're likely to find mostly Mini-ITX and uATX boards, so they will fit in just about any case you want.
I've done this a few times, and the killer has always been the RAM. By the time you're buying decommissioned enterprise hardware, it's hard (or just expensive) to source RAM from its QVL list - even harder so when you want ECC, which not all of the QVL'd SKUs will be... And, in my painful experience, if it's not QVL RAM there's a good chance it just won't post.
The top spec 32GB RAM on the qualified list for the specific board I linked earlier runs $40-$41 on ebay. The same speed and capacity for generic DDR4 runs in the mid $30s. The board has 4 RAM slots. So the premium for QVL RAM for that linked board is $20 over generic if you want to max out at 128GB.
Supermicro did have RAM issues in the early teens.
This sort of research is the non-dollar price of finding deals on server-class hardware.
If you want to get something pre-validated, and usually with a warranty, it can be worth it to look for resellers for the major OEMs that also offer refurbished hardware. It's more expensive than the typical options available on eBay, but still much cheaper than new hardware.
I've had no problem with getting the RAM (either new or just more eBay stuff) - and even got a replacement mainboard for relatively cheap when one burned up (it was easier to swap the mainboard than reconfigure everything for a new server).
The killer is the power consumption. It's better now compared to old enterprise gear of 20 years ago, but you can sometimes still pay for a brand new low-power system just in power savings alone.
Of course, if you have solar onsite that doesn't matter.
The power is part of why I prefer the embedded boards like the one I linked.
Especially putting one of these not in a rackmount chassis, you can opt for quiet cooling and fans that do not draw 10s of watts each. The idle loads can get fairly low, depending on your additional peripherals.
For very low power, I actually prefer small form factor machines, either used 1L SFF machines from the big OEMs (see STH's Project TinyMiniMicro[0]) or one of the many AliExpress mini PCs[1]. These sorts of machines tend to use low-power CPUs and a lot of the AliExpress ones are laptop parts.
As with all things, there are always tradeoffs. If you enjoy researching and bargain hunting for hardware, then this is a viable path. If you just want a machine that works, then this is onerous work.
Then you could have bought 4-6" Cat 5e/6a pre-made patch cables to connect in the majority of your patch panel ports. The only cables that would traverse through the cable mgmt panel would be longer cables to the servers in your rack.
I have a similar Cisco switch as your 24 port. I am not fan of how they arranged all the ports at the right side of the enclosure. Why not make the 24p one row along the top (or bottom), like the 48p with the 2nd row removed? It would be a lot easier for cable mgmt.
I noticed this as well. The patch panels seem only to bring some wire to the front of the rack, only to immediately send the patched wire toward the back of the rack again.
The reason its arranged like that is because it started as 1 switch and 1 patch panel, and slowly evolved, so its not ideal, but I am fairly happy with the outcome
If I could re-rack it all, I might make some changes. But that means turning everything off, and I'm not sure when, if ever, I want to do that
I had the patch panels above a collection of switches in both the MDF and IDF closet. About a year in I rearranged it to the more common interleaved switches and panels, mostly just to make it look a bit cleaner.
I moved from 10g to 40g for switch interconnects in a few places, as well as used the fiber I installed to do 10g to most of the desktops, etc. Fun stuff for sure!
Much nicer than my setup. I have three small form factor PCs running OpenBSD tied together with a small Netgear gigabit switch. Each has an external drive for backups using rsync and dump. Copies of the backups are made to another drive for offsite storage. One computer is a workstation, the other two are servers. One server is for internal use, the other is for external services. WireGuard is used to tie the outward-facing server to an Internet hosted server so services can be passed between them. There's no UPS though. I didn't want to deal with battery maintenance... Power outages are taken when they occur. One thing about that, the computers bios are set to leave the computers off if the power blips. The whole thing sits on a single shelf of the bookcase.
Are there any building code considerations (e.g., chimney effect for a fire) regarding a multi-floor wiring chase? Isn't this why laundry chutes are no longer permitted in some places?
I've also been curious about this in the context of making a dumbwaiter, but not enough to do the research. I would think having passive automatically closing fire rated doors would take care of it, but I don't know.
You could look into it but I think the reason laundry chutes aren't made anymore has more to do with kids getting stuck in them and less to do with fire code (after all, stairways are chutes connecting floors).
Of course there was another solution (I’m assuming you’re not still in the chute, by the way) - which is to make them so big that even a large adult male can’t get stuck.
But the main reason they’re gone now is that people put the laundry on the same floor as the bedrooms in modern homes.
Riser or plenum rated cables are required by code if you go between floors, except in plenum rated conduit. Conduit is usually more expensive though.
Non rated cable more easily catches fire and makes a lot of smoke. Plenum rated cable is about twice the price, but ethernet is pretty cheap for a single home.
463 comments
[ 3.3 ms ] story [ 423 ms ] threadAnd thanks for sharing your knowledge. Infrastructure has been an off and on hobby, but it’s hard to know how to take it to the next level. Or absurd levels (in the best possible way), in your case.
My only complaint is that there’s no public API and thus no official Terraform providers.
The UDM in particular is a masterclass in how to upset all your customers. (coming from a previous all-in UBNT customer that had a first gen UDM Pro).
Ebay Ruckus + OPNsense and my network has never been more stable and performant.
I don't need this setup at all, like I don't need cat6 ports in every room, there's definitely an element of it being a hobby. Maybe I just like flashing LEDs.
Sonos gear still using STP was an unwanted pain, all part of the fun I guess.
I also read some bad stories of the Dream Machines unable to restart after a power loss.
In all seriousness, thanks for sharing, this is really incredible. I see a few similarities (fellow Harbor Freight shopper, ADS-B receiver)... but I took the mostly lazy way out and just use the TP Link Omada router, controller and access points. Works great for well over 50 wireless clients that we had at a recent BBQ. I particularly love your note about encrypted LoRa networks at the bottom there, I'll be interested in a follow-up on that topic. Thanks again!
I've been very interested in the new TP-Link stuff, they have really come a long way. I don't know if I've really ever heard of much wrong with the TP-Link stuff
[0] https://www.youfibre.com
Competition here has gotten pretty good with lots of virtual carrier being allowed by law to use telefonicas ftth which has driven prices down.
I’m the opposite to you where I’m always surprised when I hear people in fairly big towns/cities in US/UK etc are still on less then 100mbit symmetric!
If you don't mind me asking, does your energy bill take a huge blow because of this? I had a modest homelab set up and had to start shutting things off due to how much it costs to keep it running
I apologize if I missed this info in the blog!
I have a rack setup and most of the time it consumes around 200 watts during the daytime and 100 watts at night, but can spike upto 600 watts if I put a heavy CPU+GPU load on it.
I also put my desktop into suspend at night, something which I think a lot more people with desktops could do. Don't run 24/7 services (e.g. Home Assistant) on your massive desktop with an i7/i9 and a GPU. Run that stuff on a NUC or Pi4 or anything that has low power consumption. Then turn your desktop on only when you're actually using it.
I'm curious as to what others are doing to save costs if anything. I love the hobby but we're in a recession, lol!
In an era of abundance we seem to be trying to use more power to support what we already do, rather than rethinking what we do to use less power. It's a shame!
But at last, I'm a renter and my property management isn't going to allow that.
We're literally wasting clean energy because of a bunch of rich turds hogging real estate with shitty rules about what you can't do. When my property changed management the new management even banned EV charging on the property and locked up all the outlets in the parking lot. These kind of management are not welcome in Silicon Valley but somehow they are here.
I'd almost be willing to bet money that your landlord would charge a "renewable energy" fee even if they're just dumping power back to the grid for $$$ even if they had the panels
My personal approach is separate archived storage from working storage and keep archive storage offline until I need it. Keeping hard drives spinning is costly.
That’s what the generator is for.
The catch? _No IPv6 support whatsoever_
I wasn't even aware you could use HE's services without being a customer!
It's been around for a long time! If you do the IPv6 training certification, they do eventually send the t-shirt :P
EDIT: They also provide free DNS for up to 50 domains at dns.he.net which is also worth looking at if you don't want to run your own for every little thing.
I had to run a DNS proxy to filter out AAAA records for any Netflix domains.
For sites that use HE.net and have more than one IPv6 subnet, I use the initial /64 for the DMZ, so that didn't matter on most of them. Small sites had just been using the /64 though, and I had to renumber those. Total pain.
The norm seems to be for hosting services to contract with both HE and Cogent if they decide to go with one of them. This is why for typical client browsing you wouldn't notice--the other side has ensured you can reach them. HE seems to be the cheaper option, so people will route most of their traffic through HE, and then separately contract with Cogent for the smaller amount of traffic originating from/to Cogent networks. Alternatively, you contract with another network provider who handles that idiocy for you.
Honestly I just use ipv4 and turn off ipv6 everywhere. Then I have just one configuration and one set of firewall rules (in and out).
In some magic happy land where you can use your IPv6 prefix over both WANs, then yeah, things can work, but that doesn't work for residences, generally.
To be completely honest, I find ipv6 wildly alloying, but I guess that means I need to learn more!
But I also now have a bunch of electrician tools for running wires, some private cameras that aren’t google or Amazon spying on everyone that walks by, and a network that isn’t bogged down by a couple people watching Netflix.
And I have plans for MORE.
I’m not in any position to state anything about the competition.
I am glad to have went the PoE route, even if running those wires sucked ass.
I only did UI cameras cause everything else I have is ui. There wasn’t much more thought than that.
You really gotta do some due diligence to make sure you're not buying lemon parts but it's very possible to get a beefy homelab at fair prices!
My 10G switch with PoE and 4 SFP+ ports cost only $100, for example, while many new products with similar specs cost $600+.
Best $60 purchase I ever made!
But, its all used and all over a number of years, so not as much as you'd think. If you went out and tried to get it all at once, it would be crazy. Most things I spent a good while waiting for the right deal to come up
Like the UPS alone, I saved probably close to $3000 off MSRP
I get blisters on blisters on my finger tips when making lots of cables.
Gave up on that years ago
Got anything more I can read? Dunno if I'm just blind but I couldn't find anything on this. I'd like to learn more!
We had no problem routing gigabit on an Intel Atom D525 years and years ago at a previous job. Consumer Mini-ITX board, onboard gigabit NIC, and an Intel gigabit NIC in the one expansion slot. It did require minor tuning, but nothing that couldn't be done thru the web UI.
Everything I have to manage runs plain OpenBSD managed with Ansible now, so I don't know what the current state of pfSense/OPNsense throughput is.
My only recommendation would be switching your virtualization over to Proxmox (LXC / KVM) and setting up an HA cluster with Ceph and MLAG. It's relatively easy and free and will give you a lot more features than plain ESXi and even free vSphere/vCenter.
Yeah, the price on this genset I think has gone up around $4000 since I bought it, not including the install
I've been meaning to try Proxmox, but my day job heavily relies on ESXi, so its nice having something to mess with at home. I am also running vSphere with an Enterprise licence, so I get all the fancy stuff
I am sure there are PCIe solutions that could easily go into a more standard form-factor machine, but perhaps these are more expensive or less readily available; this is just conjecture. The ESXi machines have minimal PCIe connectivity. Perhaps such a card could be put into one of the Supermicro chassis.
Edit: USB to TTL costs like $3.5, it's like a standard RS232 serial port but much simpler and faster.
Your accuracy will be less otherwise.
I also like having them bare metal
You might however want to read about the grocer's apostrophe: https://www.grammar-monster.com/lessons/apostrophe_error_wit...
I wouldn’t bring it up but… glass houses.
Curious to know how hitting HN impacted your setup (if at all)!
Total bandwidth over the past 5 hours is 253.37 GB. But total cached and served by CloudFlare is 237.82 GB. Uncached is just 15.54 GB
Here is CPU load on the VM: https://files.networkprofile.org/apps/files_sharing/publicpr...
Networking on the VM: https://files.networkprofile.org/apps/files_sharing/publicpr...
They can legally do that as we don't have Net Neutrality anymore.
I'm sorry for not contributing with hard evidence but I've seen this behavior (though I never heard about it from anyone else before now)
These days I rarely need large bandwidth to any single host, but have a lot of bandwidth use between many hosts including streaming applications. Bufferbloat is far more apt to be problematic under this workload than ISP priority, unless there is a bandwidth shortage on the local links.
Cloudflare's - https://speed.cloudflare.com
Cloudflare's speedtest often performs worse for me.
iperf3 - https://iperf.fr Is the most accurate CLI speed test I've found (depending on which server you choose).
The best speed test is to download some large linux ISOs with 100+ peers, so you max out your connection to tons of networks.
Downloading packages with $PACKAGE_MANAGER, downloading files from $CLOUD_STORAGE, downloading videos from youtube, downloading linux ISOs, receiving files through $INSTANT_MESSENGER, ... they all are very close to the speed I get with fast.com.
I like this one because your speed really depends on what you're downloading and this one tests some of those scenarios.
Extra note- I would NEVER recommend a business use their home resources for work unless you know what you're doing (and why you're doing it). Even my homelab has vulnerabilities and I'm a security professional- more moving parts = much larger attack surface. There's a reason why corporations pay big buck$ for managed security services. Most homelabs I've seen are mostly for fun and personal comfort.
"What good is knowledge that is never applied" is what drives me to stuff like this. I can't speak for others but I'm sure I'm not the minority here.
If it makes you feel better, megacorps are getting out of the "self-managed data center" industry and embracing the cloud, to exemplify your very point.
I certainly have hobbies that go above any need or reasonable collection, namely synths and books. Haha.
He eventually got rid of the cameras because, well... They were shitty and only told him the critters were near AFTER he popped 'em. I think he's got a for-purpose system (in his own words "the new cameras didn't fall off a truck") now but it was a fun project!
Some tech people enjoy "homelabbing" and happily throw money at their hobby.
It's the same thing with 3d printers, some like them because they want to install clipper and tune the best/fastest benchy. I do it because they love being able to CAD something and have it in my hands as soon as possible.
That said, nobody goes as far as this guy without really enjoying the tinkering
My big services are a Ceph cluster, a VPN, Borg (backups) and a K8s cluster (using kubespray).
In K8s I have the main stuff: Plex, Gitlab, Gitlab runner (for CI), vaultwarden (passwords), miniflux (RSS, might be moving to freshrss though), rust desk (remote-desktop), home assistant (smarthome).
My next project is to stream metrics into grafana. I have soil moisture sensors in my garden connected to stm32 boards, I just need to setup the receiving side and I can control my drip lines (they're using opensprinkler) with soil moisture information along with weather info.
This is a use case close to my interests. I want a network of environmental things monitoring and doing some detection stuff.
Wifi is rock solid, networking is fast, backups good and outages are almost nonexistent and are always due to me messing with things. I have Netflix, Apple TV+ (is that what it’s called?) and a few others. Particularly with Netflix, the quality is junk so I watch off Plex.
Other benefits are pretty neat too. POE just works. You can power cycle things remotely if you want (have never needed to) and the abundant local storage makes everything easy.
Enterprise stuff that is getting old is also dirt cheap. Converting sections of the network to 10gbe was very inexpensive.
It’s all rather addictive…
But mostly? It felt exactly like exploring computers in my childhood for the first time (I'm 43 now). I was bumping into things I had never bumped into it before, and it was really satisfying to figure all that stuff out.
I'm a game developer, too, and I realized awhile ago a big part of that is I like technical puzzles, and game development is very fertile soil. But there is a whole new world of enterprise networking/storage/virtualization/etc things that you wouldn't normally bump into in the course of software development.
And as a bonus, I have set up a lot of build systems for friends; being infrastructure-savvy is like the digital equivalent of owning a truck and helping people move all the time.
That is very good advice. I know about a datacenter outage/degradation in a very large company, which all of you know.
A couple of network switches got overloaded. The switches connected the SAN to the VMs, so it caused all kinds of weird problems in different applications.
One handy trick: I had at least one drop in each room, often one for each wall of the room. Behind the wallplate was a NEMA box with a conduit running straight down to the crawlspace (6' high in my house so hardly "crawl"). That made it easy to pull cable, not only initially but if I found I needed an extra run, which hardly ever happened. Instead of trying to run through a rat's nest of conduit the conduit was a straight shot and then the cables could easily be managed.
Now, I have three Internet connections bonded and balanced. It is not about the minutes of disconnection but more about the disconnection when I needed it most. I have ample non-internet time and personal/family downtime. However, when the time comes, I’m happy that the Internet is never a bottleneck in my work and play. I have had this setup since around the early days of the Pandemic and our home, per se, “never had an Internet Outage” since.
I'm not into servers, devops, networking or anything of that sorts, but I love tinkering with them. I would love to have an "overkill home network" one day.
Internet is super cheap in India and I can afford all three for a really decent price. https://www.instagram.com/p/CUWeopdPVOp/
Just wondering: 1. How often do you actually need the third? As in, both primary and secondary are down. 2. Is it that important that you always have an internet connection?
First it’s a useful skill set that helps troubleshoot problems in code and apps that run on networks by knowing how such things work.
Second, try to think of it as a private cloud instead of a network. Because it uses a lot of the same types of software cloud provider do. Proxmox is one example that is a self hosted vps provider that is tremendous. So, it’s a private hybrid cloud that can push from your cloud to the other voids (or back). Build using a private hybrid cloud and you can push to,or between manu clouds
Wifi is for convenience, wired is for reliability. When wifi gets jammed, spotty, interfered with which can happen more often than imagined, especially during break ins, wired wins. Transferring files? Wired wins. 4K streaming ther doesn’t cut out when multiple devices are doing it at once? Wired wins.
Self-hosting is much easier than it was 5,10,15 years ago. Tools like proxmox loaded with yunohost running on a 1L usff/mini/tiny pc with 64 gb of ram and mirrored ssds ther maybe mirror to another i de oval box can sip power but power production grade apps for you personally. If you buy big beefy servers it will cost a bit more electricity.
Still, hosting locally can quickly pile up the savings on saas not spent.
Imagine being able to keep up all your test projects you might spin up a paid VPS for and run for way too long. They might not be production grade, but there is something valuable about having them around.
Data backups - the cloud is just someone else’s computer sold as convenient but not secure. Having your own local copy when the internet goes down means not as much of your life or work goes down. Remember it’s not a backup if there is only one copy of it.
Backing up your computers in the cloud are only so helpful when they are down, a lot of time to download them. Local backups win again.
Multiple connections can matter for people who wfh and can’t be down, or have spotty internet that is up.
If you like smart home tech, it’s a ticking clock until the cloud based supports for it that are usually free dissapear leaving perfectly good gear unusable. Instead you can run a local instance of home assistant, etc.
I used to host morethan I wanted ina. Data centre much like this rack. I was hesitant to come back to self hosting or homelabs but j have realized a home server that runs like an appliance (in between that wiring) capturing the sum of all my data and worn as services come and was pretty much unavoidable. Luckily it’s getting easier and easier to do.
Hope that helps.
This is exactly the kind of nuts and bolts guy that is indispensable when keeping a cloud running.
If, say, you use AT&T and you need Internet for your job (WFH?), redundant Internet is crucial. It's not exactly "minutes" they are out. (Their outages states are often "meh, we'll get around to it in a day or so", figuratively)
If you live in a rural area, even more so. You're not high priority for fixes in the first place. (And AIUI, the author lives somewhere in rural Texas)
Do you need to run it as high tech as that? Probably not. My alternate Internet is my cell, tethered. (And if that breaks e.g. during travel, yes, I have a second cell with a separate provider, because heaven forbid the US had consistent cell coverage)
And most of us probably don't give a damn about e.g. 1,000 year photo retention. Or many of the other things he's doing. But it sure is fun if it appeals to your personality, and you can afford to keep it running.
Brightspeed did basically the same thing a few months ago, but they kinda suck.
That Generac generator though: it powers everything but only in case there's a power outage right?
P.S: there's a slight typo in TFA: it's Mellanox, not Mollonox.
Thanks, I'll correct that!
[0] https://www.ebay.com/itm/166190039675?epid=17034031881&hash=...
Supermicro did have RAM issues in the early teens.
This sort of research is the non-dollar price of finding deals on server-class hardware.
If you want to get something pre-validated, and usually with a warranty, it can be worth it to look for resellers for the major OEMs that also offer refurbished hardware. It's more expensive than the typical options available on eBay, but still much cheaper than new hardware.
The killer is the power consumption. It's better now compared to old enterprise gear of 20 years ago, but you can sometimes still pay for a brand new low-power system just in power savings alone.
Of course, if you have solar onsite that doesn't matter.
Especially putting one of these not in a rackmount chassis, you can opt for quiet cooling and fans that do not draw 10s of watts each. The idle loads can get fairly low, depending on your additional peripherals.
For very low power, I actually prefer small form factor machines, either used 1L SFF machines from the big OEMs (see STH's Project TinyMiniMicro[0]) or one of the many AliExpress mini PCs[1]. These sorts of machines tend to use low-power CPUs and a lot of the AliExpress ones are laptop parts.
As with all things, there are always tradeoffs. If you enjoy researching and bargain hunting for hardware, then this is a viable path. If you just want a machine that works, then this is onerous work.
[0] https://www.servethehome.com/introducing-project-tinyminimic...
[1] https://www.aliexpress.com/w/wholesale-mini-pc.html?catId=0&...
https://www.klennet.com/notes/2022-11-25-zfs-and-ecc-rant.as...
IF you can get ECC cheap, get it, otherwise, meh.
The Latte Panda Sigma looks pretty amazing too if in-band ECC is sufficient.
1. 24p Patch Panel (Odd ports of switch) 2. 48p Switch 3. 24p Patch Panel (Even ports of switch) 4. 1u Cable Mgmt
Then you could have bought 4-6" Cat 5e/6a pre-made patch cables to connect in the majority of your patch panel ports. The only cables that would traverse through the cable mgmt panel would be longer cables to the servers in your rack.
I have a similar Cisco switch as your 24 port. I am not fan of how they arranged all the ports at the right side of the enclosure. Why not make the 24p one row along the top (or bottom), like the 48p with the 2nd row removed? It would be a lot easier for cable mgmt.
If I could re-rack it all, I might make some changes. But that means turning everything off, and I'm not sure when, if ever, I want to do that
Decent robustness check
When I put together my own home setup (see https://www.reddit.com/r/HomeDataCenter/comments/ktz6yo/my_s... )
I had the patch panels above a collection of switches in both the MDF and IDF closet. About a year in I rearranged it to the more common interleaved switches and panels, mostly just to make it look a bit cleaner.
I moved from 10g to 40g for switch interconnects in a few places, as well as used the fiber I installed to do 10g to most of the desktops, etc. Fun stuff for sure!
Granted, that means you NEED an Internet connection... But it's as close to "walking data center" as anyone can get.
But the main reason they’re gone now is that people put the laundry on the same floor as the bedrooms in modern homes.
Non rated cable more easily catches fire and makes a lot of smoke. Plenum rated cable is about twice the price, but ethernet is pretty cheap for a single home.