I attempted something similar to this in a 20U cabinet some time back. The biggest issue is the fan noise that 1U form factor servers and network gear produce, with their rather high RPMs. One can hear the noise across the other side of the house.
We've since switched to fanless network gear and ATX form factor servers with large diameter fans to keep the family happy. It definitely doesn't look as nice, though.
You can get pretty much the same result from a couple of fanless routers (mikrotik, something running ddwrt, etc) -- resilient against hardware failure, power failure, and wan failure.
Not as cool though, and clearly not running any servers, but that's what things like AWS or Linode are for -- or for low power stuff, something like a fitlet [0]
> ATX form factor servers with large diameter fans to keep the family happy.
It's insane how quiet you can go with this approach, while remaining air-cooled. I know when my home server is running backup scripts because the noise increases at least tenfold when the hard drives spin. Fortunately, I have coordinated that to be only once a day -- the rest of the time the drives are in standby.
Modern servers can be decently quiet as well, assuming you don't run them at 100% load all the time. I've currently got 2xR210 II's, 1xR520, 1xR320, and a Juniper EX2200-48T running in a rack right behind me. It's audible while I'm in my office under normal load, but as soon as I leave the room and close the door you can't hear a thing.
It's not whisper silent, especially during the summer when the fans on the R320 speed up to around 6000RPM (and this is with a E5-2430L) - but that's mostly due to my office remaining closed from the rest of the house leaving the ambient intake temperature around 75-80F (rest of the house stays at 72F). I'm probably going to stick with 2U's (probably R520's) when I start expanding again to lower the noise at higher temps, since the more equipment I add the more heat gets trapped in the room.
Whenever I see solutions like this I think back to an org I worked at where a high-visibility day-long database outage gained upper level management attention. The response, after the managers talked to our vendor (IBM), was to re-architect everything to use HACMP clusters for all of our production databases company-wide.
That was followed by a couple years of 100+ hour/year cumulative outages due to HACMP stability issues, and an environment that everyone was deathly afraid to touch.
The hardcore network engineer in me appreciates the detail in these kinds of solutions, but these days the practical side of me is satisfied with usability and maintainability of SPOF cable access with a manual failover to mobile hotspot on the rare occasions that drops offline.
Reminds me of what my brother in law says: I don't want to be stuck doing tech support for my family.
With my luck, it would catastrophically fail while out of town, leaving the wife and kids without internet.
My dad set up a lot of complicated stuff like this. As people are prone to do, eventually he died, and it just made it difficult to troubleshoot technical problems for mom. So now the equipment sits in some corner, unused, because we replaced it all with something your average AT&T technician could troubleshoot.
> Are you advocating buying internet service from two different companies and paying for both every month in case one fails for a brief period of time?
That's not an unreasonable solution, considering most people already pay two ISPs (one fixed, and another for their phone/tablet). When your home wifi goes down, you're going to fall-back to your mobile anyway. I'm thinking of getting an extra data SIM, an LTE modem and do auto-failover.
--edit--
My needs are somewhat unique - my traveling laptop is on its last legs (and will be replaced by a cheap chromebook. Desktops/servers get better bang for the buck compared to laptops. Go figure!), so I tunnel onto a server at home for heavy-lift computing. If the internet fails when I'm not home, I'd be left stranded (and this has happened).
In my case my Surface Book 2 gives me all the firepower I need to not miss my desktop, and it also has a PCIE SSD on it like my desktop. I do agree, sometimes tethering is highly useful, at least in my case on my laptop. I try to keep as many things as offline capable as possible.
That's literally what the author of the article describes.
From a practical point of view I think it's silly to do such a thing for a residential situation, but I can appreciate using it as a learning experience for building systems like this.
Depends how reliable your isp is ans how much it costs if it goes down.
3g is good enough backup for me, but for the office we go for two routers two isps and vrrp on the lan side, load balance across the wans, with failover to the other one.
If I wanted a seemless non-SPOF network for my family, I'd put in two mikrotiks, with the primary on mains, and secondary on UPS, £120 for a pair to do routing at a decent (1gig) speed on the main, and built in 4G on the reserve.
Then I'd put the primary router on the wired line, the other one on a 4G sim which did nothing but heartbeats unless the wired line went down. If the wired line shut down, traffic would reroute via 4G within 10 seconds or so. If the primary router went down, the backup router would take over in a similar time frame. Might put some capping on the 4G router to the netflix/etc boxes to keep bandwidth costs down.
UPS would be about 10W, so £45 for a 4 hour one. Possibly look at renewable energy of some sort to keep the UPS going during an extended outage.
I'd then VRRP on the lan side with primary on the main router (which would have a backup route via the secondary router)
Cloud based VM to do monitoring/alerting and land outgoing openvpn tunnels from both routers to allow secure remote access.
£170, £10 a month plus main ISP, and an hour of config.
However in reality having an ISP provided router and showing them how to tether in a problem works fine. OK, they lose their devices if the main circuit goes off, but running those over 4G can be pricey.
I used to work for a company whose setup was super simple.
ADSL Modem > Firewall > Router > Web/DB servers
It was basic, but it worked. Our web servers were mission critical, but as a B2B business they, and the ADSL connection, didn't sustain a heavy load. The only issues we had over several years were with the ADSL modem. Everything else just worked.
When we moved office we moved our servers to a co-hosting centre with an upgraded network setup with all sorts of backup and redundancy. Every week something went wrong. Sometimes simple is best.
Former network engineer here, can confirm. Time and again I've seen redundant systems create their own problems where without all that extra complexity things would have been fine.
Even ISPs and CDNs I worked with sometimes have surprisingly uncomplicated redundancy systems (sometimes just a handful of small routers they are very much ready to power down to cut over to backup paths or bring up new paths) and often they do not use the more complicated methods.
The catch with complicated redundancy is there is always a very close relationship or protocol or something between redundant components, bet it storage systems, network systems, anything. Inevitably a system goes down or loses its mind and takes it's redundant peers with it.... every new system you introduce is one more piece that could reach out and take everyone else with it. I saw it time and again, and again...
My home network is built with Mikrotik kit which is priced where it's affordable to have spares. I have yet to encounter a failure, but could drop in a new router in a couple of minutes with the saved configs.
I have SNMP monitoring feeding from telegraf into influxdb on an RPI. Dashboard rendered with Grafana on PC. Also have telegraf pinging to all 24x7 devices and collecting data from electricity meter, smartplugs, and Nests. It's been fun to do.
The key part of redundancy is that your "redundancy glue"[1] must be significantly more reliable than each component, including its software and implementation -- because often the glue failing in isolation itself can cause outages. So the probability of failure was simply P(single failure); now for 2x parallel redundant systems it is P(single failure)^2 + P(glue failure). If P(single failure)^2 ~ 0, we need P(glue failure) < P(single failure), at the very least.
[1] i.e. the systems that interconnect the multiple redundant system, detect failures, redirect traffic, etc.
Very similar to the 'infrastructure as code' story, where you're still left with the construction and maintenance of the infrastructure that bootstraps the infrastructure as code systems.
Indeed it is important in this case of course that this does not happen :) To see the increased reliability and P(glue failure)<P(single failure) you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Another adequate expression to apply here is
"Who watches the watchmen?"
The answer again is the watchmen must watch themselves and be very reliable.
On this topic I recommend von Neumann's (the brilliant mathematician) "Computer and the brain" book, where he explores how computing systems can be reliably interconnected and how those failure probabilities interact. He was interested on how the brain could be so robust to failure -- don't worry there's no time spent speculating on how the brain works, instead he derives from first principles properties of reliable computing components, and possible reliable designs (the brain's unknown internal workings at the time, and now to a lesser extent, would follow as a special case). He used this same approach in analyzing the principles of life, where he came up with a self-replicating machine with a tape encoding of itself, predating the discovery of DNA -- it's a very inspiring and powerful approach. Unfortunately he could not complete 'Computer and the Brain', he was in declining health due to cancer and died while writing it. What was left is still very interesting imo. He is one of those giants whose shoulders we can sit on to peek over the horizon :)
As a caution against tenanting the deployment tools in-band, I'm reminded of an incident I witnessed about five years back. Company was moving their compute from on-prem to colo datacenters. Pretty good, mature setup: Almost entirely virtualized, 10Gb iSCSI SAN, credentials managed via a dedicated COTS tool, etc. They got most things over-the-wire to the DC. But the final migration had to be done cold - Shut the last bits down that were keeping everything running, move them to the DC and power back on.
Everything went very well until the SAN wouldn't come up. To get into the SAN and troubleshoot they needed the domain, which wasn't available. They had a local account on the SAN, the key for that was safely stored in the password manager. Which was a virtual machine. On the hyper visors. That wouldn't come up until the SAN was booted. Oops!
OK, that's a very obvious foot-in-mouth, in hindsight. As a more likely example, how about the Amazon S3 outage a few years back that wasn't reported on the status page, because the images for the status page were stored on... S3 :D
>you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Certainly it's possible to build redundant systems properly. But it's expensive. All the well-built redundant systems you listed understand that and budget for it.
Most half-baked redundant systems I've seen are a result of "I want four nines, but I only want it to cost 20% more than a two or three nines solution" type thinking.
I’ve seen overengineered and undermaintained HA systems result in much lower uptimes than a simple system with multiple SPOFs. I’ve seen well built and maintained HA systems fail under “rare” edge cases.
I’ve also seen well built and maintained HA systems work exactly as desired.
As a general rule, the cost of building and operating a reliable HA solution is not 2x, but at least 10x. If the system being protected is not worth that, you’ll very likely find the MTTR acronym far easier to catch than the rather more slippery HA.
I worked at a place that hosted the servers in-house. They even built a special little air-conditioned room and put a generator on the roof. I never knew all the details but there was dual everything, 2 lines coming in, stuff to switch between them, nothing could possibly go wrong... until the day it did. Turns out someone has plugged all the machines into a single extension cable, and the fuse popped.
Even the big boys do that in the big storm of 87 in the uk Telecom Gold (an early online service) was quite proud that the UPS kicked in - only to realize that the modems that linked to the x.25 network where not on the UPS :-)
My anecdata: I used to admin a SWIFT cluster. It was built by the manuals on IBM hardware, that included HACMP with quorum determined by a shared disk.
Nobody understood exactly how the cluster worked to the point that a correction my boss made on the physical connections, made us loose a couple of million of dollars in transactions not processed.
The funny part is, when the cluster was working fine, a takeover took at least 20 minutes. During that time nothing was "available". The thing is, no matter what, SWIFT Alliance took that time to properly close and open the DB.
* Cantenna/laser link to a house some blocks away to avoid local WAN link disruption
* For less performance-intense networks, remove the physical impediments: 2 routers, each with 1 APC, connected to 2 separate power circuits, connected to 2 WAN links, providing 2 radios each. No switch to go down or cables to trip over, redundancy of access point, redundancy of frequency/radio, redundancy of WAN link, redundancy of power. Hardware-wise this is pretty cheap and still highly available. If the routers are cheap, use a hardware watchdog.
I think the redundant outlink dwarfs all other improvements mentioned here. All but one of the incidents in my home have been due to ISP or optical fibre company issues. (Which is not surprising -- they have many more miles of cabling to maintain than I do.)
He specifically states the setup draws 220 watts at idle and that his electricity costs $0.0974/kWh. So 22024/10000.0974 = 0.514272 per day, or about $15.40 a month at idle.
yea if it's idle the entire month, which is doubtful. but even if it's not, it's not likely to be too much more than the $135 you calculated. I figured the internet service would have been more, since the rest of us get screwed by our ISPs on costs.
For those that assume that was just a joke on escalating size, the joke was actually made it a real thing by Netflix when they actually named the component that randomly shuts down not just services, but entire AWS availability zones of Netflix services.
Child with a water balloon? Hope you have multiple data-closets in your house...
def had myself set up to have wifi-only during power outages when i was a student in my first apartment in SF, but the no-SPoF here is above and beyond. i'm really curious about the switch configs, nothing like UniFi existed the last time I tried to do network-HA.
I love Unifi. They're the only consumer-grade access points that will do roaming worth a damn. They're access points, though, and don't function as a router.
I use the UAP-AC and an EdgeRouter. The EdgeRouter has relatively sophisticated capabilities for a piece of consumer great network gear. I have GigE fiber to the home and get ~900Mbps through the router (and ~400Mpbs through the access points).
I generally don't recommend the USG, which has similar functionality and is integrated with the Unifi management platform.
>I love Unifi. They're the only consumer-grade access points that will do roaming worth a damn.
What advantage do you get with that versus just running a bunch of APs with the same SSID/password bridging to a single router? I do that with 3 cheap tp-link routers (1 as router, 2 as APs) and LEDE and both my laptop and phone work seamlessly. At one point I considered actually doing full 802.11r AP roaming but the only actual use case I had for that was doing VoIP calls while roaming between APs with no drops. Everything else works fine with the small interruption of switching APs.
Unifi has a centralized controller that each AP talks to, and is able to coordinate roaming. Gives you 802.11r-like functionality but with close to zero setup beyond what's already needed to set up each AP (which is also centralized at the controller, so adding an Nth AP to an existing site is almost trivial).
> Note that UniFi Fast Roaming is not a direct implementation of 802.11r - it is a solution taking inspiration from 802.11r, with a few key proprietary differences. We've found that Fast Roaming provides about 90% of the roaming improvement offered by BSS Transition. However, Fast Roaming does not require client support, allowing backwards compatibility with all clients.
Reiterating this: Management and updates are wonderful through the unifi app. It functions similarly to Meraki, better in my experience and there's no recurring fees.
The app can be self-hosted, run in the cloud, or on something they call a "cloud key" that's not much more than a Raspberry Pi.
I've run it on a Pi3, and it's a little bit laggy, but tolerable. I prefer to run it on my little x86 server.
Both the self-hosted and the cloud version can be managed remotely, which is neat.
There are a host of other benefits, but given the price (around $100 per AP), I see no reason to use the more commonplace consumer grade stuff. Check out the unifi demo here:
For me LEDE/OpenWRT are easy enough to configure and fully open source. APs are also about half the price you quote and the selection is much wider as it works with most manufacturers (including UniFi's). So I see no reason to depend on a more expensive closed-source solution that I never know when it might go away.
OpenWRT is pretty powerful, certainly. If you prefer an open source solution, that's definitely the way to go. I've used OpenWRT which I liked quite a lot, but I prefer the ease of having an integrated solution with the Unifi products.
I knew that. My question is what advantage do you get out of roaming. I don't use 802.11r because I don't need it. It's not easy to configure in LEDE but it's doable.
Wifi roaming is pretty poor because it's entirely a client side decision (unless you have the AP forcibly disassociate a client - this is rarely done as it interrupts streaming / VoIP etc).
APs can provide the client with a list of neighboring APs and things like 802.11r speed up the reassociation process when there's a complex backend authentication process like 802.1x, but it's still entirely up to the client when to roam. Ubiquiti's early products did provide a "zero roaming" option which faked one big network, but this was very inefficient bandwidth-wise as it required all APs to share a single channel.
Proper network design such as minimizing TX power to encourage a client to roam due to a "weak" signal and maximizing coverage with lots of APs typically is the best way to get clients to roam reliably.
It's great in that it integrates with the unifi app and provides complete visibility.
However, I've had a couple of the USG fail under certain circumstances with enough bandwidth (>200mbps) or tickling a certain feature of the firewall. The EdgeRouter has been more reliable for me.
I'm not recommending against them, I suppose. I've just been bitten on a couple issues that I imagine have likely been corrected at this point. I was a bit disappointed that it took Ubiquiti a couple of years to stabilize the product.
I do have one on hand that I intend to replace my EdgeRouter with at some point so that I can manage all the devices with one app.
(edit) I think I was also waiting on IPv6 which it looks like they've added at this point:
I went with a US8-60W and a UAP to start doing Power over Ethernet everywhere I could (it's annoying that the USG doesn't support this). The USG hasn't arrived yet, but it'll let me do PPPoE, firewalling and stats for the network.
Good move on having not just two WANs, but two technologies. I've seen setups before where people have had two wans, from two different ISPs, but both cables ran down the same duct in the road. Single digger took them both out. It would be a pretty severe problem if fibre and wireless goes at the same time!
I assume you're not running a full BGP handoff to each ISP, so any existing sessions will die should your WAN die (as your lan get natted behind a different IP address). Presumably your nat state will move over in the case of router failure as it's a floating VM of some sort, so what's the failover time for each component? How does it compare to using say VRRP?
How are you detecting ISP failures -- are you pinging beyond the next hop, or are you assuming if you can ping/arp the upstream router, it's working? I've had failure scenarios with ISPs where the next hop works, but nothing past that.
What benefits are there of tcpproxy over something like nginx (for http/s) or dst-nat (for other connections)?
It looks like all your traffic defaults to WAN1, and only uses WAN2 in certain cases. Do you have the ability to send traffic for a given client to WAN2 by default?
What type of queuing are you using -- can 1 client hog all the bandwidth?
Look at a QWERTY keyboard. Start with your finger on the N key. Move it up to the key above, H. Move it up again to the Y key. Now once more, move it up to the 6.
"Above" here is kind of incorrect, it's actually "beyond". Colloquially we say the keys are above and below each other.
IIRC, the Unifi stuff as well as Meraki will do multiple ISPs. They do outbound NAT, and have a liveness check which is just a ping sent to the next hop. Ping fails, or the interface goes down and the device simply sends the traffic the other direction. Any established TCP sessions simply fail, but any new traffic will failover just fine.
I'm using this setup in my office. Easier than finding a last-mile type ISP that supports BGP.
If you use the same ISP you can probably get a routing working. But you're not going to get your own AS for a home network, even if you find an appropate ISP to provide you transit.
Next hop checking isn't always good enough. I had a 7 minute outage on one line last week, next hop was fine, but outside the ISP network it all fell apart.
Presumably the requirement includes having a couple of ISPs advertising your IP space, which I assume means having a /24. Can you still get those easily from ARIN?
The European version of ARIN allows IPv6-only networks.
/24 cost you about 3-6k$ each, depending on if you can spare a month to get a good price or need it announced tomorrow morning for your AS.
Do you have any idea how the "upstream port" detection works on Unifi gear? While I'm waiting for the piece of Unifi kit that does PPPoE and DHCP, I've got their switch plugged into my old router - straight away the switch was able to work out that this was a WAN connection and none of my other traffic gets routed through that. To set the same thing up on LEDE took hours.
If you want to feel more inferior about your home lab, https://www.reddit.com/r/homelab is a good source of safe-for-work porn and information on over-engineered setups.
weird... that link is blocked for me in the UK (redireting to contentcontrol.vodafone.co.uk).
Wonder what thats is about since its basically the network topology of a, albeit crazy, home network..
~10 years ago, I had a completely full 42U cabinet in my house, along with another 8U or so of gear and several devices that aren't measured in RU's (access points, Cable and DSL modems, VoIP phones, etc.).
Most of the gear was used for lab scenarios and such for various (Cisco, Juniper, et al) networking certs and was (mostly, but not completely) isolated from my "real" network. IIRC, I had ~35 VLANs at one point.
My extremely over-engineered home lab certainly served its purpose but I think I spent as much time maintaining it as I did actually using it, although it really came in handy for building out PoCs for projects I was handling at $work (my test/lab network at $work wasn't nearly as well-equipped as my home lab was!).
For the last several years, though, I've managed to get by with a single subnet that is shared by everything -- a few laptops, a couple desktops, a server hosting the handful of obligatory VMs, and, of course, the various phones, tablets, and streaming devices that are ubiquitous in all of our homes nowadays.
Just within the last few weeks, however, I've acquired a new server (2 x 10-core Xeons, 256 GB RAM, 4 "Enterprise" SSDs and 12 "Enterprise" HDDs (600 GB 15k SAS)), dug a couple switches out of storage in the garage, replaced my Internet router with a small industrial box running OpenBSD, and started building out a few more subnets for proper separation of various devices (I've twice been offered a 42U cabinet recently but, thus far, managed to say no!). Like probably most HN'ers, I've got a few VPSes spread out here and there as well. Finally, I've got a decent (but was over-built) 2U box in a rack at $work ($work == ISP) that I am planning to use to tie all of this together (using Wireguard, of course).
Yes, I'm fully aware that I'm in the beginning stages of a relapse. After these upcoming changes, however, I don't intend to "grow" this lab much larger (although this kinda stuff does just creep up on you sometimes).
I used to also have a 42U cabinet in my garage for several years. It housed a bunch of servers, mostly Dell poweredge but also some no-name boxes, plus some switches and other miscellaneous gear.
The power draw was too strong for my poorly garage circuit and after any power outage I had to power up the rack one device at a time - it was a massive pain. I also spent WAY too much time tinkering with it all, instead of actually using it in anger. Sure, it help me immensely doing PoCs for work or for my own learning, but it was always overkill. Funnily enough though, every other tech-head that saw it was envious, until I started detailing the horror stories of keeping it all running.
Thankfully Virtualisation became a usable and affordable platform for tinkerers, and I migrated everything (via a streamlined custom P2V process) to ESX, then later on migrated/rebuilt the VMs over to Hyper-V.
I now just run 2x Tower servers (HP 8xxx series workstations - dual Xeon based) and run 20+ VMs on each. Plus a single NAS for file storage. Life is so much easier... and the Garage is so much quieter.
Which company is that from, and did you include the surcharges? I'm on Georgia Power on the R-22 residential tariff, and I pay about twice as much as the "headline rate", once I've added them all in: http://www.psc.state.ga.us/calc/electric/GPcalc.asp
bradfitz, any idea why the soekris maxes out at 300 Mbps? I have been looking for info on that since thats my gateway (PFSense) at home and I think its limiting my speed since I recently got gigabit fiber. I might replace it with my espressobin running OpenWrt.
Hey, another Espressobin user! Did they ever fix the part where PCIe will kernel panic the machine?
I think you'll still hit bottlenecks with the switch on the Espressobin - Marvell hasn't enabled hardware acceleration, at least for the open source parts.
Very impressive. Thank you for the incredible write-up. I got a whole bunch of ideas for my own business's network architecture while going through your post as most of my needs match up with that of yours, as you elaborated in your post. Can't wait to post back the results post-implementation.
Have you also given yourself a mobile equivalent for those times when you are traveling, or when your primary environment is unsuitable and you must work at a place with public WiFi?
This guy doesn't appear to be a Twitch streamer. Aside from his rack having stickers for Go, Kubernetes, GitHub, and more, his Twitter description doesn't say anything about that.
If you're asking about in general would this be a good thing for a Twitch streamer... then I would say no. Mostly because most Twitch streamers are not going to know how to maintain something like this and they don't need all the servers.
If someone not so technical, Twitch streamers included, needed the redundant internet I would recommend something more along the lines of two ISPs like this guy (specifically over two technologies if possible: fiber and wifi, but that comes down to bandwidth requirements) but instead of going into multiple switches and having 3 servers running with VMs moving around just plug the two ISPs into something like the Unifi Security Gateway (USG) or USG Pro.
hardwired all the desktops and a few access points via cheap 1gbit hardware (literally found some at the thrift store/ebay), usually using tomato/shibby.
have a backup router.
battery backup on main routers/modem.
large external battery wire nutted to my desktop UPS.
NAS is an old laptop with battery intact, doubles as second display/machine.
use my phone via usb on my desktop if all else fails.
total cost, probably less than $100.
Oh, and I use a $5/month server for stuff that absolutely needs to be on full time. Otherwise the only external access is me occasionally remoting into my desktop and I am happy to stop and smell the flowers if that is interrupted briefly.
I have an even simpler setup: if my cable connection dies, I simply tether my phone to replace it. There are no UPSes because both the laptop (TP25 w/ 24 + 72 Wh batteries) and the phone (it's a Moto Z Play with a battery mod) have large enough batteries to last much longer than a domestic blackout in downtown Vancouver.
My laptop is enough for me to stay productive (it's a ThinkPad 25! very productive). Everything that needs to be online is on a Hetzner server I rent for all sorts of purposes so the 51 EUR monthly bill kind of spreads out.
I've been there, splurged on an alienware 17 a while ago, but mostly I only use it on the road now.
I went with desktop because I wanted everyone in the house to have a decent machine and I could get several I5s for less than $70 apiece (5 machines, one in each bedroom) and wanted easy/cheap upgrades for some of them, and they are all the same optiplex model, which makes my life easier.
I like my desktop setup a lot though, 3.3ghz I-5, 27" 1080, 16 gig ram, 1tb ssd, 8tb in "cold storage", g402 mouse, gt710 vid, clicky keyboard, Nubwo N2 headset, decent posture, 100+ fps gaming. Probably threw $500 at it above the initial $70 though, but most of the machines didn't get that treatment, but their users aren't using it to make a living either.
Posture wise I am normally using a Matias Ergo Pro mounted vertically and an Evoulent Verticalmouse and of course an external monitor. But, in a pinch / short travel I can just work on laptop. I tried desktop before but since everything I work on needs to be on laptop too, the necessary sync becomes old quick.
Fun solution, but seems like overkill for just about every home user.
I used to use a dual-WAN setup with cable modem + DSL backup. It worked well with automatic failover. I use a pfSense APU based router and, with no moving parts, it's been very reliable, nearly 4 years without any unscheduled downtime.
Then I moved and only had a single ISP to choose from, so my backup is to manually turn on a Wifi hotspot. I thought about using a cellular router with ethernet or a wifi connection to the hotspot for auto-failover, but it just wasn't worth the time and/or money to set it up -- if I'm home when the internet goes down, I can just switch to the hotspot, if I'm not home, then all I really lose is the ability to control the lights and thermostat remotely, not exactly a critical function.
I think that's quite the understatement. The thing that really stands out to me is the claim that all of that is only drawing 220W at idle. I'm curious if he means truly idle, like literally just booted up and not doing anything at all, zero traffic, etc. Or if that's the draw with stuff actually being used. Because 220W just for your home network is hilarious. I mean I feel dumb often because my little pfsense box pulls about 15W.
If you taunt it, even by accident, it has a habit of biting back. And as it has your data, you don't really feel too comfortable just nuking it in that case.
Sure. It's an extraordinarily complex system that's difficult to engineer correctly. It provides extraordinary durability, but the radius of failure isn't obvious. Pro tip, it's the entire cluster. As such, an issue with an OSD in one pool could potentially cause the entire cluster to have issues.
Recovery is difficult and there's no support unless you have a subscription from Redhat and also run RHEL plus their stable distro of Ceph (RedHat Storage or whatever). IIRC, they quoted me $90k for a petabyte of raw disk.
I haven't messed with it much in the last couple of years. Bluestore looked really promising. I've thought about taking a look at rook, but haven't yet.
If I were in a position to deploy a bunch of storage on bare metal again, I'd likely go with ceph. I do know that $GLORIOUS_FORMER_EMPLOYER ended up making the migration to ScaleIO and report being happy with it and having good performance.
I also thought having everything on UPS would allow me to keep an Internet connection during a power outage. Turns out that when the power goes out so does my ISP. Having a second ISP on LTE or Wifi like this setup may or may not be enough to fix that.
294 comments
[ 2.7 ms ] story [ 309 ms ] thread> to have a highly-available home Internet setup, with no SPOF (Single Point of Failure)
> to learn and have fun.
I attempted something similar to this in a 20U cabinet some time back. The biggest issue is the fan noise that 1U form factor servers and network gear produce, with their rather high RPMs. One can hear the noise across the other side of the house.
We've since switched to fanless network gear and ATX form factor servers with large diameter fans to keep the family happy. It definitely doesn't look as nice, though.
Not as cool though, and clearly not running any servers, but that's what things like AWS or Linode are for -- or for low power stuff, something like a fitlet [0]
[0] http://www.fit-pc.com/web/products/fitlet/
If your home is directly connected to their datacenter...
Not everyone has 10 Gbit upload with best peering!
I'm happy with a QNap as the only home server I need.
It's insane how quiet you can go with this approach, while remaining air-cooled. I know when my home server is running backup scripts because the noise increases at least tenfold when the hard drives spin. Fortunately, I have coordinated that to be only once a day -- the rest of the time the drives are in standby.
It's not whisper silent, especially during the summer when the fans on the R320 speed up to around 6000RPM (and this is with a E5-2430L) - but that's mostly due to my office remaining closed from the rest of the house leaving the ambient intake temperature around 75-80F (rest of the house stays at 72F). I'm probably going to stick with 2U's (probably R520's) when I start expanding again to lower the noise at higher temps, since the more equipment I add the more heat gets trapped in the room.
That was followed by a couple years of 100+ hour/year cumulative outages due to HACMP stability issues, and an environment that everyone was deathly afraid to touch.
The hardcore network engineer in me appreciates the detail in these kinds of solutions, but these days the practical side of me is satisfied with usability and maintainability of SPOF cable access with a manual failover to mobile hotspot on the rare occasions that drops offline.
With my luck, it would catastrophically fail while out of town, leaving the wife and kids without internet.
My dad set up a lot of complicated stuff like this. As people are prone to do, eventually he died, and it just made it difficult to troubleshoot technical problems for mom. So now the equipment sits in some corner, unused, because we replaced it all with something your average AT&T technician could troubleshoot.
Two ISPs, two networks. One called "main", one called "backup".
If "Main" fails, move over to "Backup", either with a cable, or on a different SSID.
That's not an unreasonable solution, considering most people already pay two ISPs (one fixed, and another for their phone/tablet). When your home wifi goes down, you're going to fall-back to your mobile anyway. I'm thinking of getting an extra data SIM, an LTE modem and do auto-failover.
--edit--
My needs are somewhat unique - my traveling laptop is on its last legs (and will be replaced by a cheap chromebook. Desktops/servers get better bang for the buck compared to laptops. Go figure!), so I tunnel onto a server at home for heavy-lift computing. If the internet fails when I'm not home, I'd be left stranded (and this has happened).
From a practical point of view I think it's silly to do such a thing for a residential situation, but I can appreciate using it as a learning experience for building systems like this.
3g is good enough backup for me, but for the office we go for two routers two isps and vrrp on the lan side, load balance across the wans, with failover to the other one.
I would not discount the possibility completely. But I judge it unlikely.
Then I'd put the primary router on the wired line, the other one on a 4G sim which did nothing but heartbeats unless the wired line went down. If the wired line shut down, traffic would reroute via 4G within 10 seconds or so. If the primary router went down, the backup router would take over in a similar time frame. Might put some capping on the 4G router to the netflix/etc boxes to keep bandwidth costs down.
UPS would be about 10W, so £45 for a 4 hour one. Possibly look at renewable energy of some sort to keep the UPS going during an extended outage.
I'd then VRRP on the lan side with primary on the main router (which would have a backup route via the secondary router)
Cloud based VM to do monitoring/alerting and land outgoing openvpn tunnels from both routers to allow secure remote access.
£170, £10 a month plus main ISP, and an hour of config.
However in reality having an ISP provided router and showing them how to tether in a problem works fine. OK, they lose their devices if the main circuit goes off, but running those over 4G can be pricey.
[1] https://en.wikipedia.org/wiki/Superiority_(short_story)
>because of its own organizational flaws and its willingness to discard old technology without having fully perfected the new.
Link to the story online.
ADSL Modem > Firewall > Router > Web/DB servers
It was basic, but it worked. Our web servers were mission critical, but as a B2B business they, and the ADSL connection, didn't sustain a heavy load. The only issues we had over several years were with the ADSL modem. Everything else just worked.
When we moved office we moved our servers to a co-hosting centre with an upgraded network setup with all sorts of backup and redundancy. Every week something went wrong. Sometimes simple is best.
Even ISPs and CDNs I worked with sometimes have surprisingly uncomplicated redundancy systems (sometimes just a handful of small routers they are very much ready to power down to cut over to backup paths or bring up new paths) and often they do not use the more complicated methods.
The catch with complicated redundancy is there is always a very close relationship or protocol or something between redundant components, bet it storage systems, network systems, anything. Inevitably a system goes down or loses its mind and takes it's redundant peers with it.... every new system you introduce is one more piece that could reach out and take everyone else with it. I saw it time and again, and again...
My home network is built with Mikrotik kit which is priced where it's affordable to have spares. I have yet to encounter a failure, but could drop in a new router in a couple of minutes with the saved configs.
I have SNMP monitoring feeding from telegraf into influxdb on an RPI. Dashboard rendered with Grafana on PC. Also have telegraf pinging to all 24x7 devices and collecting data from electricity meter, smartplugs, and Nests. It's been fun to do.
Web, Power, Internet, Network, Military systems at scale use reliable redundancy and work w/ very little downtime.
[1] i.e. the systems that interconnect the multiple redundant system, detect failures, redirect traffic, etc.
Turtles all the way down, I guess.
Indeed it is important in this case of course that this does not happen :) To see the increased reliability and P(glue failure)<P(single failure) you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Another adequate expression to apply here is
"Who watches the watchmen?"
The answer again is the watchmen must watch themselves and be very reliable.
On this topic I recommend von Neumann's (the brilliant mathematician) "Computer and the brain" book, where he explores how computing systems can be reliably interconnected and how those failure probabilities interact. He was interested on how the brain could be so robust to failure -- don't worry there's no time spent speculating on how the brain works, instead he derives from first principles properties of reliable computing components, and possible reliable designs (the brain's unknown internal workings at the time, and now to a lesser extent, would follow as a special case). He used this same approach in analyzing the principles of life, where he came up with a self-replicating machine with a tape encoding of itself, predating the discovery of DNA -- it's a very inspiring and powerful approach. Unfortunately he could not complete 'Computer and the Brain', he was in declining health due to cancer and died while writing it. What was left is still very interesting imo. He is one of those giants whose shoulders we can sit on to peek over the horizon :)
As a caution against tenanting the deployment tools in-band, I'm reminded of an incident I witnessed about five years back. Company was moving their compute from on-prem to colo datacenters. Pretty good, mature setup: Almost entirely virtualized, 10Gb iSCSI SAN, credentials managed via a dedicated COTS tool, etc. They got most things over-the-wire to the DC. But the final migration had to be done cold - Shut the last bits down that were keeping everything running, move them to the DC and power back on.
Everything went very well until the SAN wouldn't come up. To get into the SAN and troubleshoot they needed the domain, which wasn't available. They had a local account on the SAN, the key for that was safely stored in the password manager. Which was a virtual machine. On the hyper visors. That wouldn't come up until the SAN was booted. Oops!
OK, that's a very obvious foot-in-mouth, in hindsight. As a more likely example, how about the Amazon S3 outage a few years back that wasn't reported on the status page, because the images for the status page were stored on... S3 :D
>you need to assure the glue systems are very simple and well built -- and preferably they need to be much smaller than the system you're protecting.
Absolutely agree.
Most half-baked redundant systems I've seen are a result of "I want four nines, but I only want it to cost 20% more than a two or three nines solution" type thinking.
I’ve also seen well built and maintained HA systems work exactly as desired.
As a general rule, the cost of building and operating a reliable HA solution is not 2x, but at least 10x. If the system being protected is not worth that, you’ll very likely find the MTTR acronym far easier to catch than the rather more slippery HA.
Nobody understood exactly how the cluster worked to the point that a correction my boss made on the physical connections, made us loose a couple of million of dollars in transactions not processed.
The funny part is, when the cluster was working fine, a takeover took at least 20 minutes. During that time nothing was "available". The thing is, no matter what, SWIFT Alliance took that time to properly close and open the DB.
* Cantenna/laser link to a house some blocks away to avoid local WAN link disruption
* For less performance-intense networks, remove the physical impediments: 2 routers, each with 1 APC, connected to 2 separate power circuits, connected to 2 WAN links, providing 2 radios each. No switch to go down or cables to trip over, redundancy of access point, redundancy of frequency/radio, redundancy of WAN link, redundancy of power. Hardware-wise this is pretty cheap and still highly available. If the routers are cheap, use a hardware watchdog.
His gig internet is $80 a month: https://www.centurylink.com/fiber/plans-and-pricing/seattle-...
His wifi backup internet is $40: http://www.gigabitseattle.com/residential-services
He specifically states the setup draws 220 watts at idle and that his electricity costs $0.0974/kWh. So 22024/10000.0974 = 0.514272 per day, or about $15.40 a month at idle.
So around $135 a month.
yea if it's idle the entire month, which is doubtful. but even if it's not, it's not likely to be too much more than the $135 you calculated. I figured the internet service would have been more, since the rest of us get screwed by our ISPs on costs.
No modem, just an Ethernet drop into your home.
For those that assume that was just a joke on escalating size, the joke was actually made it a real thing by Netflix when they actually named the component that randomly shuts down not just services, but entire AWS availability zones of Netflix services.
Child with a water balloon? Hope you have multiple data-closets in your house...
I use the UAP-AC and an EdgeRouter. The EdgeRouter has relatively sophisticated capabilities for a piece of consumer great network gear. I have GigE fiber to the home and get ~900Mbps through the router (and ~400Mpbs through the access points).
I generally don't recommend the USG, which has similar functionality and is integrated with the Unifi management platform.
What advantage do you get with that versus just running a bunch of APs with the same SSID/password bridging to a single router? I do that with 3 cheap tp-link routers (1 as router, 2 as APs) and LEDE and both my laptop and phone work seamlessly. At one point I considered actually doing full 802.11r AP roaming but the only actual use case I had for that was doing VoIP calls while roaming between APs with no drops. Everything else works fine with the small interruption of switching APs.
> Note that UniFi Fast Roaming is not a direct implementation of 802.11r - it is a solution taking inspiration from 802.11r, with a few key proprietary differences. We've found that Fast Roaming provides about 90% of the roaming improvement offered by BSS Transition. However, Fast Roaming does not require client support, allowing backwards compatibility with all clients.
https://help.ubnt.com/hc/en-us/articles/115004662107-UniFi-F...
The app can be self-hosted, run in the cloud, or on something they call a "cloud key" that's not much more than a Raspberry Pi.
I've run it on a Pi3, and it's a little bit laggy, but tolerable. I prefer to run it on my little x86 server.
Both the self-hosted and the cloud version can be managed remotely, which is neat.
There are a host of other benefits, but given the price (around $100 per AP), I see no reason to use the more commonplace consumer grade stuff. Check out the unifi demo here:
https://demo.ubnt.com/manage/site/outlets/dashboard
APs can provide the client with a list of neighboring APs and things like 802.11r speed up the reassociation process when there's a complex backend authentication process like 802.1x, but it's still entirely up to the client when to roam. Ubiquiti's early products did provide a "zero roaming" option which faked one big network, but this was very inefficient bandwidth-wise as it required all APs to share a single channel.
Proper network design such as minimizing TX power to encourage a client to roam due to a "weak" signal and maximizing coverage with lots of APs typically is the best way to get clients to roam reliably.
However, I've had a couple of the USG fail under certain circumstances with enough bandwidth (>200mbps) or tickling a certain feature of the firewall. The EdgeRouter has been more reliable for me.
I'm not recommending against them, I suppose. I've just been bitten on a couple issues that I imagine have likely been corrected at this point. I was a bit disappointed that it took Ubiquiti a couple of years to stabilize the product.
I do have one on hand that I intend to replace my EdgeRouter with at some point so that I can manage all the devices with one app.
(edit) I think I was also waiting on IPv6 which it looks like they've added at this point:
https://medium.com/@poolski/setting-up-ipv6-on-unifi-securit...
I went with a US8-60W and a UAP to start doing Power over Ethernet everywhere I could (it's annoying that the USG doesn't support this). The USG hasn't arrived yet, but it'll let me do PPPoE, firewalling and stats for the network.
I assume you're not running a full BGP handoff to each ISP, so any existing sessions will die should your WAN die (as your lan get natted behind a different IP address). Presumably your nat state will move over in the case of router failure as it's a floating VM of some sort, so what's the failover time for each component? How does it compare to using say VRRP?
How are you detecting ISP failures -- are you pinging beyond the next hop, or are you assuming if you can ping/arp the upstream router, it's working? I've had failure scenarios with ISPs where the next hop works, but nothing past that.
What benefits are there of tcpproxy over something like nginx (for http/s) or dst-nat (for other connections)?
It looks like all your traffic defaults to WAN1, and only uses WAN2 in certain cases. Do you have the ability to send traffic for a given client to WAN2 by default?
What type of queuing are you using -- can 1 client hog all the bandwidth?
And finally, what keyboard layout is 6 above N?
"Above" here is kind of incorrect, it's actually "beyond". Colloquially we say the keys are above and below each other.
I'm using this setup in my office. Easier than finding a last-mile type ISP that supports BGP.
Next hop checking isn't always good enough. I had a 7 minute outage on one line last week, next hop was fine, but outside the ISP network it all fell apart.
ARIN, at least, will happily assign you an ASN assuming you 1) meet the multi-homing requirements and 2) pay the bill for it.
Getting an AS is easy. Getting portable IPv4 address space or an LOA to readvertise is more tricky.
(Of course, you can start your own company for something like 13 quid/year. Now that I have one maybe I should revisit that.)
Seems like a lot of effort to ensure your ssh session doesn't drop
yikes
Most of the gear was used for lab scenarios and such for various (Cisco, Juniper, et al) networking certs and was (mostly, but not completely) isolated from my "real" network. IIRC, I had ~35 VLANs at one point.
My extremely over-engineered home lab certainly served its purpose but I think I spent as much time maintaining it as I did actually using it, although it really came in handy for building out PoCs for projects I was handling at $work (my test/lab network at $work wasn't nearly as well-equipped as my home lab was!).
For the last several years, though, I've managed to get by with a single subnet that is shared by everything -- a few laptops, a couple desktops, a server hosting the handful of obligatory VMs, and, of course, the various phones, tablets, and streaming devices that are ubiquitous in all of our homes nowadays.
Just within the last few weeks, however, I've acquired a new server (2 x 10-core Xeons, 256 GB RAM, 4 "Enterprise" SSDs and 12 "Enterprise" HDDs (600 GB 15k SAS)), dug a couple switches out of storage in the garage, replaced my Internet router with a small industrial box running OpenBSD, and started building out a few more subnets for proper separation of various devices (I've twice been offered a 42U cabinet recently but, thus far, managed to say no!). Like probably most HN'ers, I've got a few VPSes spread out here and there as well. Finally, I've got a decent (but was over-built) 2U box in a rack at $work ($work == ISP) that I am planning to use to tie all of this together (using Wireguard, of course).
Yes, I'm fully aware that I'm in the beginning stages of a relapse. After these upcoming changes, however, I don't intend to "grow" this lab much larger (although this kinda stuff does just creep up on you sometimes).
I used to also have a 42U cabinet in my garage for several years. It housed a bunch of servers, mostly Dell poweredge but also some no-name boxes, plus some switches and other miscellaneous gear.
The power draw was too strong for my poorly garage circuit and after any power outage I had to power up the rack one device at a time - it was a massive pain. I also spent WAY too much time tinkering with it all, instead of actually using it in anger. Sure, it help me immensely doing PoCs for work or for my own learning, but it was always overkill. Funnily enough though, every other tech-head that saw it was envious, until I started detailing the horror stories of keeping it all running.
Thankfully Virtualisation became a usable and affordable platform for tinkerers, and I migrated everything (via a streamlined custom P2V process) to ESX, then later on migrated/rebuilt the VMs over to Hyper-V.
I now just run 2x Tower servers (HP 8xxx series workstations - dual Xeon based) and run 20+ VMs on each. Plus a single NAS for file storage. Life is so much easier... and the Garage is so much quieter.
What box and how's it performing?
I'm in the Atlanta area. $0.07181/kWh.
For example, Grant County PUD for residential customers: $0.04547 per kWh
I think you'll still hit bottlenecks with the switch on the Espressobin - Marvell hasn't enabled hardware acceleration, at least for the open source parts.
If you're asking about in general would this be a good thing for a Twitch streamer... then I would say no. Mostly because most Twitch streamers are not going to know how to maintain something like this and they don't need all the servers.
If someone not so technical, Twitch streamers included, needed the redundant internet I would recommend something more along the lines of two ISPs like this guy (specifically over two technologies if possible: fiber and wifi, but that comes down to bandwidth requirements) but instead of going into multiple switches and having 3 servers running with VMs moving around just plug the two ISPs into something like the Unifi Security Gateway (USG) or USG Pro.
My home setup:
hardwired all the desktops and a few access points via cheap 1gbit hardware (literally found some at the thrift store/ebay), usually using tomato/shibby.
have a backup router.
battery backup on main routers/modem.
large external battery wire nutted to my desktop UPS.
NAS is an old laptop with battery intact, doubles as second display/machine.
use my phone via usb on my desktop if all else fails.
total cost, probably less than $100.
Oh, and I use a $5/month server for stuff that absolutely needs to be on full time. Otherwise the only external access is me occasionally remoting into my desktop and I am happy to stop and smell the flowers if that is interrupted briefly.
My laptop is enough for me to stay productive (it's a ThinkPad 25! very productive). Everything that needs to be online is on a Hetzner server I rent for all sorts of purposes so the 51 EUR monthly bill kind of spreads out.
I went with desktop because I wanted everyone in the house to have a decent machine and I could get several I5s for less than $70 apiece (5 machines, one in each bedroom) and wanted easy/cheap upgrades for some of them, and they are all the same optiplex model, which makes my life easier.
I like my desktop setup a lot though, 3.3ghz I-5, 27" 1080, 16 gig ram, 1tb ssd, 8tb in "cold storage", g402 mouse, gt710 vid, clicky keyboard, Nubwo N2 headset, decent posture, 100+ fps gaming. Probably threw $500 at it above the initial $70 though, but most of the machines didn't get that treatment, but their users aren't using it to make a living either.
I used to use a dual-WAN setup with cable modem + DSL backup. It worked well with automatic failover. I use a pfSense APU based router and, with no moving parts, it's been very reliable, nearly 4 years without any unscheduled downtime.
Then I moved and only had a single ISP to choose from, so my backup is to manually turn on a Wifi hotspot. I thought about using a cellular router with ethernet or a wifi connection to the hotspot for auto-failover, but it just wasn't worth the time and/or money to set it up -- if I'm home when the internet goes down, I can just switch to the hotspot, if I'm not home, then all I really lose is the ability to control the lights and thermostat remotely, not exactly a critical function.
I think that's quite the understatement. The thing that really stands out to me is the claim that all of that is only drawing 220W at idle. I'm curious if he means truly idle, like literally just booted up and not doing anything at all, zero traffic, etc. Or if that's the draw with stuff actually being used. Because 220W just for your home network is hilarious. I mean I feel dumb often because my little pfsense box pulls about 15W.
Clearly hasn't been bitten by it, yet.
I mean... I love Ceph, too, but I don't ever want to run it again.
Recovery is difficult and there's no support unless you have a subscription from Redhat and also run RHEL plus their stable distro of Ceph (RedHat Storage or whatever). IIRC, they quoted me $90k for a petabyte of raw disk.
I haven't messed with it much in the last couple of years. Bluestore looked really promising. I've thought about taking a look at rook, but haven't yet.
If I were in a position to deploy a bunch of storage on bare metal again, I'd likely go with ceph. I do know that $GLORIOUS_FORMER_EMPLOYER ended up making the migration to ScaleIO and report being happy with it and having good performance.