This is super cool! I'd love to get this working for my home servers as well one day (priority number one is making them accessible outside my LAN again).
I'm currently using a Digital Ocean droplet with wireguard and my home server connects to that VPS via wireguard site-to-site. I have Nginx on the VPS and do a proxy_pass to my local services. Just set it up a week ago and works great, very simple once you understand the setup.
Wireguard requires a more complex setup than for IPSec, believe it or not.
To give my home router a static IP address, I use a Vultr VPS and pay for an additional IP address. I then use IPSec encapsulation to forward traffic for the second IP address directly to the router, which initiates the IPSec SA setup. The router has a loopback device configured with the static IP address; the routable static address is effectively local, even from the kernel's perspective. The IPSec rules handle forwarding of everything, so I don't even need packet filtering (e.g. OpenBSD PF in my case) rules.
No need to fiddle with private address ranges, NAT'ing, reverse proxying, or packet filtering of any sort. The IKEv2 setup is a single line on the VPS and a single line on the router, though OpenBSD's OpenIKED (as well as their isakmpd fork for IKEv1) is awesome that way; StrongSwan and other Linux IKE daemons require a more verbose configuration on account of their key-value pair syntax, but in any event it's still relatively simple.
Unfortunately I can't fiddle with the router I'm behind, so I think the wireguard route is the way to go. I also like the wireguard approach because as far as I can tell it should be completely portable -- if I move to be behind another router/NAT, I can just plug in my box (it's a laptop with a broken screen), run my dynamic DNS update script (I'm using DuckDNS) and I'll be off to the races, as I understand it.
I just happen to put my ISP's (Sonic's) router in passive mode and use a PC Engines APU2 running OpenBSD for NAT'ing my home network, but the same IPSec setup works even if the termination point is behind a NAT gateway. Sonic doesn't offer static IP addresses, so the VPS could never directly connect to my router unless I fiddled with dynamic DNS or scraped together some other hack, which IMO is far more hassle than it's worth. Also, I've used this exact same setup before except with the termination point inside a corporate LAN, tucked away on someone's desk. IPSec+IKE has done NAT traversal and dead peer detection for many, many years, notwithstanding that some IKE implementations have sucked at it.
Wireguard is just a better wire protocol for moving packets, but IMO it's mostly a solution in search of a problem. Most of the complexity that people disdain about IPSec actually comes from IKE, but it's largely irreducible complexity. IKE is what initiates and negotiates security associations based on abstract flow rules--in this case 0.0.0.0/0 <-> W.X.Y.Z, where the latter is the static, routable address. Wireguard doesn't diminish the need for this layer, which is why you've been forced to hack a complete solution using a reverse proxy. IKE is far more mature than the experimental additional software Wireguard developers have been cooking up to automate tasks like this (abstract flows, key management, etc), and definitely more mature than the homebrew solutions people put together.
I bring this up because I think IPSec has gotten an undeserved bad rap. If people make do with Wireguard + hacks, great, but this particular scenario is a great example that shows the strengths of IPSec+IKE and the weakness of Wireguard; a reality people don't hear about given the unabashed enthusiasm for Wireguard.
I was planning to do exactly that! I already have a Digital Ocean droplet with Nginx on it (serving a couple of very-low-traffic apps) ready to go, I just haven't had time to dive into the problem yet. Glad to hear it worked for you!
I need to do this because port forwarding isn't working with my current ISP/router/etc setup (not sure where the problem lies). Did you forward the wireguard port to your home server, or another approach? I did some research and it looks like there's a "keep-alive" option that hopefully should get my wireguard connection open through NAT.
I didn't have to touch my router settings at all. There are a million guides out there, but I used this one [0]. I'm not sure about the keep alive, but I'm assuming it's this setting that they mention in the guide:
# Ensures that your home router does not kill the tunnel, by sending a ping
# every 25 seconds.
PersistentKeepalive = 25
I set up the server .conf, ran wg-quick up wg0, and then did the same on the client and it worked like a charm. Just make sure to open your UDP port for Wireguard on your VPS, that cost me a solid 10 minutes of confusion.
Mine went 3 moves back. I had a great collection of real RS232 cards and USB cables from various years and am now really missing them since I've gotten into ESP32.
Interesting but why not a kvm rack unit from ebay? Cheaper and easier only issue is most vendors stop supporting the switches and the software on them can be complete garbage (only works with firefox version x or IE version y).
> Maybe I missed it, but what device is on the other end of the COM connection? His router?
The plan was to use a Raspberry Pi to access the two PCs (via USB-to-serial adapters), though he hasn't made it to that point yet ("I have not used it for this project yet, but it is still in my plans."). Right now, the two PCs are just connected to each other, back-to-back.
> What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection?
I would assume that he does just use SSH -- under normal circumstances.
It's when SSH access isn't available that access via the serial console would be used -- such as the network connection being down or b0rked, when the kernel (or required) modules failed to boot/build/load properly, a bad firewall change has locked you out, and so on. This method even provides access to the bootloader.
If your BIOS supports "console redirection" (to a serial port), you also gain access to the BIOS, can watch the startup (POST) process, or even change the boot device -- in order to fire off a kickstart installation, for example.
He mentioned that his PCs don't have ILO/DRAC/IPMI, so this is his backup solution. -- remember that his machines are headless.
We are searching for a new serial to ethernet solution which is not as expensive as most B2B products. Right now this [1] would be our dream product. We are also looking into a combination of a mini-pc (like PC-Engines [2]) with two of these miniPCIe serial cards [3].
If you have better recommendations feel free to share them :)
Old multi-port serial consoles can often be found used quite cheap.
Personally, I'm a big fan of the OpenGear units. They're *much^ more than just basic serial consoles and are quite extendable. You can even write your own scripts to interact with them, trigger them on specific events, and so on.
I've used Startech FTDI based multi-port serial to USB adapters along with a Mikrotik Routerboard to make serial-to-ethernet or serial-to-LTE boxes. The Mikrotik boxes expose the serial port on a TCP port for you to connect to.
Not only server. If you have a Q-series desktop motherboard with AMT you can enable Serial-Over-Lan and access the machine remotely (and power on/off or reset it as well). Q series chipsets are also found on Fujitsu/Dell/HP prebuilts which can be picked up quite cheaply when companies decide to upgrade. For servers you usually have IPMI/iDrac etc. which uses more power, but usually has a better (web)interface.
Yep. I've just picked up a few second hand HP EliteDesk 800 G2 desktops with the intention of turning them into a homelab for k8s learning. They have AMT and the web power control, VNC based KVM and SoL work well.
There is no BIOS serial redirection, so SoL is no use until the OS starts booting, and grub has a bug[0][1][2] which means it can't find the AMT SoL when booting in EFI mode. But SoL works fine as soon as the kernel starts, and the KVM VNC works fine during BIOS and GRUB if I have an emergency.
One gotcha which would catch someone out who wasn't expecting it - the onboard video disables itself when no monitor is connected, so the AMT KVM just shows a black screen. This can be solved by DisplayPort or VGA "ghost" dongle in the back of the machine to talk EDID and pretend to be a screen.
I'm a big fan of conserver, and intend to use it in this project to spawn amtterm processes to connect to the SoL ports.
... and, if you don't have Serial-over-LAN (SOL) access via IPMI, etc., you can always just wire the serial port up to another host to give yourself access to that serial port remotely.
Then, just SSH in to the other host and use something like "minicom" to connect directly to the serial port -- or use "conserver" (mentioned in TFA) when you've got multiple consoles and/or multiple users to deal with simultaneously.
Plenty of HN'ers are all-in on "cloud" but for anyone dealing with physical devices (routers, servers, whatever), remote access to the console can be a lifesaver at times!
For a minimal serial-port access program, I usually prefer the classic cu(1) (originally a part of the UUCP program suite); miniterm is basically an old-DOS-style program in the same vein as Telix or PC-Talk, and is generally too heavyweight for my purposes.
"conserver" (which the author is hsing) is really handy for quickly and easily accessing and/or monitoring the (serial) console of not just physical machines (like the author is doing) but also virtual machines as well.
I've used it with virtual machines running on both vSphere and QEMU/KVM to make their serial consoles available over the network... to access the serial console you can just connect to the corresponding TCP port. Just make sure to lock down / restrict access appropriately and ensure you don't send sensitive info over the network in clear-text!
It works the same way with physical devices too (Cisco devices, primarily). It's great when you want to a "permanent record" (on disk) of any output that is sent to the console.
As mentioned, it's really nice when you need to provide "shared" console access and/or access to multiple devices and/or users simultaneously. It's really an under-rated tool.
> One small disadvantage not addressed is that neither of the two motherboards support BIOS access via COM1, which is a bummer.
I've always wondered if you could burn SGABIOS (https://code.google.com/archive/p/sgabios/) to a PCI(e) card ROM to get this working on real hardware instead of just in QEMU.
Why not use a Raspberry Pi populated with several USB to console adaptors, there are USB console cables with up to four serial ports per connection which would result in 16 console ports per Pi.
> more than a few times I have gotten into a situation where I have attempted to preform a system update on one of the pair remotely from cafe or sofa and the PC failed to come back up after a reboot.
This happened to me as well. In fact it happened so often that I took all of the critical network software off the home server and run it on an edge router-x.
Also, setting “power on after power loss” and running the computer through a smart plug is a good idea. When it gets jammed, you can at least power cycle it remotely and it attempts to reboot.
41 comments
[ 2.3 ms ] story [ 94.6 ms ] threadTo give my home router a static IP address, I use a Vultr VPS and pay for an additional IP address. I then use IPSec encapsulation to forward traffic for the second IP address directly to the router, which initiates the IPSec SA setup. The router has a loopback device configured with the static IP address; the routable static address is effectively local, even from the kernel's perspective. The IPSec rules handle forwarding of everything, so I don't even need packet filtering (e.g. OpenBSD PF in my case) rules.
No need to fiddle with private address ranges, NAT'ing, reverse proxying, or packet filtering of any sort. The IKEv2 setup is a single line on the VPS and a single line on the router, though OpenBSD's OpenIKED (as well as their isakmpd fork for IKEv1) is awesome that way; StrongSwan and other Linux IKE daemons require a more verbose configuration on account of their key-value pair syntax, but in any event it's still relatively simple.
Wireguard is just a better wire protocol for moving packets, but IMO it's mostly a solution in search of a problem. Most of the complexity that people disdain about IPSec actually comes from IKE, but it's largely irreducible complexity. IKE is what initiates and negotiates security associations based on abstract flow rules--in this case 0.0.0.0/0 <-> W.X.Y.Z, where the latter is the static, routable address. Wireguard doesn't diminish the need for this layer, which is why you've been forced to hack a complete solution using a reverse proxy. IKE is far more mature than the experimental additional software Wireguard developers have been cooking up to automate tasks like this (abstract flows, key management, etc), and definitely more mature than the homebrew solutions people put together.
I bring this up because I think IPSec has gotten an undeserved bad rap. If people make do with Wireguard + hacks, great, but this particular scenario is a great example that shows the strengths of IPSec+IKE and the weakness of Wireguard; a reality people don't hear about given the unabashed enthusiasm for Wireguard.
I need to do this because port forwarding isn't working with my current ISP/router/etc setup (not sure where the problem lies). Did you forward the wireguard port to your home server, or another approach? I did some research and it looks like there's a "keep-alive" option that hopefully should get my wireguard connection open through NAT.
[0] https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard...
I used to have a glorious box of ram, spare drives, and every cable and adapter imaginable.
My box also mysteriously disappeared some time after I got married. I feel your pain :)
Sometimes I wonder what would have happened if we didn't already have kids.
I recall at one job we had very long runs of rs232 cables to some vt100's which where very sensitive to electrical charges.
To the point where one of our electronics shop brought the official Vt100 manuals so they could repair any blown chips.
DEC field circus loved us :-) It was the 11/03 covered I black coal dust that really upset them.
I'm just saying, this is not normal or acceptable behaviour.
My box of concert tickets from every show I ever saw disappeared.
'graph 2 of TFA.
What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection?
The plan was to use a Raspberry Pi to access the two PCs (via USB-to-serial adapters), though he hasn't made it to that point yet ("I have not used it for this project yet, but it is still in my plans."). Right now, the two PCs are just connected to each other, back-to-back.
> What's the reason against SSH'ing from the gateway to the servers instead of the physical COM connection?
I would assume that he does just use SSH -- under normal circumstances.
It's when SSH access isn't available that access via the serial console would be used -- such as the network connection being down or b0rked, when the kernel (or required) modules failed to boot/build/load properly, a bad firewall change has locked you out, and so on. This method even provides access to the bootloader.
If your BIOS supports "console redirection" (to a serial port), you also gain access to the BIOS, can watch the startup (POST) process, or even change the boot device -- in order to fire off a kickstart installation, for example.
He mentioned that his PCs don't have ILO/DRAC/IPMI, so this is his backup solution. -- remember that his machines are headless.
If you have better recommendations feel free to share them :)
[1] https://freetserv.github.io/ [2] https://www.pcengines.ch/apu2c2.htm [3] https://www.delock.de/produkte/G_95244/merkmale.html
Personally, I'm a big fan of the OpenGear units. They're *much^ more than just basic serial consoles and are quite extendable. You can even write your own scripts to interact with them, trigger them on specific events, and so on.
https://imgur.com/a/fdMfjdl
There is no BIOS serial redirection, so SoL is no use until the OS starts booting, and grub has a bug[0][1][2] which means it can't find the AMT SoL when booting in EFI mode. But SoL works fine as soon as the kernel starts, and the KVM VNC works fine during BIOS and GRUB if I have an emergency.
One gotcha which would catch someone out who wasn't expecting it - the onboard video disables itself when no monitor is connected, so the AMT KVM just shows a black screen. This can be solved by DisplayPort or VGA "ghost" dongle in the back of the machine to talk EDID and pretend to be a screen.
I'm a big fan of conserver, and intend to use it in this project to spawn amtterm processes to connect to the SoL ports.
0: https://savannah.gnu.org/bugs/?42026
1: https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-se...
2: https://wiki.networksecuritytoolkit.org/index.php/HowTo_Head...
Then, just SSH in to the other host and use something like "minicom" to connect directly to the serial port -- or use "conserver" (mentioned in TFA) when you've got multiple consoles and/or multiple users to deal with simultaneously.
Plenty of HN'ers are all-in on "cloud" but for anyone dealing with physical devices (routers, servers, whatever), remote access to the console can be a lifesaver at times!
I've used it with virtual machines running on both vSphere and QEMU/KVM to make their serial consoles available over the network... to access the serial console you can just connect to the corresponding TCP port. Just make sure to lock down / restrict access appropriately and ensure you don't send sensitive info over the network in clear-text!
It works the same way with physical devices too (Cisco devices, primarily). It's great when you want to a "permanent record" (on disk) of any output that is sent to the console.
As mentioned, it's really nice when you need to provide "shared" console access and/or access to multiple devices and/or users simultaneously. It's really an under-rated tool.
I've always wondered if you could burn SGABIOS (https://code.google.com/archive/p/sgabios/) to a PCI(e) card ROM to get this working on real hardware instead of just in QEMU.
Apparently someone did try this, so it might just work: https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_...
Maybe I should dig up some old flashrom supported NIC card or something.
This happened to me as well. In fact it happened so often that I took all of the critical network software off the home server and run it on an edge router-x.
Also, setting “power on after power loss” and running the computer through a smart plug is a good idea. When it gets jammed, you can at least power cycle it remotely and it attempts to reboot.