Show HN: Wag, MFA and Enrollment for WireGuard (github.com)
I've been building this project as both a side project and my job for a little while now. The rationale behind it is while wireguard is a fantastic protocol cryptographically it leaves a lot to be desired when it comes to enrollment and end user device security.
Obviously instead of using an off the shelf solution like tailscale, I decided to reinvent the wheel which has honestly been quite fun with learning about eBPF, and recently clustering and HA with etcd!
The most recent version (in the docker container) contains about 6 months of very new work bringing it all from sqlite3 to etcd. So please be forgiving if it does some weird things!
Hope you all enjoy!
P.s Im not a web developer and any tips on that front to make it less teeth pullingly awful are welcome!
67 comments
[ 2.9 ms ] story [ 176 ms ] threadYou'd think that sites choosing wireguard would have a more modern setup and might make heavy use of (self service) ULAs.
Is there something specific you were thinking about when you mention ULAs?
The scenario I'm imagining is: someone opens the TOTP entry UI in their browser, opens devtools, and starts to loop through all possible TOTP codes.
If you have username and password and are able to force the TOTP in the 60s window, the TOTP would be useless imho.
https://github.com/NHAS/wag/blob/cdbdbec3393fa86bf6c823117c8...
Specifically to force people to have a bit of a think as to why their device is trying to force auth to begin with, as it indicates an endpoint compromise.
I didn’t take it as malicious, but trying to understand more about this method. I’d love for the author to tell us a bit more about how it works. I’m curious about what obstacles the author hit and how they got around them.
Note: re: the flagged sibling comment. Yeah, that one doesn’t get the benefit of the doubt and was out of bounds.
I just didn’t get that vibe from the now flagged comment by @aragilar. I saw it as a genuine curiosity about the design choices. Maybe I was wrong.
How it works:
In short, Wag adds an eBPF program to a WireGuard device that it instantiates. The eBPF program uses a number of hash maps and LPM (longest prefix matching trie) maps to determine the policies that are applied traffic coming in on the wireguard device. These policies based on the ACLs defined per user/group, and contain MFA/Allow/Deny rules which require mfa, allow without auth and deny always respectively.
Wag also watches all the wireguard peers ingress IP addresses, and when an address changes it deauthenticates the user and requires the user to complete a login challenge. This is done by basically setting a bit in the maps exposed to eBPF that says "unauthorised"
Challenges:
First and foremost with WireGuard there is no good way of determining if an "external ip" i.e where the user is connecting from has changed. There was a patch set submitted for review in 2022~ that was never actually added to the kernel that would have added netlink compatibility and thus event based notification that things had changed, but alas that was never reviewed by Jason Donenfeld and has quietly died the death.
Secondly was defining multiple policies per route was quite difficult as eBPF doesnt do dynamic memory even in userland exposed maps and I wanted multiple rules per route, i.e you might allow port 80/tcp when MFA has passed but otherwise always allow 22/tcp. So to do that I had to define a maximum number of rules that could be inserted as one memory blob into the LPM map that the ebpf program would then linearly search to make its decision.
Thirdly has been making everything highly available which has been a bit of an on-going battle with ETCd mainly around how it manages cluster certificates as they dont (as of 2024 but it may be coming soon) expose the right structures to allow for dyanmic certificate creation, so you have to kind of make a wrapper around that in order to get everything going.
Im sure there are other things that I've had struggles with, but these are what come to mind immediately!
On the patch, maybe try reposting it on the list, with the pointer to your project to see if that provokes a new review?
Best of luck with the project!
Like the time I had to optimise map insertion because the linux kernel does some truly insane locking when you use specific types of eBPF maps:
https://github.com/NHAS/wag/issues/84
This is slated to be improved (or has already been improved in kernel 6.8?). But for now wag sort of just side steps it in a horribly stateful way.
I still play video games.
Making a Tailscale replacement doesnt sound fun for me, but for some I guess it is? Why shit on that? Who cares?
> Obviously instead of using an off the shelf solution like tailscale, I decided to reinvent the wheel which has honestly been quite fun with learning about eBPF, and recently clustering and HA with etcd!
Just because you don’t see or don’t care to see the point doesn’t mean that it doesn’t exist. And just because it’s not fun to you does not mean it’s not fun to anyone.
As everyone else has said, people can choose to have fun with their free time and what they do with it is up to them. “This isn’t a viable competitor” is an extremely shallow dismissal of one’s work when the motivating factor isn’t even remotely making a competitor. Which some people couldn’t give two shits about.
If it's not fun or interesting for you, just read something other than Show HN's. This sort of meta is totally offtopic in an actual Show HN which is about the work being shown rather than your personal feelings about the category.
Most technical people learn by doing. There is value in doing something even if you throw it away, even with hardware in meatspace, because it gains you knowledge and intuition about how the better/established thing works. This knowledge can be golden.
I have a very deep understanding of a lot of systems that most people never touch because they “just work” and are well implemented. I have that understanding exactly because of garbage toy projects (most of which I never shared). This knowledge has made me invaluable in many projects/jobs in my life, because nobody else bothers to understand those systems.
In fact, one of the most obvious (although perhaps not most efficient) ways to do clean room reverse engineering is to treat a system as a modeled black box and try to naively reimplement it using your own assumptions and then benchmark your outputs against the black box model. This reveals significant knowledge about the workings of the black box.
1: https://github.com/DefGuard/defguard discussed at https://news.ycombinator.com/item?id=36056080
This is why I asked, the phrase "I decided to reinvent the wheel which has honestly been quite fun with learning about eBPF, and recently clustering and HA with etcd" makes it sound like it's doing a bunch of cool stuff (which I want to hear about!), but the readme says nothing about those.
Primarily because I want people to have a reasonably good time setting it up, rather than having to go through my explanation on things!
I'd be super interested to know how they track "session state" as their do seem to rely very heavily on adding proxies and other additional software layers in front of the wireguard connection itself (https://defguard.gitbook.io/defguard/admin-and-features/wire...)
With wag specifically it's all just wireguard and a tiny bit of ebpf to do the management, along with tracking the external IP to determine if its time to re-challenge a user.
Most people do not use TailScale. I'd encourage doing the work of understanding why, there is likely a big opportunity somewhere here.
When you use Tailscale extensively, it becomes your new network. Now all your systems depend on a piece of software that you do not fully control. The control plane is not open source, and it is a key component of Tailscale. Headscale is a great effort, but it doesn't have feature parity with Tailscale (1). Moreover, the dedicated team at Tailscale keeps releasing amazing new functionality regularly.
That being said, if I had to buy software from a company, Tailscale would be my first pick. I respect and trust the founders and the early engineers working there.
As a side note, I'm planning to contribute to Headscale. This technology is crucial, and I want to help ensure its success.
(1) The functionality offered by Headscale is sufficient to build a robust mesh network and enjoy its benefits. Kudos to the team and to Tailscale for supporting it.
An attacker with access to coordination or relay servers would be able to change whatever is in the admin console, which are basically ACLs.
Am I missing anything?
I would expect software that implements the wireguard transport layer to implement session management to be called a working VPN server solution. This means a second channel to the server for periodically rotating session keys, terminating sessions, changing IP addresses, configuring new routes and repeating authentication if necessary.
Both solutions are custom for that VPN, so I’d just have one WG tunnel that’s controlled through a secured (web)service. The bonus is that you wouldn’t need a hard coded initial key. (You’d still need some authentication mechanism, but it could be more than fixed public/private key pairs).
While the wireguard key effectively gives you the ability to talk to the wag server, the session is effectively maintained by a map in ebpf, as to whether you've authorised or not.
So even if someone steals your private key material, they wont be able to access MFA restricted routes
So IPsec's IKE protocol? Why not just... use IPsec?
> curl http://public.server.address:8080/register_device?key=e83253...
> The service will return a fully templated response
It looks like the "registration" involves the server generating a private key then sending to the client, rather than the client generating a private key and sending the public key to the server.
Not only that but your example is http! Probably worth replacing that part at least in case people think http is a reasonable option.
> To authenticate the user should browse to the servers vpn address, in the example, case 192.168.1.1:8080, where they will be prompted for their 2fa code.
So when the session times out, is there any way for the client to realise this? Or does their ssh session (or whatever) just stop working?
I've on and off looked for a wireguard client which can do something like the captive portal detection on wifi. Ideally it would be an extra line in the config file (like persistentkeepalive), which does a URL pull. Could be checked periodically (like with the keepalive). If it returns "OK", then fine, if it doesn't return then there's a network problem, but if it returns a "Location" header, the client would pop up a browser at that location, allowing for session reauthentication or whatever.
I haven't found one.
That would be really cool. I hope the author of this will consider it.
My only problem is that if you capture that route and then redirect it, or whatnot, your peers wont be able to log in to wifi in public areas like coffee-shops/libraries/etc because the route will be trying to go via a VPN which wont be returning any real data.
Such as where you should be going to log in haha
So you'd put a fwmark on the interface , allowing you to route the VPN traffic separate from the traffic that should go over it. Then you have some mechanism to trigger when VPN is down. Curl someplace dedicated which also has a routing exception to go outside the tunnel. Detect captive portal IP. Add route (and clean up, like when online or switching).
Setting aside a VPN client, your browser and/or OS will send out a http get request to a site which returns "OK" or "success". If it returns a redirection, then it will go "ahha I'm behind a captive portal" and pop up a page for you to log in.
VPNs do something similar. The MozillaVPN client for example will periodically check for reauthentication by doing a http call over the non-VPN route to a server and looking for a response saying "success". If it needs a redirect it flags up a message on the client to disable your client and reroute traffic via normal networking to allow reauthenticaiton.
What I want from a wireguard client is it to check for captive portals outside (in a situation I'm routing the entire of 0.0/0 via wireguard), but also check for reachability inside the tunnel. Have "check" and "checktime" parameters, which poll a given server (presumably via the tunnel) preset a popup to reauthenticate.
I have written one in Go for the Mac which uses the command line wg from Brew and handles key gen, but it is clunky and requires sudo.
A proper native app which uses the network entitlements would be great, but it is beyond my ken
To answer your last question, eBPF XDP which is what I use can only do PASS, DROP or REDIRECT. So I stick with the easiest possible outcome and do PASS/DROP, which means your connections will just stop working.
However you can always set up the detection yourself by adding the captive portal detection pages to your wag MFA list then the browser should do everything else for you.
Unfortunately doing interception or acting like a proxy isnt something Im looking to do with wag (which makes authorisation timeout/logout a bit easier to deal with)
Hope that answers things!
That requires integration with a client, which is a massive pain to integrate
Then how do you do that for things like ssh, or other non-http based protocols.
But I'll try and give a basic run down on the differences/similarities.
Wag is good for hub and spoke design where you want to have a hard boundary, rather than a tailscale-esque mesh where everything touches everything and then the rules define the overlay.
Both wag and tailscale add SSO integrations and effectively 2FA for securing your users.
And both of us have a way to enroll and a web UI to manage things, although I'm sure TailScale is much more polished considering I'm one guy who doesnt like web development.
As for things Im definitely not going to implement, probably interception or a TLS proxy to redirect users once their session logs out. Primarily just because doing that in eBPF is a little bit beyond me right at this second, and I dont feel like writing the DNAT/SNAT components I'd probably have to in order to get it working