Launch HN: Gravitl (YC W22) – VPN Platform Based on WireGuard
Networking across environments is hard and slow. WireGuard can solve this, but it’s tough to run at scale. WireGuard is a fast and efficient VPN protocol that is growing quickly in popularity. Linus Torvalds called it a “work of art,” and it was added to the Linux kernel in 2020. It now runs on most major operating systems.
We created Netmaker to automate WireGuard-based networks at scale. It opens up a bunch of use cases that are otherwise infeasible. With Netmaker, our users are managing edge networks, connecting fleets of unmanned aerial drones, and cloud-bursting k8s clusters for machine learning.
Alex got the idea for Netmaker while he was in New Mexico, staying in the desert to escape the pandemic. We were trying to run a distributed Kubernetes cluster. Our goal was to create a cloud provider with no infrastructure, using compute provided by users. To start, we bought a couple raspberry pis and some cloud VM's, hooked them all together, and ran a k3s cluster across them using WireGuard.
We realized we needed a mesh VPN to do this at scale. None of the existing options gave us everything we needed, so we built Netmaker. We put it on GitHub, and it became so popular that we decided to work on Netmaker exclusively.
Netmaker works on a client-server model (https://docs.netmaker.org/architecture.html). A central config server tells each machine where its peers are and how to reach them. The local client automates network settings and DNS on each machine. The result is a flexible virtual network that stays in sync whenever machines are added, removed, or there is a change in state.
Without Netmaker this is challenging, because WireGuard requires reconfiguration whenever any peer in the network changes. In addition, the network can be blocked by factors like NAT, firewalls, and port availability. Netmaker anticipates and solves for these factors, while being compatible across Mac, Linux, Windows, and FreeBSD.
There are other solutions out there with similarities, but we’ve got some key distinctions. After all, we created Netmaker out of necessity, because the other solutions didn’t meet our requirements. First off, Netmaker is super fast because it can use kernel WireGuard. There are some other WireGuard-based solutions like Tailscale, but they use userspace WireGuard, which is much, much slower.
Second, Netmaker is tailored towards the cloud and Kubernetes. Stuff like OpenVPN was built before the cloud became a go-to deployment strategy.
Finally, Netmaker is fully self-hostable. A lot of existing options are SaaS, but our users want control of any servers that are routing their traffic or managing their virtual networks.
As for what’s next, with Dillon at the lead, we’re putting in a lot of work to overhaul the code base, implement community-driven features, and pull Netmaker towards a “pure WireGuard” vision. We're planning an enterprise release in the coming months which will have a few features that businesses need at scale, without taking away from the free community version. In the meantime, we have a simple support subscription for the existing community edition: https://gravitl.com/plans.
We’re always looking for ways to do things better. If you have thoughts, we’d love to hear them, and if you’re doing anything cool with WireGuard that could be relevant to our project, we’d love to hear that too. We’ve also got a community on Discord you’re welcome to join at any time: I am a user of this solution for my company, and also for my homelab. It works great! Just wanted to say that, and thank you for the open source free as in libre license. Edit - I see you mention that Tailscale uses userland WireGuard. Is that the biggest difference between the two? Do you foresee yourselves running into issues by not using the userland implementation? With Nebula, they're a lot closer on speed, but also we've got a management GUI which makes things a lot easier. We were very close to using Nebula in our early days. The main thing that stopped us was, we decided WireGuard was going to be the standard in the future, and wanted to be based on WireGuard. That leads to a bit more fundamental of a difference which is a bit harder to quantify. Our aim is to really be a "WireGuard controller." You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. We're getting close to that vision but aren't quite there yet. That last point leads back to the kernel thing. We use kernel by default, but really, Netmaker can use any WireGuard implementation. If users are scared of the security implications of using kernel, they can use the userland version, and Netmaker should be able to pick that up just fine. They can even run it in a docker container on their machine. How do you handle NAT compared to these other options? I've run into issues with WireGuard when both sides are behind residential NATs (or AWS EC2 IGW); as you know, Nebula solves this with "lighthouse" servers (which are self-hosted but externally accessible), and Tailscale uses its third-party intro devices. Do the Gravitl servers have to always listen on an exposed port? On the NAT side, we provide 3 layers of traversal options: #1 (default) port forwarding: This actually works in a surprisingly well, for about 90% of environments, but does require an exposed port. #2 UDP Hole Punching: The server acts similarly to a Nebula "lighthouse" and will tell clients where to reach each other. This covers that small situation of dualing NAT's, and doesn't require the exposed port. #3 Relay: In situations where neither works such as CGNAT, you can set a public node as a relay to route traffic to the "hidden" node Don't take this the wrong way, but subtly/implictly implying that some software is not going to be reliable in the long run because it only has a community behind is low-key FUD. This is excellent. We briefly considered building something atop Wireguard in the early days of Nebula, but decided not to do so because of scaling. Wireguard's protocol necessitates that all nodes have existing keypairs for each other ahead of time. At Slack's scale, that means every time a fresh node is launched, you would have to tell 50,000 other nodes it exists. Obviously you can smarten this up and tell only hosts it might talk to. But this adds complexity. Using PKI eliminates this key distribution problem and means that you don't have the same scaling limitations as something built on WG. Wireguard is a very very good VPN, but I cannot imagine trying to run something on the scale of tens of thousands of nodes when you need such complex coordination systems to exchange keys/trust, especially in a dynamic environment where nodes are coming and going all the time. I totally get that it solvable overall, but Slack has had 4 years of nearly perfect uptime on Nebula, whilst using it to pass >95% of all backend traffic. These considerations may seem simple to address, but there are fundamentals that mattered and led us to writing Nebula. We didn't want to create something new, but to do what Slack needed, we had to. Most companies choose between not securely connecting their stuff correctly or having a complete mess of an internal network. So congratz and all the power to you. A couple questions from me, if you don't mind. I currently use Tailscale for my home lab. My current model is similar to old-school road warrior VPNing, where I have my router as a node that routes to the LAN using the internal (non-meshed) IPs. Does your solution support that? How easy is it to deploy and configure it in that way? Another feature I use is "exit nodes", i.e. my router as a gateway to the Internet, similar to what consumer VPNs (Mullvad, PrivateInternetAccess, etc) do. This offers me 2 things: more security when I'm on insecure networks (airport or other public WiFi, etc); and allows me to access things as though I'm at home when I'm traveling abroad. This is a one-time configuration and I can easily switch it on and off on demand via the desktop or phone app. Is this an option in your product? Edit: s/customer/consumer/ I believe we actually had a feature like exit nodes before Tailscale released theirs :) Our is called an "egress gateway" and it acts in pretty much the same way. Thanks for your reply, I will try to deploy Netmaker alongside Tailscale and compare for my needs! The "hardest" part is managing the tailscale installation on the router, since the OpenWRT packages seem to be quite old. I went old-fashioned and just download the ARM tarballs and uncompress them. I lifted the init.d script from someone else's blog post [0] and adjusted the path to the binaries. After that I've pretty much just followed the official docs to implement LAN routing [1] and the exit node [2], then set the router's cert so it doesn't expire, which is done on the Tailscale GUI. Install the app on the devices, connect everything to the tailnet and Bob's your uncle. [0] https://willangley.org/how-i-set-up-tailscale-on-my-wifi-rou...
[1] https://tailscale.com/kb/1019/subnets/
[2] https://tailscale.com/kb/1103/exit-nodes/ So single point of failure? Quick question, does Gravitl plan to become a company donor to Wireguard in the near future[1]? I think it would be great if companies which, as is their right, profit from FLOSS, also give back to FLOSS, not just with code. Again, good luck with your adventure :) . - Named groups/subnets for a single NetMakerServer (lets say [Users] and [Servers]) - Access control capability: Allow/disallow inter/intra group connectivity. For example, using the previous groups : [Users] can connect to [Servers] only (not other [Users]) and [Servers] can connect to other [servers] only (not [Users]). Basically, for this case, the peer list returned to NetClient's only contain the [Servers]. More generally, it is the concept of public vs private - not sure if this is needed on a group/subnet or individual netclient level. - MFA requirement for certain Groups. Example: [Users] who have not connected to the network in the last X hours/other criteria will need to MFA (provided by NetMaker+TOTP or other IDP like Okta). Server-server connections will never need to MFA. This is one of those Enterprise security check list items in security reviews. - Ideally: Ability to run the NetMakerServer on Windows (IIS, Nginx, Apache) without having to launch containers/install VM's. Congratulation ons on your launch. Will keep an eye on things. Edit: This is a requirement for self-hosted but if your enterprise solutions offers it as a highly available service that would by-pass this issue. It would need to be HA and spread in such a way that a single cloud provider going down does not affect things. Can any client be an egress node? When I set it up, I have chosen ZeroTier instead of WireGuard. It does the following: (a) Hosts discovery and initiating handshake between clients with the help of a server from ZeroTier, (b) NAT hole punching, (c) pushing centrally managed routes to hosts, (d) network ACL rules. I primarily have chosen it because it is easy to setup by anyone (e) and I do not have to manage a server (f). - Can you tell a little bit how Graviti compares to it. I guess WireGuard itself does not have the features (a) to (f). I guess the Netmake server replaces the ZeroTier servers and provides some of these features. - Are you inclined to install Netmaker client on any host or use one node in a LAN as a router? - Is this more geared to servers/professional managed hosts or also for laptops? For my usecase with ZeroTier I found the following currenlty missing features useful: - Easy setup of a node as a router (or virtual switch) to connect a local network to the virtual one without installing it on all devices (hardware like GPS receiver do not allow to install new software). Of course, you can do it with the normal Linux tools. - Installing it only inside a Docker container and not on the host. But I guess that will not be possible because it has to live in the kernel. Router is obviously preferable when routing to LAN but is harder to support. If it's FreeBSD or OpenWRT, go router, but otherwise a client on a Linux node works fine as a router. This is definitely geared more towards servers/VM's etc, but does work on Laptops as well. We have Windows support and you can even loop in your phones. We do actually have a docker image for the client. We're not strictly tied to the kernel version of WireGuard, and you can use userspace wherever it is a necessity. I'm not expecting that I'd actually get into trouble for running a small network with my friends, but I prefer not violating software licenses. :) This is probably fine for internal usage within a company (as the whole company is legally the same entity) but for individual use it's concerning. Although it might also be problematic if you ever want to give a contractor access to your VPN too. Covers the contractor use case I think (provided the contractor isn't running services they intend to expose to you on the VPN), but not my hobbyist use case (both me and a friend share services with each other via our VPN). Can I just once again state my love for it and hope it gets merged soon? Maybe the code isn’t perfect, but I’ve skimmed it, and compared to the horrors that are OpenVPN and IPSec, it’s a work of art. I was just about to ask about differences with Tailscale, which is solving a lot of the challenges outlined in the post, but you answered it: "First off, Netmaker is super fast because it can use kernel WireGuard. There are some other WireGuard-based solutions like Tailscale, but they use userspace WireGuard, which is much, much slower." Good luck! Tools like this allow users to own their network and avoid that lock in. All that's needed here is auto-scaling integrated w/ various cloud providers, and some canned templates for creating K8s clusters and common services. At that point- we've got a self hosted cloud w/ 80% of what the big guys offer. Shown on the marketing pages, the controller supports pushing configurations which include commands for `postup` and `postdown`; does this mean that it implicitly has root code execution on every participant in the network? I don't think it would be designed like that, but I'm a bit confused how this could be working without introducing that as a concern. However, we're adding a switch for this pretty soon which will allow disabling the edit-ability of those fields.115 comments
[ 3.4 ms ] story [ 184 ms ] thread