Show HN: Rivet – Open-source game server management with Nomad and Rust (github.com)
Rivet is an OSS game server management tool that enables game developers to easily deploy their dedicated servers without any infra experience.
We recently open-sourced Rivet after working on it for the past couple of years. I wanted to share some of my favorite things about our experience building this with the HN community.
My cofounder and I have been building multiplayer games together since middle school for fun (and not much profit [1]). In HS, I stumbled into building the entire infrastructure powering [Krunker.io](http://Krunker.io) (acq by FRVR) & other popular multiplayer web games. After wasting months rebuilding dedicated server infrastructure + DDoS/bot mitigation over and over, we started building Rivet as a side project.
Some interesting tidbits:
- ~99% Rust and a smidgeon of Lua.
- Bolt [2] – Cluster dev & management toolchain for super configurable self-hosted Rivet clusters. It’s way over-engineered.
- The entire repo is usable as a library. Our EE repo uses OSS as a submodule.
- Traefik used as an edge proxy for low-latency UDP, TCP+TLS, & WSS traffic.
- Apache Traffic Server is under-appreciated as a large file cache. Used as an edge Docker pull-through cache to improve cold starts & as a CDN cache to lower our S3 bill.
- ClickHouse used for analytics & game server logs. It’s so simple, I have nothing more to say.
- Serving Docker images with Apache TS is simpler & cheaper than running a Docker pull-through cache.
- Nebula has been rock solid & easy to operate as our overlay network.
- We use Redis Lua scripts for complex, atomic, in-memory operations.
- Obviously, we love Nix.
- We keep a rough SBOM [3].
- Licensed under Apache 2.0 (OSI-approved). We seriously want people to run & tinker with Rivet themselves. We get a lot of questions about this: [4] [5]
Some HN-flavored FAQ:
> Why not build on top of Agones or Kubernetes?
Nomad is simpler & more flexible than Agones/Kubernetes out of the box, which let us get up and running faster. For example, Nomad natively supports multiple task drivers, edge workloads, and runs as a standalone binary.
> [Fly.io](http://Fly.io) migrated off of Nomad, how will you scale?
Nomad can support 2M containers [6]. Some quick math: avg 8 players per lobby * 2M lobbies * 8 regional clusters = ~128M CCU. That’s well above PUBG’s 3.2m CCU peak.
Roblox’s game servers also run on top of Nomad [7]. We’re in good company.
> Are you affected by the recent Nomad BSL relicensing [8]?
Maybe, see [9].
> How do you compare to $X?
Our core goal is to get developers up and running as fast as possible. We provide extra services like our matchmaker [10], CDN [11], and KV [12] to make shipping a fully-fledged multiplayer game require only a couple of lines of code.
No other project provides a comparably accessible, OSS, and comprehensive game server manager.
> Do you handle networking logic?
No. We work with existing tools like FishNet, Mirror, NGO, Unreal & Godot replication, and anything else you can run in Docker.
> Is anyone actually using this?
Yes, we’ve been running in closed beta since Jan ‘22 and currently support millions of MAU across many titles.
[1]: https://github.com/rivet-gg/microgravity.io
[2]: https://github.com/rivet-gg/rivet/tree/main/docs/libraries/b...
[3]: https://github.com/rivet-gg/rivet/blob/main/docs/infrastruct...
[4]: Turns out I thought I knew more than I really did thus I’ve been spending much of my hobbyist time learning more AWS. I’m pretty stubborn about a local dev experience which takes time to establish an appropriate mental model to do. So it goes. Pro tip: find your application “interfaces” and discover what tooling can be used instead of a cloud resource (e.g. a mongoDB compatible document database is the interface to which AWS DocumentDB and MongoDB are concrete implementations). I have a docker compose file I use for local dev. It’s the best setup I’ve found for myself but I am open for advice. Anyways I’d just like to voice myself as a possible target audience. Being able to develop locally is important to me and the hardest parts with AWS is having little support on how exactly to do this. Assuming this is even a fair ask to make… P.S. another pro tip; I use Unity + Mirror networking for my multiplayer games. GameLift requires code integrated into your game code for it to function. I use nested prefabs to handle two cases: (1) when I want gamelift code to execute and (2) when I just want to focus on writing the multiplayer code. My production code uses (1) and I usually do development in (2). (2) is a child of (1). P.P.S. GameLift has a lag of about 2-3 minutes when registering my running game server in Unity editor with a “gamelift anywhere fleet” before it can receive traffic. I didn’t know about this for a while and stressed out over the inconsistency. Make sure to measure timings on this so you don’t go crazy. > Anyways I’d just like to voice myself as a possible target audience Local development is tricky, and we have plenty of room for improvement here. Currently we provide a special developer token [1] which will cause API endpoints to return mock responses when used locally. This way, you can use the exact same API with no difference in production & locally, so you don't have to mess with nested prefabs. However, this doesn't help much with exposing your local machine for testing. We've toyed around with the idea of something similar ngrok built in to our Rivet CLI and integrated in to the API. [1]: https://rivet.gg/docs/general/concepts/dev-tokens#mock-respo... I do enjoy learning and this constraint requires I deeply learn. However there is an opportunity cost. I'm particularly encouraged here by the tools and approaches chosen — Redis for matchmakers, ClickHouse for analytics/logs, Rust for speedy backend operations were tools I eventually came upon as best for the job after a long and painful process of iteration with inappropriate cloud dev tooling (with basically any datastore other than Redis you will run in to issues matchmaking). High traffic/concurrency in matchmaking is a hard problem when you need fast, consistent matches. Existing industrial-sized matchmakers (Google's OSS OpenMatch, AWS GameLift, etc.) require locking in to a very specific format or spinning up an entire K8S cluster, which for game devops teams already stretched thing is a tall ask). And for all that effort, you may discover the performance characteristics of their approaches aren't what you need, or (as has happened to us with different game web service/API/middleware providers) they may choose to abruptly deprecate the service upon which you built. Appreciate the dedication to making this available and the focus on helping gamedevs get up and running as fast as possible. Looking forward to playing around with Rivet and seeing where it might fit in our stack — & would love to talk shop on this with anyone since I've been tortured by / interested in these problems while trying to simultaneously run a game studio / make new games / keep everything online the last few years! I’m glad to see the open source development of these tools. Personally, there’s a lot of stuff colleagues and I have built that I would love to see open sourced. Otherwise we just end up implementing the same services over and over again. Another example is patch delivery. It’s a solved problem yet everyone keeps rolling their own, or only shipping to Steam. Of course, all managed with some Jenkins scripts written by someone that’s no longer at the company. I want Fastlane for games that makes it easy to target the various distribution channels across desktop/mobile/console. I don't recommend it. Game CI has been in development for a long time. It only really supports GitHub Actions. It can't correctly build il2cpp Unity projects. The licenses needed to run it are more expensive than Unity Build Automation / Unity Cloud Build. If you want to automate Unity builds and you don't want to learn the Windows Containers ecosystem, Jenkins and/or Tekton, you should use Unity's service. CI/CD is a bad choice for most developers, on most platforms, for most clients. That said, it makes sense to do for your backends. > or (as has happened to us with different game web service/API/middleware providers) they may choose to abruptly deprecate the service upon which you built. Which service was that? GameSparks? That sucks. > with inappropriate cloud dev tooling Well everyone takes their own journey to discover how shitty Lambda, Cognito, CloudFormation and related are. > with basically any datastore other than Redis you will run in to issues matchmaking Matchmakers do not have to be complicated. Are you making this statement about game devs, or more broadly? I don’t know much about game dev, but if more broadly that’s a pretty spicy take. One question, am I correct in understanding you bake in the assumption that 1 lobby = 1 container? I can see how that’d be desirable for the nice properties it gives you architecturally but if your game isn’t very demanding (mine’s turn based) might that end up stranding a lot of resources? lists nomad as apache 2.0, when its either MPL or BUSL depending on version. also fwiw, your ci is failing, because your secrets detection ci hook is barfing on recently updated click house sha 256 in your nix file. - no updates after December - you probably won't have the right to use nomad after that - every providers offer Kubernetes managed solution, by using nomad now you're pushing that burden to host and manage infra since no one is offering nomad services Using Kubernetes is a huge advantage since all the dev ops tool can run on it, it's tested and everyone is using it. As for more flexible than Agones I highly doubt that, do you have example? Are you kidding me? Before you can do anything, you need... 1. API Server 2. Scheduler 3. Kubernetes Controller Manager 4. etcd 5. Cloud Controller Manager 6. Nodes 7. Pods 8. Container Runtime Engine 9. kubelet 10. kube-proxy And then before you can even expose it on HTTP you need an ingress, kube-dns, load balancer... Oh, don't forget that you'll actually want Helm and all of its components... Meanwhile, Nomad is a single binary, and it can even run scripts and other non-containerized workloads. An ex-colleague “sold” us developers nomad once as an easier, better alternative to k8s. They used exact same arguments, and we ended up with Nomad for our test and staging environments. It was such a mess, while it covered basic “run this” scenarios fine, nothing like kustomize or helm for nomad existed back then (I doubt it exists now). The tooling was non-existent, neither were good practices or recommended setups - nobody seemed to know how to tie it into a ci/cd setup without a bunch of bash scripts to get basic stuff like templating working. Implementing any kind of edge case or a tricky requirement felt like we are doing breakthrough work since no documentation, guide or SO page covered anything but “simple setup of your nomad cluster in two steps” - nomad is simply not popular enough. That experience made me swear off any unpopular but “trust-me-bro-its-actually-good” software. Sorry for the rant, your post made me remember how pissed I was at our tech stack back then. You go to any cloud provider they all offer managed Kubernetes solution. But it's complicated. Game development uses all sorts of red flags throughout all of history. The people making successful games are talented enough to deal with the downsides of anything. Nomad specifically is used by e.g. Roblox to orchestrate. Do I think that's a good idea? No. But it doesn't mean it can't work. With regards to your specific criticism I don't think that stuff really matters, there are tons of people using tons of totally unsupported / unsupportable game engines, open source libraries, etc. If people blindly accepted X has won we'd be still using Perl, XML, CGI-Bin, Apache, Angular, etc. As another comment has said, I believe years from now, folks will be looking back at in disbelief at how anyone could have considered k8s to be the best solution without room for improvement or alternative approaches. I'm somewhat inclined to agree, because iterating on what we have and developing new tech and practices seems like a good idea (the concept of 12 Factor Apps comes to mind). However, I can't help but to wonder what this would look like, when taken to the extreme. Perhaps something along the lines of: "We are only going to use Ubuntu for the servers and Java for all of our back end services, as well as React for the front end. Nothing else, no matter how many different projects there are." And on a certain level, it makes sense: consistency and skills that carry over seem like a nice thing to have, there's even terms out there like a company being "a Java shop" and so on. That said, such inflexibility might not always be nice (e.g. trying to do machine learning in Java) but at the same time if a specific technology is in demand in the job market in most companies, you don't really get much of a choice. To return from the analogy, in the particular case of Kubernetes, you'll have to learn and use it and there's only so much free time for exploring other options. > If people blindly accepted X has won we'd be still using Perl, XML, CGI-Bin, Apache, Angular, etc For many folks, some of those might indeed be good choices. Some backup software that I use is written in Perl and is actually pretty good (BackupPC). XML can be a nice format to use and has lots of tooling available. Even SOAP wasn't necessarily the end of the world thanks to WSDL - being able to plug a file into SoapUI and getting an API that you can play around with, something that took Swagger/OpenAPI years to reproduce for REST. There's something nice to be said about the simplicity of CGI and FastCGI still makes sense with something like PHP (PHP-FPM). I actually still use Apache for hosting my personal stuff - if has most of the features you might want and with mod_md it's a bit closer to Caddy than Nginx in getting LetsEncrypt certs (ACME). If you disable .htaccess, reasoning about the config also becomes about as easy as with Nginx (nginx.conf). I've also seen Angular be used reasonably well and the "batteries included" feel of it was nice, especially when React had about 1500+ dependencies for a new project. I do personally lean towards Vue, though (oh and migrating away from AngularJS after the EOL was a pain, when that was relevant). My point is that if something is good enough for your needs and helps you put food on the table, then you can just go with that choice. Let those who prefer to experiment do so (hopefully not with your livelihood), celebrate their efforts and successes and learn from their failures, though. Kubernetes is established, Nomad is interesting, even Docker Swarm or Compose is okay. What are the features only available in SaSS? Do you anticipate to have new features only for SaSS users? Is there a policy for releasing SaSS features to open source users? I would love to know more!50 comments
[ 2.7 ms ] story [ 114 ms ] thread