> We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server tells everyone what they should be seeing and everyone interpolates it in order to make it look smooth. When this happened, lockstep networking kind of stopped evolving. It’s my opinion that rollback is the next step for lockstep networking and, just like every other discipline in video game development, it’s time for lockstep implementations to evolve. The time investment is just super worth it to me.
Rollback has been in the industry for years now. The whole article reads like there's basically large swathes of the fighting game subset of the gaming industry who don't know anything about multiplayer game networking.
Which is understandable considering the game industry at-large doesn't know a thing about multiplayer game networking. Go checkout an out-of-the-box build of Unreal Engine 5. Yeah, they've got Nanite wow!, but do they have Actors who don't rubberband? No, and it's stupid and frustrating. It makes the whole engine feel like a joke. Especially compared to Source 1 which had client-side prediction and rollback support over dial-up. Just plain dumb.
If you think that's bad, Unity does basically nothing for you, and Godot pretends to have multiplayer support by saying it has ENet bindings. Wow, thanks!
It's disappointing that firms can chase physically based rendering, and invest so many man-hours into asset development but can't recreate what's existed for literal decades.
Also, AFAIK QuakeWorld was THE client-side prediction implementation to match. Maybe it was developed prior to John Carmack, but I don't know of any major software that did so with a timestep. Everything else followed. This guy doesn't know what he's talking about.
Counter-Strike: Source had rollback in 2004 and Yahn Bernier wrote about it in "Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization" in 2001.[1] Keep up dude.
To be slightly fair to the fighting game community, this article is from 2019. My understanding is that most top-line fighting games now are rollback enabled (the pandemic driven shift to online tournies put significant pressure on developers).
My history is a little hazy but wasn't rollback as a solution driven by the community. Maybe it was SF3 online emulation or something? Or was it Smash Melee
Sounds about right. Never was part of the Melee community but I was always impressed at their ingenuity to figure out multiplayer in the face of Nintendo's active community hostility and bonkers multiplayer UX
No, the first major game to use it and push it was Skullgirls.
An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves
"Series of unfortunate events" is an understatement, but I would say it gets due recognition--it's an indie game over a decade old and still earns its slot at major fighting game tournaments and (after said events' hopeful end) still has support from developers and community alike.
Looks like this was published shortly after GGPO was made open source in October 2019, along with a big wave of similar rollback hype. It has indeed become more commonplace in the last 3 years.
The big difference is in the net code. While I can remember and justify
all of my decisions about networking from DOOM through Quake, the bottom
line is that I was working with the wrong basic assumptions for doing a
good internet game. My original design was targeted at <200ms connection
latencies. People that have a digital connection to the internet through
a good provider get a pretty good game experience. Unfortunately, 99% of
the world gets on with a slip or ppp connection over a modem, often through
a crappy overcrowded ISP. This gives 300+ ms latencies, minimum. Client.
User's modem. ISP's modem. Server. ISP's modem. User's modem. Client.God, that sucks.
Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar
with PPP life. I'm adressing it now.
The first move was to scrap the current net code. It was based on a
reliable stream as its original primitive (way back in qtest), then was
retrofited to have an unreliable sideband to make internet play feasable.It was a big mess, so I took it out and shot it. The new code has the
unreliable packet as its basic primitive, and all the complexities that
entails is now visible to the main code instead of hidden under the net api.This is A Good Thing. Goodbye phantom unconnected players, messages not getting through, etc
Yep, and because of his .plan files, and no other historical details around the technology, I think it's safe to say he invented it. In 1999, QuakeWorld was the first, to my knowledge, implementation of client-side prediction in a first-person shooter.
But I'm almost certain there were rudimentary client-side prediction models that existed before John Carmack, which is why I say I think he was the first to "invent" it, but frankly I'm not so sure about that.
I mean think about terminals, as an example.
I do think with near certainty he was the first to invent the client-server with client-side prediction architecture in time stepped software, or more specifically in an FPS.
Is there any good template/library for modern multiplayer netcode for Unreal or Unity or anything else, or is it truly everybody rolls their own as the secret sauce in their AAA games?
No, people literally still roll their own multiplayer networking in 2023. Seriously. Unreal Engine 5's out-of-the-box multiplayer code isn't production ready, and they're probably the "best" of the major engines people use today.
It pales in comparison to older engines from other game companies. It's effectively unusable.
This seems wild in the free-to-play era, where multiplayer with in game purchases is, like, the main business model. Maybe Unreal figures the multiplayer implementation is the metric on which the game studios compete, so there’s no hope to commoditizing it, or something like that?
It's also why so many multiplayer games are bad. Wanna know what's worse? Unreal Engine's architecture is unreliable by design.[1]
Basically, Tim Sweeney's approach for Actor movement wasn't to replicate state accurately, but just trust that clients would eventually get the right data. This isn't the same as eventual consistency mind you, because game state constantly changes in fast-paced games.
You can have whole time spans where Actors have absolutely no accurate player position replication.
1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider...
I've tried on many occasions to get excited about these engines and started a project with them. Unfortunately, when the entire realm of multiplayer feels half-assed my hopes are dashed catastrophically.
Right now, I am spending every moment of my free time developing my own game/engine from scratch. I honestly don't give a shit about Nanite, AAA graphics, et. al. Complex multiplayer experiences are far more compelling to me than raytraced graphics and sub-pixel triangles.
I think the exact nature of the net code is much more intimately linked to the game concept than anything in the rendering pipeline (for a multiplayer experience like Quake or Counterstrike). You will never get a perfect fit if you outsource this.
Yes! I completely agree. You're not alone. There's a lot of us out here slogging through writing portions of game software we don't want to only because it's necessary to create the experiences we want that can't be delivered in a timely fashion with existing technology.
> 1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider...
Intriguing. Makes even more sense that Frost Giant Studios (who are using UE5 for their upcoming RTS) have written their own networking code (with rollback!), and thrown away other big parts of the engine.
I'd be very interested in reading more about FGS and what they've been up to around UE5.
The idea of incrementally defenestrating UE5 in favor of in-house components is compelling to me. This path definitely works a LOT better in a team setting where you have actual artists looking for things to work on.
I don't know how the licensing actually works out around this though... Assume you went full ship of Theseus on UE5 and replaced 100% of it with in-house. Will Epic Games be happy walking away from this, or is there some fundamental legal component that we know we'd never screw with?
You can modify Unreal Engine as much as you like to ship a game and the engine changes remain your property, you just have to pay their royalties.
At least that is how it is in practice - the language in the license agreement is actually a little fuzzy, when our game was sold from one company to our current one, the lawyers had a bit of a frowning session over some of the language there, but ultimately decided it was okay. Maybe in the future it will be challenged, but for now, self-owned engine changes are the norm.
> I'd be very interested in reading more about FGS and what they've been up to around UE5.
They have two subreddits: r/frostgiant, and r/stormgate (for their game). They regularly ask the community for feedback/discussion/brainstorming on some topics relevant to the gameplay (like pacing, asymmetry, monetisation, e-sports, etc), openly share their conclusions from the discussion, but unfortunately in-depth technical stuff doesn't really pop up often.
They specifically bragged about their pathfinding (apparently at least on par with StarCraft II), and the rollback netcode (which is "directly inspired" by fighting games, and apparently not a common thing for RTSs), but again, everything is scarce on technical details.
The unofficial Discord community (https://discord.gg/stormgate) has a "Yeti Tracker" channel that automatically cross-posts any comments made by FGS staff; there's also https://stormgatenexus.com (fan website) where many of the interviews are transcribed and distilled into articles.
I'm watching all of these closely, I can ping you if I see more engine-tech-heavy stuff popping up!
It's not because they wanted to, I guarantee it. It's because it's impossible to not do it. Critically important details, down to the engine's frame timing, how replication is deeply integrated into the engine, and how the default Actor movement code works means you have to not gut just a single implementation detail, but you have to cut whole swaths of code out of use.
I'm dying to be proven wrong, but the intersection of people who know multiplayer networking well, Unreal Engine 4 or 5, and have successfully implemented client-side prediction with reconciliation and rollback in said software and out of it as well, are vanishingly small.
I failed to understand how one could use the existing engine's loop which is variable framerate, the existing replication system, which favors an implementation opaque RPC system over explicit payload structures, and rewrite Actor movement and engine timing subsystems without having a parallel network loop that uses none of the above.
My guess is that it is possible to use Unreal's variable framerate and replication system, but only rewrite Actor movement implementation details by creating a class that inherits from it and overrides the behavior entirely, but you're still stuck with the fact that because the Unreal Engine uses a variable framerate, you have an inconsistent rollback timeline across every client.
You want consistent frame times, in my experience, to have rollback. Does it mean that it's impossible otherwise? No, but you can get caught rolling back in-between game states that don't actually exist on the granularity of the timestep you're running at.
It is certainly doable, there is a plug-in on the UE marketplace that meaningfully extends Unreal's movement replication to do nicer interpolation and extrapolation of movement, it is called SmoothSync. We used it in lieu of UE's character movement component on the vehicles in the game we shipped a couple years back. It is a bit heavier on bandwidth (to their credit, Unreal's movement component replication is VERY lean) but still well within budget.
The code is horribly ugly, but works well. The ugliness I think is less to do with integrating with Unreal and more to do with the author though. That said, it works.
I'll have to check this out, thanks for mentioning it. I would be thoroughly surprised if it actually integrated with the rest of the engine though--my guess is they sacrificed this deep integration (foregoing the replication replay system, etc.) just to get something that works.
Pardon my ignorance (only interested in gamedev tangentially), but I would've assumed every game engine that progressed beyond a mere toy would always fix the timestep (or at least make it a single API call to do so). Or is there an entire class of games that somehow benefits from variable timesteps?...
The Unreal Engine has a long history, and its origins did not start from a fixed or semi-fixed timestep. The Unreal Engine doesn't benefit from a variable timestep, it's hampered by it, and the engineers have deemed it too difficult to fix, because the decision permeates the entire engine.
There is a "fixed timestep" mode, but it's for cinematics, and specifically for synchronizing the engine with external hardware, IIRC. But it's not a true fixed timestep, it's actually a fixed framerate, instead.
I know I'm going against the grain and this is a super-unpopular opinion, but I really feel like there's room for more specialized engines and more independent engine middleware/tooling in 2023. The "game engines are impossibly hard, so we won't get new ones" discussion only ever talks about Unreal and Unity, where editor, engine and technology are all vertically integrated and the flexibility requires a lot of compromises. Plus you can't really replicate those two, even if you have a great team and 10 years of work on your hands (Godot). This oligopoly is super unhealthy for the industry, IMO.
I don't think it's unpopular. We need more and better game software, what exists today is unsatisfactory for a plethora of reasons. I'm personally interested in this because I ship Planimeter Game Engine 2D, which comes with client-side prediction standard, and it's the largest pure-Lua game engine on GitHub.[1]
I think in most game development related forums, tutorials, etc, you will be correct about the unpopularity of building a new game engine. However, it’s definitely not a lost cause in general. I also agree that a vast majority of the ‘engines are impossible’ are predicated on Unreal/Unity as a baseline. I look at Arch Systemworks and see them using Unreal for their heavily stylized games (Guilty Gear, Drgaonball FighterZ, etc) and watch the various talks and articles about them having to bend over backward and work to fit Unreal’s expectations to make the engine render and animate in the manner they require and always wonder if a custom engine would not make workflow and maintenance simpler. I wonder the same for more specific genres, like RTS and HD2D-style JRPGs, where a custom engine would conceivably be a better long term option for developers. I always go back to Mike Acton talking at the first HandmadeCon, where he is talking about small dev teams and their benefits in specializing tools and wonder if that logic should be applied to engines as a whole.
Exactly! Good point about forums/tutorials/courses pushing unification.
And interesting, Mike Acton's point goes all the way back to the late Fred Brooks. Not the popular main topics of MMM, but the "Surgical Team" chapter, where he mentions that there should be someone taking care of the tooling. Now, in 1974 this could have been a person. In the 90s game studios it was common to have specialized teams building reusable tooling. This seems to be disappearing.
If I had to guess it becomes a question of how easily can you replace/train up new people to be proficient with the engine/maintain it. Fighting game engines are probably the simplest game engines there are to make but new people still need to be onboarded on it, the knowledge will be internal mostly, you can't find people with knowledge in $propietary_engine to hire right off the bat let alone maintain it. Capcom will use I believe their RE engine for sf6, already used it in dmc5 replacing MT Framework so it's not like it isn't happening in some places, but the AA developers where most of fighting games sit probably can't justify the costs to train and maintain a proprietary engine versus the cost of dealing with roadblocks in UE.
My original point, that was not well articulated at all, was more that there should be a middle ground. The choice shouldn't be only between "homegrown engine" and "Unreal/Unity oligopoly".
I'm saying there should be much more competition when it comes to third-party engines, but also in third-party tooling and third-party middleware.
And this competition should be more specialized in order to demand less of the "train and maintain a proprietary engine" but also less of the "dealing with roadblocks in UE".
I’ve wondered how much of this comes from the huge amount of game programmers that dive right in with zero CS background. I’m not knocking just jumping in, but a game engine is definitely not going to be feasible with 0 experience/background. Even setting up a sophisticated enough build environment to glue together libraries and handle assets is not going to be realistic for someone who’s only experience is installing mods and maybe an OOP course.
Oh, I'm not talking about regular game developers making engines from scratch, I'm talking about the need for competition to Unity and Unreal from other companies, either in the form of new engines or new tooling. I believe new, more specialized, engines would actually make things easier for newbies.
But there's a point there, lots of recent game developers are strongly against the idea of "new" things since each new engine would be one new software and platform to learn.
The adverse reactions I occasionally receive when I talk about building my own engine are a primary source of my motivation these days.
Let me give you some even-more-unpopular opinions. I think the GPU is a gigantic distraction from fun. Low-level languages are largely a mistake for new engine/game development in 2023+. You can deliver production-quality, interactive experiences on top of GC'd languages that perform better than ASM/C/C++ implementations (both for technical reasons & human reasons).
For me, the best "game engine" on earth right now is .NET7. Nothing gives me as much control over such a broad scope of fundamental primitives. You might be balking at this point, but go look at their System.Numerics implementation. First-party, SIMD-enabled code for projection matrix calculations based upon FPS-style semantics is just the beginning. No royalty bullshit either - I get to keep 100% of the profits I extract.
I should tell you as the publisher of a large game engine that performance absolutely does matter and it will eat you alive after it is too late and you've invested too many years into a particular language ecosystem.
One of my biggest dings against Godot is not having support for Photon Networking's newest products
Photon has a new group of network libraries that are essentially the evolution of "rubber band networking" that's been default for networking middleware for the last decade
One is especially interesting in that it uses IL weaving to provide a completely deterministic engine for simulating gameplay based on inputs only: https://www.photonengine.com/quantum#
Determinism from inputs was basically a part of QuakeWorld, and how the "VCR" system in id Tech 1 worked. You could replay sessions because the inputs were what were networked, not other state. They're selling 24-year-old tech. :)
Rollback hasn’t been in fighting games for very long at all. It’s a relatively recent addition to the genre, and has resulted in dramatically improved online play. It’s still new enough that it’s even touted as a feature & selling point in marketing. Within the fighting game community, online play/tournaments have typically been regarded as a joke & not representative of skill at all because delay-based networking woes. However nowadays, with rollback, online tournaments are held regularly, more accessible than ever, and regarded more highly than before. Rollback is greatly helping the genre stay alive.
Fighting games have had rollback netcode since at least SF2 HD Remix in 2008. In terms of online fighting games, it has been around for more than half of their existence.
The problem has been the lack of adoption/prioritization, and probably more than a little NotInventedHere-syndrome.
It's indeed sad to see. Game networking techniques have been known for multiple decades and yet still haven't made their into any of the major publicly available engines (notably Unreal and Unity). What gives?
As a game dev of 10+ years myself, I have a few theories:
1. Multiplayer networking is the "secret sauce" that creates moats/barriers to entry for incumbent studios. Think Rocket League or Fortnite or StarCraft 2. The tech exists, it's functionally well understood, but hard to implement at a high level of quality. Why not keep it to yourself? There is more money in holding onto your game revenue monopoly than trying to sell the network tech.
2. Game genres differ greatly in their networking needs. Rollback is great for fighting games but that's about it. StarCraft uses delay based deterministic lockstep. FPS games use (typically) ad-hoc server authoritative state syncs with client side prediction. Other complex games use full determinism with rollback, sending only inputs over the wire. Some cheapo indie games using client authoritative models (open to cheating but easy to implement, arguably fine for coop games). There are even more variants and blended approaches but you get the picture: there is no one size fits all approach and many are mutually exclusive so it's harder to package into an engine as a comprehensive solution.
3. Networking a game properly involves very leaky abstractions. It is impossible to write gameplay code for a networked game without understanding the nuance of the network model. This makes it substantially harder to develop the game, and this hurts the major game engines' marketability, with both major players Unity and Unreal guilty of selling themselves as "look ma no code required" solutions. Similar to point 1, not worth the money to sell this.
I don't see a great way out of this unfortunately. The only good networking middleware I know of is Photon and they're not exactly an easy to use product either. Hopefully we see better open source tooling in the future.
I agree. Sell the physically based rendering sizzle, keep the multiplayer networking sauce for yourself.
I don't buy the different genres differ greatly in their networking needs, though. Dota 2 uses the same networking architecture as Counter-Strike. Both of those games use the same underlying networking techniques as RuneScape 2 (Old School Runescape). Those are a MOBA, FPS, and an MMORPG respectively.
You use lag compensation (now people refer to it as rollback) in FPS games too. Yahn famously talked about it in his paper.
In fact, any type of granular timing-dependent gameplay requires lag compensation, otherwise you end up with situations like players leading their shots or missing entirely (Halo 1 PC).
In Halo 1 PC, they actually broadcast a hit sound from the server if you actually did make bullet contact. And those weren't trace-based (hitscan) either. Halo 1 bullets actually had travel time. So, Bungie basically didn't do lag compensation correctly, and hacked a solution on top that gave players feedback to help smooth over multiplayer. Because bullets had travel time, too, it just felt natural. But make no mistake. It was still wrong, because when bullets landed on your client, they didn't land on your target on the server. You might have hit the player on the server, heard the feedback, but saw dirt fly up from the ground.
Basically, all real-time games have the same networking requirements.
If you made a 2D chess game today, you'd still have a game loop and networking would be a part of it, sending payloads probably only when players took their turns. That same basic design applies to FPS games.
Chess doesn't need client-side prediction, but as soon as you want to allow other players to see you moving pieces around based on cursor position, you're sending real-time data by payload over a time stepped game loop.
You just don't need to predict anything, nor do you need to interpolate or anything else.
Note that having rollback in mind from the very beginning can also impact how a fighting game is designed. The happy path of rollback involves "hiding" network jitter in the startup animations of various moves. But startup animations exist for a reason, to telegraph intent to an opponent and give them a chance to react. For moves with very little startup animation, a rollback correction can update the state of the game to several frames into the hit itself, making many moves unreactable! This feels about as bad as being shot despite ducking behind a wall in an FPS (with basically the same cause). The "solution" is to have a policy whereby you design the game such that you increase the amount of startup animation across the board, to smooth out these wrinkles. In the worst case, this is even used to justify massive built-in input delay on every input! This optimizes the game for being played online, at the expense of being played in-person. The irony being that, while the vast majority of play will happen online, all the most prestigious events (funded by the developers themselves to advertise their game as an e-sport) are in-person.
Many people cannot react to several frames, it is usually a predictive choice instead of a reaction. A great example of this is the Millie Blocker written in flash some time ago. It tests the reaction time of a move with a startup of several frames, and as it turns out is not so easy to react to without predicting. http://www.teyah.net/MilliaBlocker_v0.3.swf
Regardless of where we draw the line of reactability, with a sufficient number of rollback frames, a move that was on the reactable side will shift to the unreactable side.
I'm having a difficult time rationalizing this with how the original rollback netcode, GGPO, was developed for games that didn't even have online play at all like ST. Did those old games just have uniquely reasonable windup animations?
I mean, I'm sure at the same time you end up with things like the meme Shiki Nanaya win screen[1] but a bad connection is a bad connection will ultimately be a bad connection no matter how you slice it.
> developed for games that didn't even have online play at all
This is the kicker. The novelty of having any online play is enough to satisfy most hardcore fans of a game that would otherwise be dead. Melee fans may rant and rave about Slippy, and rightfully so, but they still don't enjoy finding out their inputs were eaten because they got shined ten frames ago. And the fact is, rollback netcode really is dramatically better than delay-based netcode for making precise muscle-memory inputs, and for most fighting games the precision required for inputs is one of the largest contributors to the skill ceiling, so hardcore fans will be happy compared to the alternative. But let's not overlook the fundamental tradeoffs: with delay-based netcode, you will always be able to react to everything that you are intended to be able to react to; with rollback, you will not always be able to react to everything that you are intended to be able to react to. Even if it's a good tradeoff, it's an inferior experience to playing locally... which is fine, because the convenience of playing online is already a tradeoff that plenty of people are willing to make. But if you design your game from the ground up to improve the experience of online play by smoothing out the reaction curve, that will probably come at the expense of offline play in some capacity. Devs don't want to balance a game twice.
You don't necessarily need to inflate startup animations or even worry about an unreactable LP/LK, the solution to this problem has existed since the very beginning: give the player control over how the rollback works!
For example, in Skullgirls you can actually tune GGPO on a sliding scale to add more input delay buffering. You can essentially simulate delay-based netcode within a rollback based game by pumping up the input delay to the max. Players will find their personal preference on the spectrum of "near-zero input lag, more rollbacks" to "high input lag, no rollbacks."
Also, if your game is running at 60FPS you're going to have a lower bound of roughly 3 frames input lag anyway, even if you were playing local. So if you tune your rollback to a 3 frame delay as a default value, the experience for most players is going to be pretty much perfect.
Indeed, you can allow players to adjust the base input delay or even have separate hardcoded input delay profiles for online and offline play, but that introduces its own problems when it comes to a player's muscle memory. In the worst case, this introduces its own design constraint where you might avoid designing your game to include frame-perfect inputs solely to make it easier for players to transition from high-delay to low-delay scenarios.
You can't hide in startup animations - those aren't just waste those are core to the game. People react to them.
You hide it in an artificial buffer of input lag.
Even the GameCube on a CRT has a few of frames of lag. So a high refresh rate monitor can actually beat that, so you can use the difference to hide network latency and still have GameCube-on-a-CRT level latency.
I've been playing Path of Exile again lately and it has the most straightforward approach to network code, it's sort of refreshing. The game works via a TCP stream to a server. By default the screen only updates when your client receives a packet from the server; if there's a hiccup the client freezes. This means it's immediately obvious when a network problem is causing trouble and you're never misled by prediction. Also looks crappy on a bad connection. There's a "Client predictive" mode you can enable but it's off by default. Details: https://www.poewiki.net/wiki/Networking_mode
I'm on Starlink so I get a lot of hiccups. It's pretty annoying. But then, it's also transparent. Lately the game is working as a diagnostic of Starlink's various failures. It helps the game has a very simple lag meter. https://imgur.com/a/S0VMjUM
PoE is a lot less demanding than a fighting game; it's mostly single player. Occasional 500ms latency and rubberbanding isn't game-destroying (just annoying). Just an interesting different approach.
I don't think it's literally TCP, but you're right that it's very much TCP-like behavior. I kept having stuttering in PoE, and eventually realized that NetworkManager was doing wifi channel scans way more often than it should be! Genuinely useful as a tool for diagnosing network conditions.
PoE oddly enough does use TCP for all of its netcode, its somewhat unique in this way compared to similar titles. I know they use a few different outgoing ports so its possible they deal with head-of-line blocking issues with parallel streams (not sure about that, just conjecture).
there isn't much difference in the happy path. the problem comes when you lose a packet and have to recover. so that's at least one additional round trip.
Head of line blocking is still an issue even after disabling Nagle's algorithm as ideally you don't want the lack of reception of older data in the stream delaying newer updates. Sometimes it's better to simply blast out all unacked state, and if intermediary packets get occasionally lost, it doesn't really matter. You can see this in live audio/video as well. If you missed your window you don't want the media data that's already arriving too late to actually play delaying the reception of newer data in the stream.
Street Fighter IV does that, only advances when both clients have advanced. There's never bullshit surprises. It does suck on a bad connection, but that's just how it has to work. Street Fighter V has rollbacks and it's absolutely terrible.
Many games before SFV used rollback netcode without issues. SFV in their implementation somehow managed to implement rollback in a way in which sometimes it was purely one sided.
Given delay based netcode, or rollback netcode, I'll always pick rollback. Played alot of SF4 purely online and in a game with plenty of 1-2 frame links, delay based netcode really advantages the player mashing reversal within your string.
One exploit with rewinding game state is making sure game clients don't forge that they were "first" with timing abuse. But the standard strategy is to ignore commands too far back in time. So yes, the strategy does create an exploit vector, but it's easily addressed and attempting to use it as a cheater can have negative effects on your gameplay anyway, so it's not without its cons.
Too late to edit my post, but this was achieved with a technique called lag switching[1] back in the day. Now I feel old. :(
Modern cheats just do it in software. These days, I think the technique is antiquated in any strategy due to its limited success with modern multiplayer implementations.
Yeah, don't use the term "netcode" which originated from gamers. Look up resources on multiplayer client-side prediction, interpolation, reconciliation, and lag compensation.
Together those techniques constitute what laypersons consider "netcode," which ignores still fundamental details like payload construction, serialization, deserialization, broadcasting, heartbeats, etc.
For instance, networked visibility affects bandwidth consumption, but no one talks about it or worries about it until you're playing with a cheater with wallhacks.
But wait! Modern rendering techniques cull on the GPU. So you have, you guessed it, more and more potential for modern games to broadcast all player positions at all times, unless the developers know to segment player visibility and not network their presence behind non-translucent surfaces.
Forget 'netcode' - that came from people who don't know about game development. A good networked game will be made to work well at a foundation level. There is no such thing as 'netcode'. It isn't like a shader that has it's own language and is sectioned off from the rest of the program.
Question why is rollback networking good for games? To play devil's advocate with the lag hidden wouldn't this lead me to make decisions without considering the possibility that the opponent has already made a decision that is opposite of the prediction I'm seeing is.
With no rollback code then I feel the lag and can properly make decisions based on not only what I see right now but the exact amount of lag I'm feeling. How I play the game should be adjusted based on the latency of the lag. With rollback that feedback of the latency is hidden.
You're referring to "leading the shot." This was common for games in the late 90s and early 2000s.
The problem is that leading the shot has two side effects: 1. what you shoot at in game isn't what you're hitting on the server, (you shot a player, but dirt kicked up from the ground several feet in front of them) and 2. some gameplay mechanics rely on accurate replication.
If you throw a grenade at a player and "lead the shot" maybe it sticks to them on the server. It kills them.
But on your side, it missed because you personally lag compensated, stuck to a vehicle, and now the vehicle blew up instead and flew off a cliff.
Now the game sends back updated entity positions and the vehicle and its occupants are back on the ground and everything just snapped for you.
You can rewind game state and make it not visible to players, then run hit calculations and do things like kill players and destroy objects without snapping.
Proper lag compensation implementations make it such that infrequently do you actually notice lag, even in exceptionally poor connection environments.
So consider a first person shooter with no rollback you're probably going to be aiming slightly ahead of the person to compensate for latency lag. And maybe you'll aim to the other side of the person if you think they changed direction before you saw it actually happen. Here because you feel the lag of your own movements you can take that into account in knowing how far ahead to shoot on the left side or the right side of your opponent.
With rollback if you guess they are going to continue moving in the same direction you just aim right on where you see them because of the prediction. If you think they are moving in the other direction you need to guess when they started moving in the other direction. That we are using rollback networking can't get rid of the need for you to guess this. To make that guess you'd like to know how far back they could have changed direction so you have some time interval in which to guess. From your perspective that time interval is based on the difference in time between the server's state now and the last server state you received. But my point is doesn't rollback make it impossible for you to know that while playing the game.
Whether you're using rollback or not you as the player need to know intuitively what is the time difference between the server's state and the latest server state received.
No, I mean with lag compensation, you don't actually have to do that. The server calculates the difference between when you sent the input and its current game state, then performs some abstract lag compensation function, (this is usually moving all game objects back in time to where they were relative to when the command was sent) then replays game logic.
The side effect is that if anything of importance gameplay-wise did occur back in time, this abstract lag compensation function would reset just enough game state to make a difference.
In other games, the abstract lag compensation function might need to be something else, but for most purposes, I assume moving entities or actors back in spacetime is adequate. After the lag compensation function and game logic has completed, you call a second function, the inverse abstract lag compensation function. This is the equivalent, or should be, of the opposite operations performed in the first function. So, in this default case, moving all entities or actors back to where they were.
You wouldn't want to replay sounds for example, because you don't want to do anything that gets perceived by players, but the general idea is that lag compensation itself is "invisible" to players.
That is to say, if an enemy is moving on your screen in California, no matter how fast they're traveling, if you shot them from Phoenix or New York City, and sent that input back to the server in San Francisco, it should be calculated as damage applied, regardless of latency, up to a maximum latency tolerance such that players are not able to abuse lag compensation up to some upper temporal limit within reason.
For reference, the Source 1 stores player positions for up to a second.
Ok interesting there is a lag compensation feature. I did not think of the server having something like that precisely because of what you mentioned it sounds like this is something the players can use to abuse or cheat with.
On more thought I don't see how what you're saying is relevant to my point. The server needs to maintain the current state (which is based on the latest input it has received). No matter how you do the networking rollback or not you can't escape latency causing a problem in perceiving the wrong state on the clients. Thats why with rollback players sometimes see the enemy teleporting. The networking code cannot remove the need for the player to "lead the shot" in some sense.
Correctly implemented lag compensation doesn’t result in players teleporting.
If a player drops packets for a while, they shouldn’t be teleported to a position once they start sending packets again. They’re sending input not positions.
When I say lag compensation removes latency-based leading-the-shot scenarios, I mean it, full stop. It’s completely removes this problem.
You’re describing teleporting which is caused by other malimplemented player movement and networking code.
That is wrong. Every description of rollback mentinos that. Replaying inputs after rolling back can result in a new state that is different from the client predictino which amounts to teleporting.
No, it isn't. You're talking about a player-controlled object. But you're describing "leading the shot," which occurs when other objects teleport, which doesn't happen with lag compensation.
96 comments
[ 1.3 ms ] story [ 253 ms ] threadRollback has been in the industry for years now. The whole article reads like there's basically large swathes of the fighting game subset of the gaming industry who don't know anything about multiplayer game networking.
Which is understandable considering the game industry at-large doesn't know a thing about multiplayer game networking. Go checkout an out-of-the-box build of Unreal Engine 5. Yeah, they've got Nanite wow!, but do they have Actors who don't rubberband? No, and it's stupid and frustrating. It makes the whole engine feel like a joke. Especially compared to Source 1 which had client-side prediction and rollback support over dial-up. Just plain dumb.
If you think that's bad, Unity does basically nothing for you, and Godot pretends to have multiplayer support by saying it has ENet bindings. Wow, thanks!
It's disappointing that firms can chase physically based rendering, and invest so many man-hours into asset development but can't recreate what's existed for literal decades.
Also, AFAIK QuakeWorld was THE client-side prediction implementation to match. Maybe it was developed prior to John Carmack, but I don't know of any major software that did so with a timestep. Everything else followed. This guy doesn't know what he's talking about.
Counter-Strike: Source had rollback in 2004 and Yahn Bernier wrote about it in "Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization" in 2001.[1] Keep up dude.
[1]: https://www.gamedevs.org/uploads/latency-compensation-in-cli...
An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves
Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar with PPP life. I'm adressing it now.
The first move was to scrap the current net code. It was based on a reliable stream as its original primitive (way back in qtest), then was retrofited to have an unreliable sideband to make internet play feasable. It was a big mess, so I took it out and shot it. The new code has the unreliable packet as its basic primitive, and all the complexities that entails is now visible to the main code instead of hidden under the net api. This is A Good Thing. Goodbye phantom unconnected players, messages not getting through, etc
It was Carmack - I've always loved this article https://fabiensanglard.net/quakeSource/johnc-log.aug.htm
But I'm almost certain there were rudimentary client-side prediction models that existed before John Carmack, which is why I say I think he was the first to "invent" it, but frankly I'm not so sure about that.
I mean think about terminals, as an example.
I do think with near certainty he was the first to invent the client-server with client-side prediction architecture in time stepped software, or more specifically in an FPS.
It pales in comparison to older engines from other game companies. It's effectively unusable.
Basically, Tim Sweeney's approach for Actor movement wasn't to replicate state accurately, but just trust that clients would eventually get the right data. This isn't the same as eventual consistency mind you, because game state constantly changes in fast-paced games.
You can have whole time spans where Actors have absolutely no accurate player position replication.
[1]: https://docs.google.com/document/d/1KGLbEfHsWANTTgUqfK6rkpFY...
1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider...
I've tried on many occasions to get excited about these engines and started a project with them. Unfortunately, when the entire realm of multiplayer feels half-assed my hopes are dashed catastrophically.
Right now, I am spending every moment of my free time developing my own game/engine from scratch. I honestly don't give a shit about Nanite, AAA graphics, et. al. Complex multiplayer experiences are far more compelling to me than raytraced graphics and sub-pixel triangles.
I think the exact nature of the net code is much more intimately linked to the game concept than anything in the rendering pipeline (for a multiplayer experience like Quake or Counterstrike). You will never get a perfect fit if you outsource this.
Intriguing. Makes even more sense that Frost Giant Studios (who are using UE5 for their upcoming RTS) have written their own networking code (with rollback!), and thrown away other big parts of the engine.
The idea of incrementally defenestrating UE5 in favor of in-house components is compelling to me. This path definitely works a LOT better in a team setting where you have actual artists looking for things to work on.
I don't know how the licensing actually works out around this though... Assume you went full ship of Theseus on UE5 and replaced 100% of it with in-house. Will Epic Games be happy walking away from this, or is there some fundamental legal component that we know we'd never screw with?
At least that is how it is in practice - the language in the license agreement is actually a little fuzzy, when our game was sold from one company to our current one, the lawyers had a bit of a frowning session over some of the language there, but ultimately decided it was okay. Maybe in the future it will be challenged, but for now, self-owned engine changes are the norm.
They have two subreddits: r/frostgiant, and r/stormgate (for their game). They regularly ask the community for feedback/discussion/brainstorming on some topics relevant to the gameplay (like pacing, asymmetry, monetisation, e-sports, etc), openly share their conclusions from the discussion, but unfortunately in-depth technical stuff doesn't really pop up often.
They specifically bragged about their pathfinding (apparently at least on par with StarCraft II), and the rollback netcode (which is "directly inspired" by fighting games, and apparently not a common thing for RTSs), but again, everything is scarce on technical details.
The unofficial Discord community (https://discord.gg/stormgate) has a "Yeti Tracker" channel that automatically cross-posts any comments made by FGS staff; there's also https://stormgatenexus.com (fan website) where many of the interviews are transcribed and distilled into articles.
I'm watching all of these closely, I can ping you if I see more engine-tech-heavy stuff popping up!
I'm dying to be proven wrong, but the intersection of people who know multiplayer networking well, Unreal Engine 4 or 5, and have successfully implemented client-side prediction with reconciliation and rollback in said software and out of it as well, are vanishingly small.
I failed to understand how one could use the existing engine's loop which is variable framerate, the existing replication system, which favors an implementation opaque RPC system over explicit payload structures, and rewrite Actor movement and engine timing subsystems without having a parallel network loop that uses none of the above.
My guess is that it is possible to use Unreal's variable framerate and replication system, but only rewrite Actor movement implementation details by creating a class that inherits from it and overrides the behavior entirely, but you're still stuck with the fact that because the Unreal Engine uses a variable framerate, you have an inconsistent rollback timeline across every client.
You want consistent frame times, in my experience, to have rollback. Does it mean that it's impossible otherwise? No, but you can get caught rolling back in-between game states that don't actually exist on the granularity of the timestep you're running at.
The code is horribly ugly, but works well. The ugliness I think is less to do with integrating with Unreal and more to do with the author though. That said, it works.
The Unreal Engine has a long history, and its origins did not start from a fixed or semi-fixed timestep. The Unreal Engine doesn't benefit from a variable timestep, it's hampered by it, and the engineers have deemed it too difficult to fix, because the decision permeates the entire engine.
There is a "fixed timestep" mode, but it's for cinematics, and specifically for synchronizing the engine with external hardware, IIRC. But it's not a true fixed timestep, it's actually a fixed framerate, instead.
[1]: https://github.com/Planimeter/game-engine-2d
And interesting, Mike Acton's point goes all the way back to the late Fred Brooks. Not the popular main topics of MMM, but the "Surgical Team" chapter, where he mentions that there should be someone taking care of the tooling. Now, in 1974 this could have been a person. In the 90s game studios it was common to have specialized teams building reusable tooling. This seems to be disappearing.
My original point, that was not well articulated at all, was more that there should be a middle ground. The choice shouldn't be only between "homegrown engine" and "Unreal/Unity oligopoly".
I'm saying there should be much more competition when it comes to third-party engines, but also in third-party tooling and third-party middleware.
And this competition should be more specialized in order to demand less of the "train and maintain a proprietary engine" but also less of the "dealing with roadblocks in UE".
(A lot of people think it's a custom engine though.)
But there's a point there, lots of recent game developers are strongly against the idea of "new" things since each new engine would be one new software and platform to learn.
The adverse reactions I occasionally receive when I talk about building my own engine are a primary source of my motivation these days.
Let me give you some even-more-unpopular opinions. I think the GPU is a gigantic distraction from fun. Low-level languages are largely a mistake for new engine/game development in 2023+. You can deliver production-quality, interactive experiences on top of GC'd languages that perform better than ASM/C/C++ implementations (both for technical reasons & human reasons).
For me, the best "game engine" on earth right now is .NET7. Nothing gives me as much control over such a broad scope of fundamental primitives. You might be balking at this point, but go look at their System.Numerics implementation. First-party, SIMD-enabled code for projection matrix calculations based upon FPS-style semantics is just the beginning. No royalty bullshit either - I get to keep 100% of the profits I extract.
I am not looking to support building WoW2 or COD. This is for a very specific creative vision.
Photon has a new group of network libraries that are essentially the evolution of "rubber band networking" that's been default for networking middleware for the last decade
One is especially interesting in that it uses IL weaving to provide a completely deterministic engine for simulating gameplay based on inputs only: https://www.photonengine.com/quantum#
The problem has been the lack of adoption/prioritization, and probably more than a little NotInventedHere-syndrome.
As a game dev of 10+ years myself, I have a few theories:
1. Multiplayer networking is the "secret sauce" that creates moats/barriers to entry for incumbent studios. Think Rocket League or Fortnite or StarCraft 2. The tech exists, it's functionally well understood, but hard to implement at a high level of quality. Why not keep it to yourself? There is more money in holding onto your game revenue monopoly than trying to sell the network tech.
2. Game genres differ greatly in their networking needs. Rollback is great for fighting games but that's about it. StarCraft uses delay based deterministic lockstep. FPS games use (typically) ad-hoc server authoritative state syncs with client side prediction. Other complex games use full determinism with rollback, sending only inputs over the wire. Some cheapo indie games using client authoritative models (open to cheating but easy to implement, arguably fine for coop games). There are even more variants and blended approaches but you get the picture: there is no one size fits all approach and many are mutually exclusive so it's harder to package into an engine as a comprehensive solution.
3. Networking a game properly involves very leaky abstractions. It is impossible to write gameplay code for a networked game without understanding the nuance of the network model. This makes it substantially harder to develop the game, and this hurts the major game engines' marketability, with both major players Unity and Unreal guilty of selling themselves as "look ma no code required" solutions. Similar to point 1, not worth the money to sell this.
I don't see a great way out of this unfortunately. The only good networking middleware I know of is Photon and they're not exactly an easy to use product either. Hopefully we see better open source tooling in the future.
I don't buy the different genres differ greatly in their networking needs, though. Dota 2 uses the same networking architecture as Counter-Strike. Both of those games use the same underlying networking techniques as RuneScape 2 (Old School Runescape). Those are a MOBA, FPS, and an MMORPG respectively.
You use lag compensation (now people refer to it as rollback) in FPS games too. Yahn famously talked about it in his paper.
In fact, any type of granular timing-dependent gameplay requires lag compensation, otherwise you end up with situations like players leading their shots or missing entirely (Halo 1 PC).
In Halo 1 PC, they actually broadcast a hit sound from the server if you actually did make bullet contact. And those weren't trace-based (hitscan) either. Halo 1 bullets actually had travel time. So, Bungie basically didn't do lag compensation correctly, and hacked a solution on top that gave players feedback to help smooth over multiplayer. Because bullets had travel time, too, it just felt natural. But make no mistake. It was still wrong, because when bullets landed on your client, they didn't land on your target on the server. You might have hit the player on the server, heard the feedback, but saw dirt fly up from the ground.
Basically, all real-time games have the same networking requirements.
If you made a 2D chess game today, you'd still have a game loop and networking would be a part of it, sending payloads probably only when players took their turns. That same basic design applies to FPS games.
Chess doesn't need client-side prediction, but as soon as you want to allow other players to see you moving pieces around based on cursor position, you're sending real-time data by payload over a time stepped game loop.
You just don't need to predict anything, nor do you need to interpolate or anything else.
(2019)
I mean, I'm sure at the same time you end up with things like the meme Shiki Nanaya win screen[1] but a bad connection is a bad connection will ultimately be a bad connection no matter how you slice it.
[1] https://i.kym-cdn.com/photos/images/original/002/287/289/67e...
This is the kicker. The novelty of having any online play is enough to satisfy most hardcore fans of a game that would otherwise be dead. Melee fans may rant and rave about Slippy, and rightfully so, but they still don't enjoy finding out their inputs were eaten because they got shined ten frames ago. And the fact is, rollback netcode really is dramatically better than delay-based netcode for making precise muscle-memory inputs, and for most fighting games the precision required for inputs is one of the largest contributors to the skill ceiling, so hardcore fans will be happy compared to the alternative. But let's not overlook the fundamental tradeoffs: with delay-based netcode, you will always be able to react to everything that you are intended to be able to react to; with rollback, you will not always be able to react to everything that you are intended to be able to react to. Even if it's a good tradeoff, it's an inferior experience to playing locally... which is fine, because the convenience of playing online is already a tradeoff that plenty of people are willing to make. But if you design your game from the ground up to improve the experience of online play by smoothing out the reaction curve, that will probably come at the expense of offline play in some capacity. Devs don't want to balance a game twice.
For example, in Skullgirls you can actually tune GGPO on a sliding scale to add more input delay buffering. You can essentially simulate delay-based netcode within a rollback based game by pumping up the input delay to the max. Players will find their personal preference on the spectrum of "near-zero input lag, more rollbacks" to "high input lag, no rollbacks."
Also, if your game is running at 60FPS you're going to have a lower bound of roughly 3 frames input lag anyway, even if you were playing local. So if you tune your rollback to a 3 frame delay as a default value, the experience for most players is going to be pretty much perfect.
If a frame is 1/60 of a second, that's not much of a design constraint.
in 2000 i had a CRT that could do 75fps, my current screens are 144 and 240 respectively.. 60 is a little bit passé
You want high refresh rate with 60fps tick rate though to avoid dropping frames due to refresh differences.
You hide it in an artificial buffer of input lag.
Even the GameCube on a CRT has a few of frames of lag. So a high refresh rate monitor can actually beat that, so you can use the difference to hide network latency and still have GameCube-on-a-CRT level latency.
I'm on Starlink so I get a lot of hiccups. It's pretty annoying. But then, it's also transparent. Lately the game is working as a diagnostic of Starlink's various failures. It helps the game has a very simple lag meter. https://imgur.com/a/S0VMjUM
PoE is a lot less demanding than a fighting game; it's mostly single player. Occasional 500ms latency and rubberbanding isn't game-destroying (just annoying). Just an interesting different approach.
A constant UDP stream of updates is typically the best way to ensure all clients have the correct state.
https://en.wikipedia.org/wiki/Nagle%27s_algorithm#Operating_...
Given delay based netcode, or rollback netcode, I'll always pick rollback. Played alot of SF4 purely online and in a game with plenty of 1-2 frame links, delay based netcode really advantages the player mashing reversal within your string.
Modern cheats just do it in software. These days, I think the technique is antiquated in any strategy due to its limited success with modern multiplayer implementations.
[1]: https://www.google.com/search?q=lag+switching
Together those techniques constitute what laypersons consider "netcode," which ignores still fundamental details like payload construction, serialization, deserialization, broadcasting, heartbeats, etc.
For instance, networked visibility affects bandwidth consumption, but no one talks about it or worries about it until you're playing with a cheater with wallhacks.
But wait! Modern rendering techniques cull on the GPU. So you have, you guessed it, more and more potential for modern games to broadcast all player positions at all times, unless the developers know to segment player visibility and not network their presence behind non-translucent surfaces.
https://fabiensanglard.net/quakeSource/quakeSourceNetWork.ph...
https://fabiensanglard.net/quakeSource/quakeSourcePrediction...
https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
With no rollback code then I feel the lag and can properly make decisions based on not only what I see right now but the exact amount of lag I'm feeling. How I play the game should be adjusted based on the latency of the lag. With rollback that feedback of the latency is hidden.
The problem is that leading the shot has two side effects: 1. what you shoot at in game isn't what you're hitting on the server, (you shot a player, but dirt kicked up from the ground several feet in front of them) and 2. some gameplay mechanics rely on accurate replication.
If you throw a grenade at a player and "lead the shot" maybe it sticks to them on the server. It kills them.
But on your side, it missed because you personally lag compensated, stuck to a vehicle, and now the vehicle blew up instead and flew off a cliff.
Now the game sends back updated entity positions and the vehicle and its occupants are back on the ground and everything just snapped for you.
You can rewind game state and make it not visible to players, then run hit calculations and do things like kill players and destroy objects without snapping.
Proper lag compensation implementations make it such that infrequently do you actually notice lag, even in exceptionally poor connection environments.
With rollback if you guess they are going to continue moving in the same direction you just aim right on where you see them because of the prediction. If you think they are moving in the other direction you need to guess when they started moving in the other direction. That we are using rollback networking can't get rid of the need for you to guess this. To make that guess you'd like to know how far back they could have changed direction so you have some time interval in which to guess. From your perspective that time interval is based on the difference in time between the server's state now and the last server state you received. But my point is doesn't rollback make it impossible for you to know that while playing the game.
Whether you're using rollback or not you as the player need to know intuitively what is the time difference between the server's state and the latest server state received.
The side effect is that if anything of importance gameplay-wise did occur back in time, this abstract lag compensation function would reset just enough game state to make a difference.
In other games, the abstract lag compensation function might need to be something else, but for most purposes, I assume moving entities or actors back in spacetime is adequate. After the lag compensation function and game logic has completed, you call a second function, the inverse abstract lag compensation function. This is the equivalent, or should be, of the opposite operations performed in the first function. So, in this default case, moving all entities or actors back to where they were.
You wouldn't want to replay sounds for example, because you don't want to do anything that gets perceived by players, but the general idea is that lag compensation itself is "invisible" to players.
That is to say, if an enemy is moving on your screen in California, no matter how fast they're traveling, if you shot them from Phoenix or New York City, and sent that input back to the server in San Francisco, it should be calculated as damage applied, regardless of latency, up to a maximum latency tolerance such that players are not able to abuse lag compensation up to some upper temporal limit within reason.
For reference, the Source 1 stores player positions for up to a second.
If a player drops packets for a while, they shouldn’t be teleported to a position once they start sending packets again. They’re sending input not positions.
When I say lag compensation removes latency-based leading-the-shot scenarios, I mean it, full stop. It’s completely removes this problem.
You’re describing teleporting which is caused by other malimplemented player movement and networking code.