The sample screenshot of Expedition 33 is really impressive quality considering it appears to be encoding at around 1 bit per pixel and (according to the post) it took a fraction of a millisecond to encode it. This is an order of magnitude faster than typical hardware encoders, AFAIK.
I know next to nothing about video encoding, but I feel like there should be so much low hanging fruit when it comes to videogame streaming if the encoder just cooperated with the game engine even slightly. Things like motion prediction would be free since most rendering engines already have a dedicated buffer just for that for its own rendering, for example. But there's probably some nasty patent hampering innovation there, so might as well forget it!
Instead of motion vectors you probably want to send RGBD (+depth) so the client can compute its own motion vectors based on input, depth, and camera parameters. You get instant response to user input this way, but you need to in-paint disocclusions somehow.
I love this. The widely used standards for video compression are focused on compression efficiency, which is important if you’re netflix or youtube, but sometimes latency and low complexity is more important.
Even if only to play around and learn how a video codec actually works.
> The go-to solution here is GPU accelerated video compression
Isn't the solution usually hardware encoding?
> I think this is an order of magnitude faster than even dedicated hardware codecs on GPUs.
Is there an actual benchmark though?
I would have assumed that built-in hardware encoding would always be faster. Plus, I'd assume your game is already saturating your GPU, so the last thing you want to do is use it for simultaneous video encoding. But I'm not an expert in either of these, so curious to know if/how I'm wrong here? Like if hardware encoders are designed to be real-time, but intentionally trade off latency for higher compression? And is the proposed video encoding really is so lightweight it can easily share the GPU without affecting game performance?
Very cool - That's nearly exactly what I need for a research project.
FWIW, there's also the non-free JPEG-XS standard [1] which also claims very low latency [2] and might be a safer choice for commercial projects, given that there is a patent pool around it.
Are there any solutions to game streaming that build an RPC on top of the DirectX/Vulkan API and data structures? I feel like streaming a serialized form of the command queue over the network would be more efficient than streaming video frames.
If the system knew both sides were the same vendor or used the same algorithm, would it be better to stream the scene/instructions rather than the video?
I suppose the issue would be media. Faster to load locally than push it out. Could be semi solved with typical web caching approaches.
What a great read and such a throwback for me. I worked on video compression techniques using wavelets 30yrs ago. Computing power and networking speeds were not what they are now and I had difficulty getting the backing to carry it forward. I’m so happy that this still has such active development and boundaries are still being pushed. Bravo.
One thing to note when designing a new video codec is to carpet bomb around the idea with research projects to stake claim to any possible feature enhancements.
Anything can have an improvement patent filed against, no matter the license.
One of my bucket list things is to some day build a video codec from scratch. I have no delusions of competing with h264 or anything like that, just something that does some basic compression and can play videos in the process.
It's really cool. I have always wondered if it would be possible to have video encoders designed for some specific games with prior knowledge about important regions to encode with more details. Example would be the center of the screen for the main character.
Have an LLM transcribe what is happening in the game into a few sentences per frame, transfer the text over network and have another LLM reconstruct the frame from the text. It won't be fast, it's going to be lossy, but compression ratio is insane and it's got all the right buzzwords.
Great article. One thing I've always noticed is that when you get good enough at coding it just turns into math. I hope I can reach that level some day.
This is a really nice walkthrough of matching trade offs to acceptable distortions for a known signal type. Even if you’re selecting rather than designing a codec, it’s a great process to follow.
For those interesting in the ultra low latency space (where you’re willing to trade a bit of bandwidth to gain quality and minimise latency), VSF have a pretty good wrap up of other common options and what they each optimise for: https://static.vsf.tv/download/technical_recommendations/VSF...
38 comments
[ 4.6 ms ] story [ 59.5 ms ] threadCan't wait until one day this gets into Moonlight or something like it.
Very cool work explained well.
> The go-to solution here is GPU accelerated video compression
Isn't the solution usually hardware encoding?
> I think this is an order of magnitude faster than even dedicated hardware codecs on GPUs.
Is there an actual benchmark though?
I would have assumed that built-in hardware encoding would always be faster. Plus, I'd assume your game is already saturating your GPU, so the last thing you want to do is use it for simultaneous video encoding. But I'm not an expert in either of these, so curious to know if/how I'm wrong here? Like if hardware encoders are designed to be real-time, but intentionally trade off latency for higher compression? And is the proposed video encoding really is so lightweight it can easily share the GPU without affecting game performance?
It'd be interesting to see benchmarks against H.264/AVC (see example "zero‑latency" ffmpeg settings below) and JPEG XS.
FWIW, there's also the non-free JPEG-XS standard [1] which also claims very low latency [2] and might be a safer choice for commercial projects, given that there is a patent pool around it.
[1] https://www.jpegxs.com/
[2] https://ds.jpeg.org/whitepapers/jpeg-xs-whitepaper.pdf
I suppose the issue would be media. Faster to load locally than push it out. Could be semi solved with typical web caching approaches.
You're doing something wrong if nvenc is any slower, the llhp preset should be all you need.
https://streaminglearningcenter.com/codecs/an-interview-with...
Ultra low latency for streaming.
https://www.youtube.com/watch?v=0RvosCplkCc
One thing to note when designing a new video codec is to carpet bomb around the idea with research projects to stake claim to any possible feature enhancements.
Anything can have an improvement patent filed against, no matter the license.
Here is an example of QOI being used in a video codec, https://wide-video.github.io/qov/static/demo.html https://github.com/wide-video/qov
I think the main advantage is perhaps the robustness against packet drops is better.
Maybe I should try for that next weekend.
If the author is reading this, it would be very interesting to read about the differences between this method and HTJ2K.
For those interesting in the ultra low latency space (where you’re willing to trade a bit of bandwidth to gain quality and minimise latency), VSF have a pretty good wrap up of other common options and what they each optimise for: https://static.vsf.tv/download/technical_recommendations/VSF...