The title is misleading, when they say low-delay, really they mean over UDP instead of TCP. Their answer to video over UDP is a marginal efficiency gain in a forward error correction algorithm called RLNC.
I haven't looked at RLNC, but it does seem to have some gains over more traditional FEC schemes.
As stated in the article, low delay is very hard to achieve over a reliable transport based on re-transmissions (such as TCP) - you have to wait for the loss to be detected and then the retransmitted packet to arrive. This is at least 3 x the latency of the link. Therefore if you really care about bounded low latency you need to use some form of erasure correcting algorithm (to provide upfront redundancy).
In the article we simply show that substituting one "old" algorithm for a more modern one can give you much better efficiency (protection against packet loss) for the same bandwidth and latency/delay budget.
I'm not disputing anything you mentioned but I just thought I'd point out that, at least in my experience, Soundcloud has close to instant response streaming audio. Pick a new song it starts streaming immediately. Que a different time and it jumps to that part immediately. Looking in the devtools it's pretty clear they are sending lots of small packets of audio over websockets rather than using the browser's built in audio streaming features as most music sites do. Of course it probably still has the TCP issue if there is an error but in general it's way more responsive than other music sites I've used.
Audio is much easier to stream though. You need nearly no data to start streaming audio, and it's likely you're internet will keep up.
The video is very different, the overhead is a lot more, 128kbit vs 2Mbps for example. On top of that, a video player will try to buffer ahead enough video to make sure you won't get and buffering later on, which means it needs to measure your bandwidth and try to guess how much.
UDP can be nearly instant except in for b-frames, however if it's over UDP is probably over transport stream which has it's own buffering levels built in to make sure the decoder has a large enough buffer.
So the overhead to begin a video stream is due to the size of video data? Would video behave generally more like audio data (could stream immidiately) if the typical user bandwidth was larger?
In reality it's the size yes, but there is an inherent delay because of the way B-frames work. You'll need to buffer 2-3 frames worth 33ms each (for NTSC), so 66-99ms delay regardless of throughput.
Audio doesn't have such a concept, you get 33ms worth of audio, you can play it right away.
Of course, you can opt to not have b-frames and negate that issue.
This isn't even remotely true. The brain will happily interpolate all kinds of visual information that is missing which allows you to drop frames with abandon as long as you display an older frame instead of a 'blank space'.
Audio isn't nearly as forgiving and even a relatively modest jitter or number of lost packets will result in an un-intelligible stream of audio.
I've spent many years of my life pumping video and audio across the net, even when that wasn't yet considered normal (or even an intended purpose of the net), and I've rarely heard complaints about video quality. But audio delivery requires top notch connectivity, low latency and sufficient throughput if you intend to hold a conversation with someone that does not result in irritation. Video is far more forgiving.
For something like live video conferencing, it's true the packet loss is worse for audio then video. I was responding to the previous comment about how his streaming experience over TCP was nearly instant. In the TCP case, of course you don't need to worry about loss.
> In the TCP case, of course you don't need to worry about loss.
You do, because TCP packets get lost at roughly the same rate as UDP packets do (though, there are some caveats here: on some congested links UDP packets tend to be dropped earlier).
The only major difference is that TCP will cause the packet to be - eventually - resent. The price of this complexity is added latency due to the requirement to buffer for a longer time than it could conceivably take to re-transmit packets so you can continue to stream. If you ever exhaust that time a TCP based system will grind to a halt.
In live video / real time conferencing such latency is really annoying and if it gets too long can be killing.
Elaborating on the "on some congested links UDP packets tend to be dropped earlier" ...
In our testing of lots of internet connections in the US, we've found that many of the "free" modems/wifi-routers that cable and phone companies supply really, really discriminate against UDP. It's not completely clear whether this is intentional, at least in part, or whether it's just buggy behavior.
There are several models of routers that you can reliable cause to just stop delivering UDP for a while, if you send enough UDP through them. This happens on their little internal Ethernet switches as well as over WiFi, so it's a firmware thing. Some of these routers will perk up and behave perfectly well right away, if you access one of the well-known DSL speed test sites, while flooding them with UDP packets. :-)
> however if it's over UDP is probably over transport stream
RTP is more commonly used for video calls, which doesn't have inherent buffering (but most implementations have a small jitter/reorder buffer if needed). Transport streams are better suited for broadcast, which has looser latency tolerances.
Please help me understand then. I see RLNC compared to 2022 Single mode has the same overhead of 25%. When I compare the two using your tool at the end of the article, the only change I see is an improvement from 5% random loss to 25%. The burst loose stays the same. Correct?
Yes, you get a much better random loss protection with RLNC. So if you know that all your losses are bursts you may be able to live with 2022. Essentially the difference between the two algorithms are that with 2022 only a subset of your packets are protected by a redundancy packet (and you have to choose how to protect them), whereas with RLNC you can protect all available packets. If we leave the premise of this post (namely that we want to generate traffic in the same pattern as 2022) RLNC can even protect against longer bursts compared to 2022.
Anybody have an idea how to put this into practice? I recently tried streaming video over wifi from a Raspberry (for a robotics hobby project) , and everything I tried was either unusable or very delayed.
Is there an open source low latency video streaming solution for hobbyists?
Confirmed. Best results can be had by starting with the Ubuntu Mate distro which has the rpi omx and mmal optimized ffmpeg out of the box, else steps to compile exactly right were time consuming.
Here is the core of the site. I have not had much time to keep robots on line but if you want a demo you can ping me at my username at gmail and I can pop a robot on line (during work hours if I am in the office).
I have been working on a better web interface that has on screen controls, because most people seem to want to type in like twitch, but the arrow keys are what you use.
Low delay is much, much more important for calls than for streaming. One second of buffering delay may be acceptable in streaming playback (users often contend with longer delays). That much delay will severely degrade a video call, especially if the audio stays synced with the delayed video.
There is justification in regarding that as a form of video call rather than media playback, even though the video may be only in one direction and the reverse communication (flow of bids) isn't AV.
That really depends on the service. For on-demand services like Netflix or YouTube your argument is a fair one. But for live services it matters a lot more - particularly in broadcasting. Even more so if your clients are watching the same feed and using that to determine when to inject ads (eg in a live sporting event). Then it quickly becomes very important to keep latency as low as possible.
What is the distinction that you're making here? A video call seems like a use-case of video/audio streaming, not something separate. There are cases where low-latency streaming isn't important, but the article is talking about cases where it is.
Anyone know what the best streaming solution for browser <-> browser video calling is? It probably has to be built on top of WebRTC but I'm wondering if there are codecs and forward error correction algorithms out there already in Javascript to use.
For small numbers of people in a call (say, n < 5), the "best" -- meaning lowest latency -- solution is direct browser to browser WebRTC connections. Both Chrome's and Firefox's WebRTC implementations have quite good FEC built in. And sending UDP packets directly between peers will have much lower latency than routing through a media server.
Of course, sometimes peer-to-peer won't work for you. Maybe you have requirements that push you towards routing media through a server. (Content filtering, or compositing video or mixing audio, for example.) Or maybe you have more than a few people in a call. If so, upstream bandwidth and encoding become bottlenecks for mesh/peer-to-peer. Finally, some firewalls won't allow UDP traffic from/to computers behind them, so you'll need to route UDP through a central server, or (much worse) tunnel over TCP.
Back on the subject of latency and error correction in WebRTC, here are some fun links:
Mozilla article from when they first turned on Opus FEC. Includes sample audio for calls with 19% packet loss. (19% packet loss is very, very bad. My startup makes a browser-to-browser video calling tool, and we try hard to deal well with packet loss that high, but it's a losing battle.) https://blog.mozilla.org/webrtc/audio-fec-experiments/
Periscope developed a Low-Latency HTTP Live Streaming (LHLS) technique that relies on HTTP chunked transfer-encoding to stream video bytes as they are encoded at the origin. This is still subject to TCP packet retransmission overhead, but the time-to-first-byte is reduced significantly and leads to less buffering on the client.
Most systems that serve HLS media use fixed content-length segments, which requires knowledge of the length of a segment before the first byte can be sent over the wire. So, for a 5 second segment you would need to encode the entire 5 seconds before the first byte can be sent; this does not apply when streaming the segments with chunked transfer encoding.
Incidentally, at Mux we also use chunked transfer-encoding to stream video that is encoded on-the-fly with great performance.
I've heard from colleagues that this won't be possible with DASH due to the switch to fMP4 format. One of my co-workers tells me that fMP4 requires the entire segment to be loaded before playback can begin while TS segments don't require this. We've been looking into very small segments (e.g. 1s duration) to reduce latency but I've been interested in the LHLS approach since I first heard of it.
fMP4 has an index for each chunk, so you have to buffer the whole thing to create the index on the writing side. However, with DASH you do have the option for WebM as well, which does not need and index and can be streamed. Or really small fMP4 segments work too.
Very short segment durations are effective only when latency is more important than quality.
Each TS segment must start with a key-frame, and the GOP size can't exceed the duration of a segment (e.g. one second). Lowering the segment duration increases the frequency of key-frames, which has the effect of lowering the quality you can achieve at a given bitrate.
> I've heard from colleagues that this won't be possible with DASH due to the switch to fMP4 format.
That's incorrect. With DASH the latency depends on the fragment duration, not the segment duration. You can start sending the segment when its first fragment is generated, and use chunk-based HTTP transfer as mentioned in other comments.
Anyone know what PSNow/ cloud gaming services use? I’m using it to play some ps3 games and the delay is immediately noticeable when you start playing, but your brain adjusts and it’s not noticeable. Has to be around 300-400 ms round trip
46 comments
[ 3.4 ms ] story [ 103 ms ] threadI haven't looked at RLNC, but it does seem to have some gains over more traditional FEC schemes.
https://arxiv.org/abs/1802.04873
In the article we simply show that substituting one "old" algorithm for a more modern one can give you much better efficiency (protection against packet loss) for the same bandwidth and latency/delay budget.
The video is very different, the overhead is a lot more, 128kbit vs 2Mbps for example. On top of that, a video player will try to buffer ahead enough video to make sure you won't get and buffering later on, which means it needs to measure your bandwidth and try to guess how much.
UDP can be nearly instant except in for b-frames, however if it's over UDP is probably over transport stream which has it's own buffering levels built in to make sure the decoder has a large enough buffer.
Audio doesn't have such a concept, you get 33ms worth of audio, you can play it right away.
Of course, you can opt to not have b-frames and negate that issue.
This isn't even remotely true. The brain will happily interpolate all kinds of visual information that is missing which allows you to drop frames with abandon as long as you display an older frame instead of a 'blank space'.
Audio isn't nearly as forgiving and even a relatively modest jitter or number of lost packets will result in an un-intelligible stream of audio.
I've spent many years of my life pumping video and audio across the net, even when that wasn't yet considered normal (or even an intended purpose of the net), and I've rarely heard complaints about video quality. But audio delivery requires top notch connectivity, low latency and sufficient throughput if you intend to hold a conversation with someone that does not result in irritation. Video is far more forgiving.
You do, because TCP packets get lost at roughly the same rate as UDP packets do (though, there are some caveats here: on some congested links UDP packets tend to be dropped earlier).
The only major difference is that TCP will cause the packet to be - eventually - resent. The price of this complexity is added latency due to the requirement to buffer for a longer time than it could conceivably take to re-transmit packets so you can continue to stream. If you ever exhaust that time a TCP based system will grind to a halt.
In live video / real time conferencing such latency is really annoying and if it gets too long can be killing.
In our testing of lots of internet connections in the US, we've found that many of the "free" modems/wifi-routers that cable and phone companies supply really, really discriminate against UDP. It's not completely clear whether this is intentional, at least in part, or whether it's just buggy behavior.
There are several models of routers that you can reliable cause to just stop delivering UDP for a while, if you send enough UDP through them. This happens on their little internal Ethernet switches as well as over WiFi, so it's a firmware thing. Some of these routers will perk up and behave perfectly well right away, if you access one of the well-known DSL speed test sites, while flooding them with UDP packets. :-)
RTP is more commonly used for video calls, which doesn't have inherent buffering (but most implementations have a small jitter/reorder buffer if needed). Transport streams are better suited for broadcast, which has looser latency tolerances.
Did that make sense?
Is there an open source low latency video streaming solution for hobbyists?
Here’s a gif of RPi streaming through a CDN which itself is fast, showing timestamps, using janky ffmpeg I built (aka. Worst-case): https://twitter.com/iSpoogeDaily/status/986681611600084992?s...
Tuning is ongoing, learning as I go. Thanks for client buffering pointer.
All the code is on GitHub. I can get very low latency video. Most of the delay comes in the form of the speed of light being so slow.
On the web site in use jsmpeg.
I have been working on a better web interface that has on screen controls, because most people seem to want to type in like twitch, but the arrow keys are what you use.
https://github.com/mbrumlow/webbot
Try using kurento [0].
[0] https://www.kurento.org/
Of course, sometimes peer-to-peer won't work for you. Maybe you have requirements that push you towards routing media through a server. (Content filtering, or compositing video or mixing audio, for example.) Or maybe you have more than a few people in a call. If so, upstream bandwidth and encoding become bottlenecks for mesh/peer-to-peer. Finally, some firewalls won't allow UDP traffic from/to computers behind them, so you'll need to route UDP through a central server, or (much worse) tunnel over TCP.
Back on the subject of latency and error correction in WebRTC, here are some fun links:
Draft spec for FEC in WebRTC: https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08
Mozilla article from when they first turned on Opus FEC. Includes sample audio for calls with 19% packet loss. (19% packet loss is very, very bad. My startup makes a browser-to-browser video calling tool, and we try hard to deal well with packet loss that high, but it's a losing battle.) https://blog.mozilla.org/webrtc/audio-fec-experiments/
Notes from the very knowledgeable folks at Callstats.io about WebRTC FEC. Covers some of the same material as this thread's original post: https://www.callstats.io/2016/11/09/how-to-recover-lost-medi...
Tsahi Levent-Levi's benchmarks showing how a few different media servers perform in the context of 10% packet loss: https://testrtc.com/webrtc-media-server-packet-loss/
Here's a Periscope post about LHLS: https://medium.com/@periscopecode/introducing-lhls-media-str...
Most systems that serve HLS media use fixed content-length segments, which requires knowledge of the length of a segment before the first byte can be sent over the wire. So, for a 5 second segment you would need to encode the entire 5 seconds before the first byte can be sent; this does not apply when streaming the segments with chunked transfer encoding.
Incidentally, at Mux we also use chunked transfer-encoding to stream video that is encoded on-the-fly with great performance.
Each TS segment must start with a key-frame, and the GOP size can't exceed the duration of a segment (e.g. one second). Lowering the segment duration increases the frequency of key-frames, which has the effect of lowering the quality you can achieve at a given bitrate.
That's incorrect. With DASH the latency depends on the fragment duration, not the segment duration. You can start sending the segment when its first fragment is generated, and use chunk-based HTTP transfer as mentioned in other comments.
Link for further details on low latency: https://www.gpac-licensing.com/2014/07/09/lowering-dash-live...