Does your team have any concrete plans to reduce the TCP vs. QUIC diff with respect to goodput [1]? The linked paper claims seeing up to a 9.8% video bitrate reduction from HTTP/2 (TCP) to HTTP/3 (QUIC). Obviously, MoQ is based on a slightly different stack, so the results don't exactly generalize. I can imagine the problems are similar, though.
(I find this stuff fascinating, as I spent the last few months investigating the AF_XDP datapath for MsQuic as part of my master's thesis. I basically came to the conclusion that GSO/GRO is a better alternative and that QUIC desperately needs more hardware offloads :p)
A small note: I found the styling of your post made it annoying to read. You shouldn’t highlight key words so strongly, especially using the same green you’re using for links. It makes it take mental effort to tell them apart.
Hmm…do you have the in-browser DNS over HTTPS resolver enabled? I personally can't reproduce this, but I'm using DoH with 1.1.1.1.
I've noticed that both Chrome and Firefox tend to have less consistent HTTP/3 usage when using system DNS instead of the DoH resolver because a lot of times the browser is unable to fetch HTTPS DNS records consistently (or at all) via the system resolver.
Since HTTP/3 support on the server has to be advertised by either an HTTPS DNS record or a cached Alt-Svc header from a previous successful HTTP/2 or HTTP/1.1 connection, and the browsers tend to prefer recycling already open connections rather than opening new ones (even if they would be "upgraded" in that case), it's often much trickier to get HTTP/3 to be used in that case. (Alt-Svc headers also sometimes don't cache consistently, especially in Firefox in my experience.)
Also, to make matters even worse, the browsers, especially Chrome, seem to automatically disable HTTP/3 support if connections fail often enough. This happened to me when I was using my university's Wi-Fi a lot, which seems to block a large (but inconsistent) amount of UDP traffic. If Chrome enters this state, it stops using HTTP/3 entirely, and provides no reasoning in the developer tools as to why (normally, if you enable the "Protocol" column in the developer tools Network tab, you can hover over the listed protocol to get a tooltip explaining how Chrome determined the selected protocol was the best option available; this tooltip doesn't appear in this "force disabled" state). Annoyingly, Chrome also doesn't (or can't) isolate this state to just one network, and instead I suddenly stopped being able to use HTTP/3 at home, either. The only actual solution/override to this is to go into about:flags (yes, I know it's chrome://flags now, I don't care) and make sure that the option for QUIC support is manually enabled. Even if it's already indicated as "enabled by default", this doesn't actually reflect the browser's true state. Firefox also similarly gives up on HTTP/3, but its mechanism seems to be much less "sticky" than Chrome's, and I haven't had any consistent issues with it.
To debug further: I'd first try checking to see if EncryptedClientHello is working for you or not; you can check https://tls-ech.dev to test that. ECH requires HTTPS DNS record support, so if that shows as working, you can ensure that your configuration is able to parse HTTPS records (that site also only uses the HTTPS record for the ECH key and uses HTTP/1.1 for the actual site, so it's fairly isolated from other problems). Next, you can try Fastly's HTTP/3 checker at https://http3.is which has the benefit of only using Alt-Svc headers to negotiate; this means that the first load will always use HTTP/2, but you should be able to refresh the page and get a successful HTTP/3 connection. Cloudflare's test page at https://cloudflare-quic.com uses both HTTPS DNS records and an Alt-Svc header, so if you are able to get an HTTP/3 connection to it first try, then you know that you're parsing HTTPS records properly.
Let me know how those tests perform for you; it's possible there is an issue in Firefox but it isn't occurring consistently for everyone due to one of the many issues I just listed.
(If anyone from Cloudflare happens to be reading this, you should know that you have some kind of misconfiguration blocking https://cloudflare-quic.com/favicon.ico and there's also a slight page load delay on that page because you'...
I tested the demo at https://moq.dev/publish/ and it's buttery as hell. Very impressive. Thanks for the great technology!
Watching the Big Buck Bunny demo at https://moq.dev/watch/?name=bbb on my mobile phone leaves a lot of horizontal black lines. (Strangely, it is OK on my PC despite using the same Wi-Fi network.) Is it due to buffer size? Can I increase it client-side, or should it be done server-side?
Also, thanks for not missing South Kora in your "global" CDN map!
Chrome on my oneplus ten, I get flickering black lines routinely. The fact they're going from somewhere along the top, down towards the right makes me wonder if it's a refresh artifact maybe? It's sort of like the rolling shutter effect
There's a whole "why should I care?" section in this breathless post that doesn't explain how Media over QUIC benefits either media publishers or end users - the two most important (perhaps the only important) parties involved in this exchange. So, why should I care?
I read this as first QUIC CDN, and thought that can't be true. Dug a little deeper and learned that Media over QUIC is it's own thing. Looks pretty cool.
I love this project and I've been reading kixelated blog from time to time. I also follow him on Github.
First of all, congrats for such nice work both kixelated and Cloudflare. I have a question regarding live-streaming. Usually live-streaming is seen but hundreds or thousands of people at the same time. Are there any idea to implement/possibility to use multicast with MoQ? The issue before was that TCP was used for HTTP/1.1 and HTTP/2, but now using UDP for HTTP/3 seems like a feasible idea. I would like to hear your thoughts on that. I know folks at Akamai and BBC were working on this.
One thing that would be nice is to enable the video sender to send multiple streams. In MoQ terms, the video source can send multiple subgroups.
Why? Because transcoding = latency. Back in the day clients were weak, so transcoding on the server made sense. Today your phone can easily spew multiple streams out without breaking a sweat.
Assuming a capable device, it would provide the fastest path. And it doesn't obviate transcoding.
Yes, you certainly could use MoQ to send multiple streams (WebRTC simulcast style).
There are implementations of traditional receiver-side adaptive bitrate switching with MoQ already today (mostly switching between tracks at group boundaries). There has been interest in exploring sender-side adaptation as well, but it's not clear what that might require and if it's something worth trying to support in the first version of the spec we take all the way to RFC status. Subgroups are something that can be used today, though less experimentation (at least public experimentation) has been done on fully utilizing subgroups at this point. They could be used for independently decodable tracks, though that wastes bandwidth relative to layered codecs. Even layered codecs have some overhead that may not always be worth it. If RTT is low enough and switching is straightforward, it may be that having the original publisher publish multiple tracks into a relay and allowing the end subscriber to switch between them is sufficient.
We could really use more public experimentation with all of these approaches, so if anyone is looking to do that type of research, definitely let us know how we can help support it!
We implemented both WebTransport (for browsers) and native QUIC (for server-to-server) after discovering browsers block raw QUIC. Works today in Chrome/Edge, integrates with existing RTMP/RTSP streams in MediaMTX.
34 comments
[ 3.2 ms ] story [ 28.7 ms ] threadThanks for the award, kixelated. xD
Does your team have any concrete plans to reduce the TCP vs. QUIC diff with respect to goodput [1]? The linked paper claims seeing up to a 9.8% video bitrate reduction from HTTP/2 (TCP) to HTTP/3 (QUIC). Obviously, MoQ is based on a slightly different stack, so the results don't exactly generalize. I can imagine the problems are similar, though.
(I find this stuff fascinating, as I spent the last few months investigating the AF_XDP datapath for MsQuic as part of my master's thesis. I basically came to the conclusion that GSO/GRO is a better alternative and that QUIC desperately needs more hardware offloads :p)
[1]: https://arxiv.org/pdf/2310.09423
Also our take on MoQ https://www.wink.co/documentation/WINK-MoQ-Implementation-An...
Submitters: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html
Edit: since people seem to agree that this was the wrong move, I'm going to undo it.
Edit 2: undone now!
https://moq.dev/blog/first-cdn/
https://moq.dev/blog/first-app/
I used to work in live video platform. I have found MoQ interesting enough to work on it again.
And I guess if webview engines like Microsoft Edge WebView2 supports it then developers can use it immediately (wrapping it).
But how about from the other side, I guess OBS and YouTube must start supporting it to actually be useful?
https://bugzilla.mozilla.org/show_bug.cgi?id=1979683
I've noticed that both Chrome and Firefox tend to have less consistent HTTP/3 usage when using system DNS instead of the DoH resolver because a lot of times the browser is unable to fetch HTTPS DNS records consistently (or at all) via the system resolver.
Since HTTP/3 support on the server has to be advertised by either an HTTPS DNS record or a cached Alt-Svc header from a previous successful HTTP/2 or HTTP/1.1 connection, and the browsers tend to prefer recycling already open connections rather than opening new ones (even if they would be "upgraded" in that case), it's often much trickier to get HTTP/3 to be used in that case. (Alt-Svc headers also sometimes don't cache consistently, especially in Firefox in my experience.)
Also, to make matters even worse, the browsers, especially Chrome, seem to automatically disable HTTP/3 support if connections fail often enough. This happened to me when I was using my university's Wi-Fi a lot, which seems to block a large (but inconsistent) amount of UDP traffic. If Chrome enters this state, it stops using HTTP/3 entirely, and provides no reasoning in the developer tools as to why (normally, if you enable the "Protocol" column in the developer tools Network tab, you can hover over the listed protocol to get a tooltip explaining how Chrome determined the selected protocol was the best option available; this tooltip doesn't appear in this "force disabled" state). Annoyingly, Chrome also doesn't (or can't) isolate this state to just one network, and instead I suddenly stopped being able to use HTTP/3 at home, either. The only actual solution/override to this is to go into about:flags (yes, I know it's chrome://flags now, I don't care) and make sure that the option for QUIC support is manually enabled. Even if it's already indicated as "enabled by default", this doesn't actually reflect the browser's true state. Firefox also similarly gives up on HTTP/3, but its mechanism seems to be much less "sticky" than Chrome's, and I haven't had any consistent issues with it.
To debug further: I'd first try checking to see if EncryptedClientHello is working for you or not; you can check https://tls-ech.dev to test that. ECH requires HTTPS DNS record support, so if that shows as working, you can ensure that your configuration is able to parse HTTPS records (that site also only uses the HTTPS record for the ECH key and uses HTTP/1.1 for the actual site, so it's fairly isolated from other problems). Next, you can try Fastly's HTTP/3 checker at https://http3.is which has the benefit of only using Alt-Svc headers to negotiate; this means that the first load will always use HTTP/2, but you should be able to refresh the page and get a successful HTTP/3 connection. Cloudflare's test page at https://cloudflare-quic.com uses both HTTPS DNS records and an Alt-Svc header, so if you are able to get an HTTP/3 connection to it first try, then you know that you're parsing HTTPS records properly.
Let me know how those tests perform for you; it's possible there is an issue in Firefox but it isn't occurring consistently for everyone due to one of the many issues I just listed.
(If anyone from Cloudflare happens to be reading this, you should know that you have some kind of misconfiguration blocking https://cloudflare-quic.com/favicon.ico and there's also a slight page load delay on that page because you'...
Webtransport on Firefox currently has issues See: https://bugzilla.mozilla.org/show_bug.cgi?id=1969090
Watching the Big Buck Bunny demo at https://moq.dev/watch/?name=bbb on my mobile phone leaves a lot of horizontal black lines. (Strangely, it is OK on my PC despite using the same Wi-Fi network.) Is it due to buffer size? Can I increase it client-side, or should it be done server-side?
Also, thanks for not missing South Kora in your "global" CDN map!
My Samsung S20 shows no black lines.
I read this as first QUIC CDN, and thought that can't be true. Dug a little deeper and learned that Media over QUIC is it's own thing. Looks pretty cool.
First of all, congrats for such nice work both kixelated and Cloudflare. I have a question regarding live-streaming. Usually live-streaming is seen but hundreds or thousands of people at the same time. Are there any idea to implement/possibility to use multicast with MoQ? The issue before was that TCP was used for HTTP/1.1 and HTTP/2, but now using UDP for HTTP/3 seems like a feasible idea. I would like to hear your thoughts on that. I know folks at Akamai and BBC were working on this.
Why? Because transcoding = latency. Back in the day clients were weak, so transcoding on the server made sense. Today your phone can easily spew multiple streams out without breaking a sweat.
Assuming a capable device, it would provide the fastest path. And it doesn't obviate transcoding.
There are implementations of traditional receiver-side adaptive bitrate switching with MoQ already today (mostly switching between tracks at group boundaries). There has been interest in exploring sender-side adaptation as well, but it's not clear what that might require and if it's something worth trying to support in the first version of the spec we take all the way to RFC status. Subgroups are something that can be used today, though less experimentation (at least public experimentation) has been done on fully utilizing subgroups at this point. They could be used for independently decodable tracks, though that wastes bandwidth relative to layered codecs. Even layered codecs have some overhead that may not always be worth it. If RTT is low enough and switching is straightforward, it may be that having the original publisher publish multiple tracks into a relay and allowing the end subscriber to switch between them is sufficient.
We could really use more public experimentation with all of these approaches, so if anyone is looking to do that type of research, definitely let us know how we can help support it!
GitHub: https://github.com/winkmichael/mediamtx-moq
We implemented both WebTransport (for browsers) and native QUIC (for server-to-server) after discovering browsers block raw QUIC. Works today in Chrome/Edge, integrates with existing RTMP/RTSP streams in MediaMTX.