Libs for native apps suck. But, it is compatible with React Native and works fine there.
In general, i recommend it. Only media server that i saw working (slightly) better is Jitsi, but it is 10x more cumbersome and time-consuming to learn.
Those libs for native apps are not official mediasoup components. We deliver libmediasoupclient[1] which is a native C++ implementation of the JS mediadiasoup-client[2] and uses libwebrtc[3] C++ native code directly.
Those two libs for native apps make use of libmediasoupclient, but they are not written nor maintained by us. Having said that, to the moment there is no official Android or IOS native client but there is libmediasoupclient which will be the core for both.
This is more of a low level API. Not some ready to use thing like Jitsi. That is already a big difference with alternatives.
Performance seems good:
https://webrtchacks.com/sfu-load-testing/
And it is very actively developed, which is arguably the most important metric in the current webRTC environment. Since the spec and browser implementations are a fast moving target right now and for the years to come.
Comparing Jitsi with mediasoup is like comparing Netflix (backend + apps) with Express.js + libcurl.
Jitsi developers may replace their RTC core internals (including the SFU) with mediasoup + mediasoup-client and you wouldn't even realize of it. Hope this helps.
"Cutting-edge" implies that these guys managed to apply secure E2E encryption, but I can't find any references to it on the website. Correct me if I'm wrong.
You are wrong. This is the current state of webrtc.
Without considerable hacks (patent-encumbered unaccelerated wasm ffmpeg encoding of pixel data, rolling your own SRTP-like encrypted stream over datachannels and full mesh distribution of keys) this is not currently possible for anything other than a full mesh. Whenever an SFU (effectively a MITM) is involved (needed for more than a small number of participants) e2e encryption is lost.
If/when Insertable Streams are commonplace, this will be possible without so many hacks.
Do you have any examples of a webrtc video conferencing platform that provides a sufficient user experience over full mesh with 4 or more participants on disparate internet connections?
Im just saying that this library (and similar platforms) ARE the cutting edge of webrtc. At small number of participants, they use full mesh (which is e2ee), but at larger scale, they need to use an SFU (which is not e2ee without jumping through some crazy hoops - but this is being worked on)
No (but i hope they exist, havent looked) - however not all webrtc apps are general videoconf platforms. You don't even need servers if you go full p2p.
You can certainly manually handle the signalling to set up the offer/answer. e.g. read them out to your peer(s).
You would still need some form of STUN server (but there are a number of 'freely accessible' ones, even configured as defaults in some browsers) to get your reachable address/port/proto. You cant, AFAIK, handle this manually - it is handled internally as part of ICE. Then you would need to manually handle the signalling of these as well.
Thats the bare minimum you have to do to peer over webrtc under ideal circumstances, but its certainly doable.
So you do that for each peer. If you (or a peer) change streams you will do the same thing all over again
As mentioned, you cant manually set up your listening ports - this is handled internally by ICE, which needs to connect to a STUN server to get (at a minimum) your public IP address.
edit: cant reply to your comment fulafel, but on that project you posted: https://github.com/cjb/serverless-webrtc/blob/master/serverl... Also, note that even if this wasnt defined, some browsers contain defaults. You 100% need STUN, but you can handle the signalling manually - as I stated.
edit2: cant reply to your comment ibc, but I was explaining the bare minimum 'serverless' webrtc case still required a STUN server. I appreciate that mediasoup SFU uses ice-lite instead.
And somehow this discussion become a general topic about WebRTC scenarios. They do exist, yes, but mediasoup is a SFU scenario. No STUN is required at all. TURN may be needed if the client network/router blocks UDP.
If you have N participants into the same "room", and do not have a central SFU that relays streams to others, then you have a full mesh network with all participants sending audio and video to all the others. Is that the "go full p2p" you meant?. Well, try it, and you'll see how your CPU burns when the browser/app tries to encode your webcam video source N times. And of course, you'd need N x uplink.
So you don't know what mediasoup does but you assure that "no need to use a SFU". Too much free time to comment maybe?
e2ee makes sense in one-to-one and N-to-N scenarios. It's just more complex in N-to-N scenarios because you DO need a centralized server so each participant just sends its audio/video once (to the server) and the server distributes it to others given whichever application policy/logic. That's a SFU, and that's what mediasoup does.
We may be talking past each other, I'm talking about webrtc in general as the subthread was discussinh that. Think about a limited user count screen sharing or vr app with a/v comms built in for example.
For some reason I can't reply to fulafel. But a full mesh is definitely not very doable when you're dealing with a larger amount of people. Close to all conferencing solutions today are working via an SFU for a reason. E2E encryption via the SFU gets more and more attention and is being worked into the spec. But it can't be counted as state of the art in webRTC today
Mediasoup has a more low level API. It's more of a library/framework to make your own SFU (although there's an example SFU). Jitsi is more of a ready to use SFU. Mediasoup is newer, but has seen very fast development. Last performance comparison I know of had mediasoup as a clear winner over jitsi. But that's a fast moving target and the usual limitations of a benchmark apply here as well:
https://webrtchacks.com/sfu-load-testing/
Mediasoup has a c++ core with node api. Jitsi is java.
If you're experienced with webRTC, you will probably like mediasoup better. Otherwise I'd stick with jitsi.
Comparing Jitsi with mediasoup is like comparing Netflix (backend + apps) with Express.js + libcurl.
Jitsi developers may replace their RTC core internals (including the SFU) with mediasoup + mediasoup-client and you wouldn't even realize of it. Hope this helps.
Can this be used as an alternative to Kurento? I use an ffmpeg command to send an RTP stream to Kurento which then broadcasts that to all clients that connect via WebRTC. I've had to rewrite the code that interacts with Kurento twice already. It doesn't feel like a rock solid solution. e.g. when you accidentally switch audio and video ports in the ffmpeg command the entire kurento instance crashes. I can't expose Kurento to untrusted instances otherwise there is a risk of DOS. I don't feel like running a whole Kurento instance per ffmpeg command is a sustainable solution.
It might not be obvious but I am not using WebRTC for video conferencing. My use case is basically regular live streaming but with latencies in the 100ms range.
To understand the terms: webrtc, stun, turn, mesh, sfu, mcu, ice and trickle ice, there is (1). 15 minutes to understand what is all this about. What about IPv6 stun and turn?, it seems other people asked the same idea I thought: (2)
bout all of this, one is the answer is: As IPv6 takes over the complexity of new networks, STUN and ICE will become irrelevant. I think that with the surge in video conferences and rtc, ipv6 with take off.
In my very humble opinion, I would suggest to reserve some address space in IPv6 for rtc, so that a peer is able to adopt a new special ip reserved for rtc. Nothing new under the sun, in 2014 someone comment along this line of thought (2) and (3).
(3) 2014, AshleysBrain,
https://news.ycombinator.com/item?id=7496986
I think the solution is IPv6. Once every device on the Internet is uniquely addressable again, we can do away with these NAT hacks and two endpoints should be able to reliably connect to each other again, no matter where they are. Of course, that's assuming we don't get more short-sighted engineering that breaks things again...
IPv6 will certainly REDUCE the need for STUN, but there are still (many) cases where you don't want to be "reachable by default", in which case you need a stable reference for negotiating routing and reachability (e.g. STUN).
Yeah, but if it’s reachable by default then it’s (by definition) open to the world. Otherwise (if you mean routable by default) you will still end up temporarily punching holes in your firewall, which you will need to close afterwards, and possibly recycling your ip so you aren’t still routable on that last used address. sounds like you would personally end up being a STUN server!
In order to send a datagram to multiple IP the first, and naive idea, one can think of is to change the datagram protocol to allow for multiple destination. Today, 2020, is the right time. I am thinking about platforms that have hundred or thousand of simultaneous receiving ends, so that the branching point occurs near the destination. Again, googling this proposal is not new (1) RFC 1770, category informational.
Edited: It seem that RtcDataChannel can be used with SFU, example LiveSwitch in 2018, but they don't use multiple destination datagrams (3)
More on similar proposal (2).
(1) IPv4 Option for Sender Directed Multi-Destination Delivery. The Selective Directed Broadcast Mode (SDBM) is an integral part of the U.S. Army standard for tactical data communication networks as defined in MIL-STD-188-220().
And you are wrong. RTP video cannot be sent that way because each receiver must also send back feedback to the sender. RTP protocol is more complex than just sending UDP packets everywhere.
Each receiver sending feedback does not prevent the server for using datagrams with multiple destinations. I can see that each peer use a different resolution and bitrate but that is another layer, is like sending information at several resolutions and each peer selecting the best one.
Edited: I must learn something about multicast in IPv6, the idea seems interesting.
> is like sending information at several resolutions and each peer selecting the best one.
Well, no. This is not about sending all video layers to all receivers and let them choose which one to render. Not al all.
The purpose of video simulcast/SVC is the opposite: make the SFU decide (based on estimated per receiver bandwidth or whatever) which video layers to deliver to each receiver, so a HQ video of 8 mbps does not break your Internet downlink (you just receive the lowest video layer which is 1 mbps, for example).
And more important: no, the server can not send the same UDP datagram (the same RTP packet) to all receivers. The server needs a different RTP sequence number count and a different SRTP encryption keys with each receiver.
I'm afraid this is not so easy as you say, not at all.
BTW: Do you want to say something about mediasoup? or just about your stuff?
Thanks for all the info. I think that mediasoup is a very good SFU. I wish you the best and I hope mediasoup SFU to become a crucial tool for rtc.
Edited: In a recent article (1) it seems that multicast is better than SFU in webrtc. In the PhD. Thesis (2) a hybrid model is used: Hybrid multicast-unicast video streaming over heterogenous cellular networks.
mediasoup is a SFU that must be deployed in a reachable server, so STUN is not needed at all. You may need a TURN server if a client has a restrictive firewall that blocks UDP. mediasoup is not a TURN server but you can deploy a TURN server (i.e. coturn) in your backend.
I just tried the demo - btw, best UX so far from all that I've tried.
It works fine on the same wifi network, but won't connect if one of the devices is using 4G network - is this because the TURN server is not setup? Is it easy to implement that?
The demo is just a demo. mediasoup is a low level library (no UX into mediasoup). The online demo backend does not have any TURN server. It's obviously recommended to deploy a TURN server.
Are there plans to support a TCP candidate so a TURN server isn't needed at all? It feels a bit wasteful to effectively use a TURN server as a TCP->UDP proxy for a publicly accessible server.
> TCP candidate so a TURN server isn't needed at all?
This is not true. A router may still block TCP traffic different than TLS or traffic that does not have destinatiuon port 80 or 443. So ICE TCP candidates do not avoid the need for a TURN server in certain cases.
You cannot select a specific listening port for a specific transport, because each WebRTC transport requires, at least, a different listening port in the server:
Is there a reason for the restriction of one connection per port? I would have thought you would be able to use the same port for each peer source ip/port tuple?
Not doubting you - but I never experienced this limitation with other client/server applications. I have an http server serving over 200k concurrent websockets on port 443, for example.
rfc3550 states that it is per destination ip/port tuple. So you should be able to support multiple connections per local port. Is it possible this is an oversight in the current implementation? I appreciate this isn’t TCP, which is why I have just read through all relevant RFCs.
Why is that so important? As I said, choosing a specific port is not enough. This is not TLS. An aggressive firewall may drop those TCP connections because there is no TLS data on them.
TLS port was just a thought, as I want to reduce cases where turn server is used because of a limitation with scalability (65k connection limit per turn server due to a shared source ip). But our discussion has raised another issue regarding mediasoups limitation of one source per local port - which compounds the issue.
I’m replacing a web socket server with a data channel server. If I use mediasoup then I will need to listen over 4 ip4 addresses to support the 200k clients I can currently support on 1 ip address with web sockets. Not a huge deal right now, but if I want to support millions of user it means managing 40 or so ip addresses instead of 1 or 2.
Not knocking mediasoup at all, just now aware of a limitation that sounds like it doesn’t need to exist so seeing if we can do something about it.
This is RTP not WebSocket or HTTP. Media servers need a separate port for each RTP communication. A hack could be done to make all WebRTC endpoints to use a single port in mediasoup side. However mediasoup also support plain RTP endpoints and, in those, you need to be ready to listen for RTP from any remote IP:port (you don't know it in advance due to NATs). In WebRTC we can use ICE user/pwd (previously given to the server via signaling) but that's not possible with plain/regular RTP (no ICE).
Isn’t that what the SSRC is for? I.e you use the SSRC (sent as part of media in the SDP) to identify the stream, rather than trusting an authentic stream is the only one to send to an open port? At least, that is how I understood the (multiple) rfcs. Not an expert here by any means.
In WebRTC spec (although not super mandatory but the current way to go), the client no longer signals its sending SSRCs into the SDP but a MID and optional RID values (if simulcast is in use), and those MID and RID are not supposed to be unique across all participants (not at all, but neither SSRCs are supposed to). Those MID and RID values are signaled in the SDP and then included into RTP packets as header extensions. The remote matches RTP packets based on them and then learns the associated SSRC for a faster lookup for future packets.
Anyway, WebRTC is not just about RTP. In fact, before RTP happens, ICE and DTLS must de done.
I've read many comments here asking about "how mediasoup is different than XXX" or about "mobile apps". I think the Overview in the website should be self explanatory, I'll just paste a fragment here:
---------------------
Design goals of mediasoup and its client side libraries:
- Be a SFU (Selective Forwarding Unit).
- Support both WebRTC and plain RTP input and output.
- Be a Node.js module in server side.
- Be a tiny JavaScript and C++ libraries in client side.
- Be minimalist: just handle the media layer.
- Be signaling agnostic: do not mandate any signaling protocol.
- Be super low level API.
- Support all existing WebRTC endpoints.
- Enable integration with well known multimedia libraries/tools.
Use cases:
- Group video chat applications.
- One-to-many (or few-to-many) broadcasting applications in real-time.
- RTP streaming.
Just to easily explain that mediasoup is not a replacement for Jitsi or Zoom, but a low level set of libraries for building build different kind of real-time applications, including multi-party videoconference apps (such as Jitsi or Zoom) and others completely different.
The performance as mentioned on webrtchacks seems promising, I was planning to go with openvidu but now i am more inclined towards mediasoup. My only concerns are, please provide simple tutorials with demo and code explanation (it would be 30 lines of code) so that beginners like me easily understand it. If you check out openvidu they have provided demos for every use case with explanation. It is hard for a beginner to understand all this stuff and there isn't a js v3 broadcasting demo available. Also if you could mention some performance results on various machines like c5.large, c5.xlarge, c5.2xlarge with number of participants it would be helpful.
Thanks a lot
68 comments
[ 6.6 ms ] story [ 157 ms ] thread* How good is this compared to other alternatives? (Or why choose this instead of something else?)
* Where are the client apps for this, more importantly on mobile?
And here's the source code for the demo: https://github.com/versatica/mediasoup-demo/
Libs for native apps: https://github.com/haiyangwu/mediasoup-client-android
https://github.com/ethand91/mediasoup-ios-client
In general, i recommend it. Only media server that i saw working (slightly) better is Jitsi, but it is 10x more cumbersome and time-consuming to learn.
Those libs for native apps are not official mediasoup components. We deliver libmediasoupclient[1] which is a native C++ implementation of the JS mediadiasoup-client[2] and uses libwebrtc[3] C++ native code directly.
Those two libs for native apps make use of libmediasoupclient, but they are not written nor maintained by us. Having said that, to the moment there is no official Android or IOS native client but there is libmediasoupclient which will be the core for both.
[1]: https://github.com/versatica/libmediasoupclient [2]: https://github.com/versatica/mediasoup-client [3]: http://webrtc.github.io/webrtc-org/native-code/
And it is very actively developed, which is arguably the most important metric in the current webRTC environment. Since the spec and browser implementations are a fast moving target right now and for the years to come.
Comparing Jitsi with mediasoup is like comparing Netflix (backend + apps) with Express.js + libcurl.
Jitsi developers may replace their RTC core internals (including the SFU) with mediasoup + mediasoup-client and you wouldn't even realize of it. Hope this helps.
Without considerable hacks (patent-encumbered unaccelerated wasm ffmpeg encoding of pixel data, rolling your own SRTP-like encrypted stream over datachannels and full mesh distribution of keys) this is not currently possible for anything other than a full mesh. Whenever an SFU (effectively a MITM) is involved (needed for more than a small number of participants) e2e encryption is lost.
If/when Insertable Streams are commonplace, this will be possible without so many hacks.
Edit: I see I was downvoted for correcting you, but there was an article on this very subject yesterday: https://webrtchacks.com/you-dont-have-end-to-end-encryption-...
This platform - and jitsi/janus/zoom/whatever - all use an SFU for more than a handful of participants.
I don't know what Mediasoup does, was just commenting on what webrtc can do re the "the current state of webrtc" subject.
Im just saying that this library (and similar platforms) ARE the cutting edge of webrtc. At small number of participants, they use full mesh (which is e2ee), but at larger scale, they need to use an SFU (which is not e2ee without jumping through some crazy hoops - but this is being worked on)
You would still need some form of STUN server (but there are a number of 'freely accessible' ones, even configured as defaults in some browsers) to get your reachable address/port/proto. You cant, AFAIK, handle this manually - it is handled internally as part of ICE. Then you would need to manually handle the signalling of these as well.
Thats the bare minimum you have to do to peer over webrtc under ideal circumstances, but its certainly doable.
So you do that for each peer. If you (or a peer) change streams you will do the same thing all over again
edit: cant reply to your comment fulafel, but on that project you posted: https://github.com/cjb/serverless-webrtc/blob/master/serverl... Also, note that even if this wasnt defined, some browsers contain defaults. You 100% need STUN, but you can handle the signalling manually - as I stated.
edit2: cant reply to your comment ibc, but I was explaining the bare minimum 'serverless' webrtc case still required a STUN server. I appreciate that mediasoup SFU uses ice-lite instead.
If you say it's different with media channels, I'll believe you.
WebRTC can tell you your ip address without any external server, that only breaks if you are dealing with NAT.
edit: to clarify the NAT-less use case, i'm thinking of apps that can rely on/require p2p supporting IPv6 connectivity.
> You don't even need servers if you go full p2p
If you have N participants into the same "room", and do not have a central SFU that relays streams to others, then you have a full mesh network with all participants sending audio and video to all the others. Is that the "go full p2p" you meant?. Well, try it, and you'll see how your CPU burns when the browser/app tries to encode your webcam video source N times. And of course, you'd need N x uplink.
So you don't know what mediasoup does but you assure that "no need to use a SFU". Too much free time to comment maybe?
e2ee makes sense in one-to-one and N-to-N scenarios. It's just more complex in N-to-N scenarios because you DO need a centralized server so each participant just sends its audio/video once (to the server) and the server distributes it to others given whichever application policy/logic. That's a SFU, and that's what mediasoup does.
The product I work on signals this to the user, and asks for confirmation before "upgrading" to an SFU.
jitsi has been production tested far longer i suppose, through its freely available videoconferencing service https://meet.jit.si/
Mediasoup has a c++ core with node api. Jitsi is java.
If you're experienced with webRTC, you will probably like mediasoup better. Otherwise I'd stick with jitsi.
Comparing Jitsi with mediasoup is like comparing Netflix (backend + apps) with Express.js + libcurl.
Jitsi developers may replace their RTC core internals (including the SFU) with mediasoup + mediasoup-client and you wouldn't even realize of it. Hope this helps.
It might not be obvious but I am not using WebRTC for video conferencing. My use case is basically regular live streaming but with latencies in the 100ms range.
Edit: I just compared the docs https://raw.githubusercontent.com/versatica/mediasoup/v3/art... vs https://doc-kurento.readthedocs.io/en/6.13.0/features/kurent... This is exactly what I need!
In my very humble opinion, I would suggest to reserve some address space in IPv6 for rtc, so that a peer is able to adopt a new special ip reserved for rtc. Nothing new under the sun, in 2014 someone comment along this line of thought (2) and (3).
So what are we waiting for?
(1) https://webrtcglossary.com/ (2) https://www.quora.com/Will-the-IPv6-result-in-the-death-of-S...
(3) 2014, AshleysBrain, https://news.ycombinator.com/item?id=7496986 I think the solution is IPv6. Once every device on the Internet is uniquely addressable again, we can do away with these NAT hacks and two endpoints should be able to reliably connect to each other again, no matter where they are. Of course, that's assuming we don't get more short-sighted engineering that breaks things again...
IPv6 will certainly REDUCE the need for STUN, but there are still (many) cases where you don't want to be "reachable by default", in which case you need a stable reference for negotiating routing and reachability (e.g. STUN).
Edited: It seem that RtcDataChannel can be used with SFU, example LiveSwitch in 2018, but they don't use multiple destination datagrams (3)
More on similar proposal (2).
(1) IPv4 Option for Sender Directed Multi-Destination Delivery. The Selective Directed Broadcast Mode (SDBM) is an integral part of the U.S. Army standard for tactical data communication networks as defined in MIL-STD-188-220().
(2) https://www.researchgate.net/publication/238663190_IPv4_Opti...
(3) https://www.frozenmountain.com/developers/blog/archive/indus...
And you are wrong. RTP video cannot be sent that way because each receiver must also send back feedback to the sender. RTP protocol is more complex than just sending UDP packets everywhere.
Edited: I must learn something about multicast in IPv6, the idea seems interesting.
Well, no. This is not about sending all video layers to all receivers and let them choose which one to render. Not al all.
The purpose of video simulcast/SVC is the opposite: make the SFU decide (based on estimated per receiver bandwidth or whatever) which video layers to deliver to each receiver, so a HQ video of 8 mbps does not break your Internet downlink (you just receive the lowest video layer which is 1 mbps, for example).
And more important: no, the server can not send the same UDP datagram (the same RTP packet) to all receivers. The server needs a different RTP sequence number count and a different SRTP encryption keys with each receiver.
I'm afraid this is not so easy as you say, not at all.
BTW: Do you want to say something about mediasoup? or just about your stuff?
Edited: In a recent article (1) it seems that multicast is better than SFU in webrtc. In the PhD. Thesis (2) a hybrid model is used: Hybrid multicast-unicast video streaming over heterogenous cellular networks.
(1)https://ieeexplore.ieee.org/document/8811590 (2) https://summit.sfu.ca/item/16802
It works fine on the same wifi network, but won't connect if one of the devices is using 4G network - is this because the TURN server is not setup? Is it easy to implement that?
https://mediasoup.org/documentation/v3/mediasoup/api/#WebRtc...
> TCP candidate so a TURN server isn't needed at all?
This is not true. A router may still block TCP traffic different than TLS or traffic that does not have destinatiuon port 80 or 443. So ICE TCP candidates do not avoid the need for a TURN server in certain cases.
For example, I could supply the generated udp and tcp candidates in addition to a tcp:443?
What are your thoughts?
https://mediasoup.org/documentation/v3/mediasoup/api/#WebRtc...
YouIf you want to listen in TLS 443 for all clients, add a TURN server into your backend. Just that.
Not doubting you - but I never experienced this limitation with other client/server applications. I have an http server serving over 200k concurrent websockets on port 443, for example.
I'm happy to help out with this if I can.
I’m replacing a web socket server with a data channel server. If I use mediasoup then I will need to listen over 4 ip4 addresses to support the 200k clients I can currently support on 1 ip address with web sockets. Not a huge deal right now, but if I want to support millions of user it means managing 40 or so ip addresses instead of 1 or 2.
Not knocking mediasoup at all, just now aware of a limitation that sounds like it doesn’t need to exist so seeing if we can do something about it.
Anyway, WebRTC is not just about RTP. In fact, before RTP happens, ICE and DTLS must de done.
TL'DR': Pornhub uses mediasoup.
I've read many comments here asking about "how mediasoup is different than XXX" or about "mobile apps". I think the Overview in the website should be self explanatory, I'll just paste a fragment here:
https://mediasoup.org/documentation/overview/
--------------------- Design goals of mediasoup and its client side libraries:
- Be a SFU (Selective Forwarding Unit). - Support both WebRTC and plain RTP input and output. - Be a Node.js module in server side. - Be a tiny JavaScript and C++ libraries in client side. - Be minimalist: just handle the media layer. - Be signaling agnostic: do not mandate any signaling protocol. - Be super low level API. - Support all existing WebRTC endpoints. - Enable integration with well known multimedia libraries/tools.
Use cases:
- Group video chat applications. - One-to-many (or few-to-many) broadcasting applications in real-time. - RTP streaming.