Thanks Feross for creating webtorrent. The biggest problem I see with this and also ipfs is that the latency is up to 30seconds even for files that are distributes over many clients. Is there any work in progress to reduce the latency?
Maintainer of WebTorrent here: the problem is on how WebRTC works and its various settings. Using a good STUN servers with your own tracker server should be the best solution. Using TURN servers also helps peers connect, but then it's no longer real P2P. Current work is being dedicated to updating the code base to ES6 and soon to ES7, which will improve many things and force recent versions of Node/Browsers to be used. See https://github.com/webtorrent/bittorrent-tracker/issues/199
Why would you take perfectly good code that works across many browsers and force breaking changes onto existing NodeJS/Browsers? WebRTC Condit, STUN, TURN, etc. do not require advanced language features.
Most, if not all, ES6 features are supported where WebRTC is supported, and it has great benefits for both users and developers. async/await and other ES7 have been supported for 14 versions of chrome now, but that change will come later this year, so we'll be supporting 95%+ of the market. Anyone can always use webtorrent as a module and transpile it.
From the theoretical point of view we can get huge improvements both in terms of torrent availability and latency, by making use of "fountain codes".
Basically it is a family of error correcting code which splits the data in N+M distinct pieces, and allow you to reconstruct the data once you have any distinct N of these pieces. This allow you to seed any one of these small pieces yet allow the reconstruction of the whole data provided there are enough other people seeding other pieces (Typically number of "seeds" will increase between x100 and x1000 (Note that it obviously don't increase the outgoing bandwidth of your network, but it makes it a lot easier to conserve and distribute rare files). Which also means any of these pieces can help getting the first packets needed to begin streaming which reduces latency.
The negative aspects, are :
- bigger "torrent" files (because you need to store the hashes of more pieces).
- Some time you'll download an extra piece.
- More compute power needed on the clients to decode the stream.
- More complexity in the code.
Why is it not implemented ? Mainly because it is not a standard yet. There is quite a lot of freedom in the ways to implement it, and it will probably fracture the pools with plenty of non compatible protocols. And if you have a new version of the protocol to improve some things you need the seeds to re-encode the data to be protocol compatible.
That is also probably a subject which is encumbered by software patents.
Economically targeting the fat tail of rare torrents is probably not worth it.
It's fun algebra you should check if you don't know it already :)
It’s pretty straightforward math! I did an M:N implementation in Elixir via on shamir’s key sharing [1]. As you say, there’s no real standard for how to pack the hash bytes for even basic Shamir’s and so I followed Hashicorp Vault’s impl and tested it to their test data. Multiplexing an entire file over this wouldn’t be too hard, but the index/hash table would be need a good protocol. Wasn’t there some cloud storage systems tried using M:N at one time..
Plus the recombination computation can get expensive over lots of data.
Quite the opposite (it's like a RAID storage system), because of the error correcting scheme, you need a lot less storage to provide a good level of redundancy. You only need to store pieces of size 1/N * file size, and as long as you can get N of those pieces you can reconstruct the file. To guarantee a high level of availability, you only need that there are more than N hosts up containing distinct pieces. So typically you will use (~3 * size of file) of storage on your network, instead of (number of seeds * size of file) bits, and still provide with the same level of service as if number of seeds where seeding the whole file. So each node is able to store a lot more pieces of other files.
I might need to look up how they work because that's basically what my next question is: how do the chunks overlap and how does that affect the probability of a file being unrecoverable?
If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out.
I guess I just want to know, given arbitrary p, what the probability of the file being gone is.
Error correction codes are quite magical.
Instead of splitting in 100 pieces each piece 3 times, you split it in 300 distinct pieces of size 1/100 * filesize, which any 100 can reconstruct the file. So basically you can drop 2/3 of the pieces and yet be able to reconstruct the file.
So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number.
For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.
Well - and dat - and perhaps secure scuttlebutt, right?
This is built with WebRTC - so it uses the browser's native peer-to-peer API. But these other protocols also have WebRTC libraries, so it kind of all washes out. I only know dat really well, so I'm not an expert on all of this yet - but I'd say that the choice of protocol is less important than the maturity of the software.
For me, I was drawn to dat because of its incredible wealth of libraries (and the sweet Beaker Browser) - but if you're just serving video over WebRTC, you'd need to jump in and see how each library works for your specific project. In my mind, these are all really great projects.
So for keeping something in ipfs you need to have a server running ipfs seeding the files. You can use ipfs.js but it still needs to connect to an IPFS server. Webtorrent don't require any server running. All you need is the webtorrent.js file. That said you can still use a third party service like Infura for putting files in IPFS without running your own server. But Infura discards the file after seeding it for like 2 weeks. So you are not in total control of the file. In webtorrent you don't need any third party service. All you need is a browser with the webtorrent.js file. At the same time it is reccomended to run a STUN server to increase the reach of the files seeded by your webtorrent application. If you are creating a webtorrent application without a STUN server chances are that a lot of people would not be able to connect to it. I think IPFS on the other hand don't have this issue. Once a file is uploaded and is seeded by atleast one server it should be accessible to everyone else. (I am not 100% sure about this though). So choose wisely.
It is also the advantage of webtorrent. All a person need to seed a file in webtorrent is open a browser window. You can't rely on users of your application to start an IPFS server and seed the files.
If you want to seed a file for long term you can use the webtorrent-cli or webtorrent app as well.
See also https://github.com/mozilla/libdweb for stuff that browsers should/could have that would make much better to write true decentralized applications.
Financial cost of self expression isn't much of a barrier. Having static website hosting in public cloud is affordable to quite many, few dollars in month and more if you have traffic. How does WebTorrent affect load times for users of your site?
23 comments
[ 0.25 ms ] story [ 58.1 ms ] threadBasically it is a family of error correcting code which splits the data in N+M distinct pieces, and allow you to reconstruct the data once you have any distinct N of these pieces. This allow you to seed any one of these small pieces yet allow the reconstruction of the whole data provided there are enough other people seeding other pieces (Typically number of "seeds" will increase between x100 and x1000 (Note that it obviously don't increase the outgoing bandwidth of your network, but it makes it a lot easier to conserve and distribute rare files). Which also means any of these pieces can help getting the first packets needed to begin streaming which reduces latency. The negative aspects, are : - bigger "torrent" files (because you need to store the hashes of more pieces). - Some time you'll download an extra piece. - More compute power needed on the clients to decode the stream. - More complexity in the code.
Why is it not implemented ? Mainly because it is not a standard yet. There is quite a lot of freedom in the ways to implement it, and it will probably fracture the pools with plenty of non compatible protocols. And if you have a new version of the protocol to improve some things you need the seeds to re-encode the data to be protocol compatible.
That is also probably a subject which is encumbered by software patents.
Economically targeting the fat tail of rare torrents is probably not worth it.
It's fun algebra you should check if you don't know it already :)
Plus the recombination computation can get expensive over lots of data.
1: https://github.com/elcritch/keyx
While this might reduce latency, it sounds like in practice it'd multiple the redundancy of the system and make it even less cost effective?
If I have a stupid simple scheme where I split a file into 100 pieces and replicate each piece 3 times, I have a 100C1 * p^3 chance of irrecoverability where p is the independent probability of pieces dropping out.
I guess I just want to know, given arbitrary p, what the probability of the file being gone is.
So as long as your nodes uptime is greater than 33% + eps, the file is not gone. The eps is quite small because of the law of great number.
For example if your node uptime is 50%, the pieces will be in average available on 300 * 50% = 150 +- C*uptimevariance /sqrt(N) nodes which is greater than 100 with probability ~ 1, so the file will be recoverable almost surely.
This is built with WebRTC - so it uses the browser's native peer-to-peer API. But these other protocols also have WebRTC libraries, so it kind of all washes out. I only know dat really well, so I'm not an expert on all of this yet - but I'd say that the choice of protocol is less important than the maturity of the software.
For me, I was drawn to dat because of its incredible wealth of libraries (and the sweet Beaker Browser) - but if you're just serving video over WebRTC, you'd need to jump in and see how each library works for your specific project. In my mind, these are all really great projects.
Or just start using IPv6 already.
If you want to seed a file for long term you can use the webtorrent-cli or webtorrent app as well.
This should also be in the spirit of Tim Berners-Lee.