HN Explorer

Ask HN: Recommendations for Peer to Peer Frameworks?

276 points by zilchers ↗ HN
I'm looking for peer to peer framework recommendations to build a distributed app. In particular, a framework that supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

I'm not particular about language (Java, Go, Javascript, etc), since it's for a new project. I found this thread from 2016 (https://news.ycombinator.com/item?id=11145691), it had some interesting recommendations, but quite a few were out of date, and with the big push to distributed apps nowadays, I thought it might be a good time to readdress the topic.

Some of the solutions form the other thread or that I've found:

- Smokesignal (Javascript / Node, not updated in >10 months) https://github.com/marcelklehr/smokesignal

- ZeroTier: https://www.zerotier.com/product-netcon.shtml

- Maki: https://maki.io

- Telehash: http://telehash.org/

- A whole host of blockchain projects, Bitcoin, Ethereum, HyperLedger, Scorex (Scala blockchain)

Bitcoin is an interesting option, it certainly has a large user base and is well tested and secure code, but forking it has it's own set of issues.

77 comments

[ 3.0 ms ] story [ 149 ms ] thread
Dowwie
How about IPFS? OpenBazaar is an IPFS case study. They recently released a revamped v2 built with Go.
zilchers
That's interesting, I hadn't thought about IPFS - it doesn't really have a strong message passing semantic so much as it has good replication properties, but maybe you could build a distributed log on top of it.
jjallen
By good "replication properties" you mean "having to run many scripts/nodes yourself" to get widely replicated that is true.

We found IPFS in general to be awesome software that just wasn't quite ready for production.

There's IPFS-cluster[0] but that's still in capital-letters ALPHA.

https://github.com/ipfs/ipfs-cluster

xendipity
Yeah, I'd look at IPFS's networking library, libp2p. It's the peer-to-peer backbone of IPFS and has a big community and active development https://libp2p.io/. It's also implemented in both go and js (as most of IPFS is)
eeZah7Ux
The wire protocol is practically almost not documented.
jjallen
We moved away from libp2p at the project I work on because of major instability problems in browsers. Our dapp that used libp2p would crash after a few minutes and was generally not easy to use.

EDIT: I'm referring to the JS version here.

lgierth
Sorry to hear you had move to something else -- my guess is these crashes were from WebRTC, which isn't very performant in browsers yet. Especially with more than a handful of WebRTC connections, we've seen it crash regularly.

We've switched the default browser-to-browser transport in js-libp2p to a websockets relay recently, so you should give it another try.

ComputerGuru
Did you try to investigate the stack trace or file a bug upstream? No snark intended!
jjallen
Yes, of course, and it was WebRTC. We ended up with a better solution (encoding the metadata we were storing in IPFS in a bytes32 stored in the Ethereum blockchain).

There were extra scripts and setup. Overall moving off simplified our stack, so it was probably a good move despite libp2p's crashing.

Could be totally fixed by now for all I know.

That said, it was believed that most Protocol Labs resources were/would be devoted to Filecoin, which is probably a safe bet.

lgierth
We have no intention of letting libp2p stagnate - blog post with announcements coming soon!
adamnemecek
Look into crdt it’s the shit for collaborative editing (idk if that’s what you ares looking for tho).
malloryerik
Do you happen to know of a decent and easy-ish crdt library in Go?
zilchers
CRDT is definitely a good pattern for the data passing itself, but one of the really tough parts of these distributed apps is the plumbing work, just getting peers to reliability communicate. That's part of what I'm looking for.
rauljordan
Look into libp2p created by the Protocol Labs team (created IPFS). We're thinking of moving to this framework internally at Ethereum.
TACIXAT
Second vote for libp2p. It looks very mature and generic enough to build whatever off of it. It has implementations in js and go.

If you end up rolling your own, I found js to have the most building blocks (pmp and upnp libraries).

fro0116
Can anyone point to some higher-level libraries or open-source apps built on top of libp2p?
lgierth
Apart from the obvious js-ipfs and go-ipfs, check out ipfs-cluster and LivePeer. libp2p is actually meant to be pretty high-level so I'm not aware of more high-level libraries.
lgierth
Actually, a few minutes later, diggan set my understanding of your question straight :) Check out peer-crdt, the ipfs/y.js bridge, ipfs-pubsub-room, and orbitdb!
no_identd
>We're thinking of moving to this framework internally at Ethereum.

Why not GNUnet? Check the recently released 181 pages long dissertation on it:

https://grothoff.org/christian/habil.pdf

tlrobinson
GPL license is probably a non-starter.
thomastjeffery
Why?
tlrobinson
> The core of Ethereum includes the consensus engine, the networking code and any supporting libraries. For C++, this includes libethereum, libp2p, libdevcore, libdevcrypto, libethcore, libevm and libevmface.

> The core of Ethereum will be released under the most liberal of licenses. This reflects our desire to have Ethereum used in as many diverse environments as possible, even those which, for various reasons can require modifications or augmentations to the software which cannot be released to the public.

https://github.com/ethereum/wiki/wiki/Licensing

misrab
definitely the most promising out there
0XAFFE
Maybe Scuttlebutt? https://www.scuttlebutt.nz/

Scuttlebutt is advertised as a social network, but in core it is an eternal distributed log. You can pass either public or private messages and assets around.

zilchers
This is a great suggestion, have started looking into it. Thanks!
freshhawk
Scuttlebutt and Dat (https://datproject.org/) are definitely looking like good bets right now.
yoshuaw
Also perhaps fun to note is that we're currently hacking on a Rust version of Dat. Hopefully we'll be able to bring the Dat protocol to more languages and platforms soon! https://github.com/datrs
robotmay
Oh man that makes me happy. I just posted about that this morning. Hopefully I can find a way to contribute!
[deleted]
(comment deleted)
petegordon
I’m curious what your goal is. Do you want the complete immutability of transactions like bitcoin/blockchain; or do you want the peer/seed capability like BitTorrent and IPFS with a DHT (Kademlia)? I’d be interested in seeing a good list of the technology choices that can make up a P2P project if anyone knows of one, please share.

But you definitely should checkout IPFS and Kademlia.

dnomad
I2P works well enough.
mcjiggerlog
Maybe check out Dat[0] - especially useful if you need some kind of distributed data store. The main implementation is in Javascript.

[0] https://datproject.org/

robotmay
I really wish there was an additional implementation in something like Rust (personal preference, but C would be good as it's fairly easy to bind to) for DAT, because what they're doing is rather cool and the experience is tidy.
indescions_2018
Awesome List of P2P resources:

https://github.com/kgryte/awesome-peer-to-peer

Personal bias tilts to web tech. WebRTC will require you to centrally implement all your peer broker and logic. But take a look at something like Peer5 Video CDN to appreciate scalability:

https://www.peer5.com/

inglor
Hey, Benji from Peer5 here - our product is really cool and helps you scale your video delivery but I don't believe it works in this case.

We solve the problem of distributing HLS/DASH delivery over large p2p swarms and scaling video. Doesn't sound like it could work for OP.

Thanks a lot for keeping us in mind though :)

adam_ellsworth
Hey Benji. I saw you guys setting up at NAB yesterday. Would there be a good day this week to come by and talk shop?
inglor
Hey Adam, I'm not attending NAB this year - Shachar and Hadar would love to talk (shachar@peer5.com or hadar@peer5.com) in terms of technical people attending :)
wslh
Re blockchain technologies: while there is an obvious intersection between the world of blockchain and P2P, blockchain technologies are slow because BFT adds many requirements to the game. Basically blockchain technologies use P2P to reach a common state and this could not be necessary in your use case.

You can take a look at OpenBazaar and Particl for P2P implementations.

no_identd
The 181 pages long GNUnet dissertation very recently got released: https://grothoff.org/christian/habil.pdf

I'd strongly recommend going for GNUnet, nobody takes me serious when I say this, but it WILL surplant IP eventually.

Ixiaus
I agree. I think gnunet gets a lot right.
yid
> but it WILL surplant IP eventually.

The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

no_identd
>The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

GNUnet can function as an overlay and an _underlay_ simultaneously, quoting from the dissertation:

> GNUnet is an overlay network, as it is initially supposed to primarily operate over the existing Internet network. However, GNUnet does not assume that this is always the case. For example, GNUnet can also operate directly over WLAN or Bluetooth to create an ad-hoc wireless mesh network, and with the GNUnet “PT/VPN” subsystems we can run TCP/IP over GNUnet. So in this configuration, GNUnet would be an underlay network, a bit like MPLS or B.A.T.M.A.N. [NAL07]. In reality, we in fact expect to see both at the same time: some peers will run over the existing Internet, while others may connect to GNUnet on Layer 2.

martindale
Maki 0.3 will include components from Fabric: https://fabric.fm — automatically available to anyone upgrading from 0.2. :)
JamesLeonis
The WebTorrent Project [1] has a whole suite of tools for decentralized applications based around BitTorrent. It's not a framework per-se, but it breaks up the protocol into libraries for your use.

If you're interested in Ethereum, it has other non-blockchain technologies that you can use for your apps. It has the Swarm network for storing bits of data, the Whisper network for chat and messaging, as well as Public-Key signing to verify messages. I'm using the latter functionality for a demo to replace password as the security mechanism.

[1]: https://github.com/webtorrent

tomp2p
TomP2P (https://tomp2p.net/) is a library handling firewall / UPNP support, however, the focus is academic/experimental.