HN Explorer

Ask HN: P2P Databases?

12 points by culpable_pickle ↗ HN
I had an idea for a project that would be best served by publishing a database that all the clients would sync locally and query against directly.

I've looked around a bit and I've found BigchainDB, however it's a quite heavy load. I want to update the authoritative database and have it generate deltas that are synced to the clients via a p2p like network, rather then having anything 'mined'.

Is anyone aware of anything existing that would be of use?

15 comments

[ 2.9 ms ] story [ 46.6 ms ] thread
pkhuong
https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand.

Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!

bbrks
Couchbase Lite offers delta-sync and P2P replication over any transport. Has client libraries for all mobile OSes, Java, and recently C.

CBL is backed largely by Sqlite with features layered on top. Local query support, server replication and a bunch of other nice features useful for an offline/P2p database.

Disclaimer: Employee working on the server-side Sync Gateway component for Couchbase Mobile.

culpable_pickle
This is super cool. Thanks for letting me know about it!
tacostakohashi
Ignite works as you describe:

https://ignite.apache.org/

I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.

culpable_pickle
Thanks for this! I will give my approach more thought with a messaging system model.
GauntletWizard
Git. Seriously, git is, depending on your data model, possibly the best delta compression and sync model you can find. There pitfalls and sync issues to consider, but if it's just going to be published by one authority, git is a great solution.
culpable_pickle
This is an interesting idea. I wanted to use p2p distribution to save bandwidth costs, but maybe I have to re-evaluate with git in mind.

Thanks for the idea!

GauntletWizard
Git can be immensely P2P, though there's a coordination problem that would be involved. There was an attempt at "Gittorrent" at one point, Don't know the current status or if there was ever a working prototype.