One of the most interesting things to me about CRDTs, and something that a skim of the article (with its focus on low-level CRDTs) might give the wrong impression on... is that things like https://automerge.org/ are not just "libraries" that "throw together" low-level CRDTs. They are themselves full CRDTs, with strong proofs about their characteristics under stress.
Per the Automerge website:
> We are driven to build high performance, reliable software you can bet your project on. We develop rigorous academic proofs of our designs using theorem proving tools like Isabelle, and implement them using cutting edge performance techniques adopted from the database world. Our standard is to be both fast and correct.
While the time and storage-space performance of these new-generation CRDTs may not be ideal for all projects, their convergence characteristics are formalized, proven, and predictable.
If you're building a SaaS that benefits from team members editing structured and unstructured data, and seeing each others' changes in real time (as one would expect of Notion or Figma), you can reach for CRDTs that give you actionable "collaborative deep data structures" today, without understanding the entire history of the space that the article walks through. All you need for the backend is key-value storage with range/prefix queries; all you need for the frontend is a library and a dream.
Automerge is a great project, but it feels still way to academic in it's setup. If you need a superior DX and CRDT-based full-stack database, I'd recommend you to look at Triplit.dev and their docs.
(while development has decreased somewhat, the product is in a fully-featured phase and should work well for anything from small to medium, probably also very large projects depending on your configuration). Give it a try, you will like it.
Well yeah, who expected them not to be a full blown CRDT? Similarly, I like Loro (https://loro.dev) but the fundamental problem remains that they're document based without a good query engine, ie you have to literally target a specific nested entry in the CRDT to get the data you want.
Could this be solved by (periodic) snapshotting into a system with good support for indexing in nested documents, for an initial eligibility search, then perhaps re-snapshotting with live updates, mixing in recently updated documents by the current user for read-your-writes optics, as a final in-memory filter?
CRDTs are something you still have to write by hand, I finished creating a custom sequence based CRDT engine about 2 months ago (inspired by diamond types) and it was hilarious to ask Ai for assistance.
It's interesting when you are working on something that:
1. Is essentially a logic problem.
2. That LLMs aren't trained on.
3. That can have dense character sequences when testing.
4. To see how completely useless an LLM is outside of pre-trained areas.
There needs to be some blackbox test based on pure but niche logic to see if an LLM model is capable of understanding and even noticing exposure to new logics.
what this calls OR-Set looks equivalent to what Monotone uses (used? It's kinda mostly dead now) for merging scalar values (eg names, content hashes) since 2005.
Interesting read. I’ve spent the past two years developing my own CRDT, but along the way, I realized a CRDT involves too many trade-offs, so I ended up implementing an ID-based OT framework. Coincidentally, I’m planning to launch it this Tuesday, so here’s an exclusive for you: https://docnode.dev. I'd like to hear your thoughts!
In the future, I plan to add a CRDT mode for scenarios where P2P is required.
The thing I find interesting about CDRT and OT is it’s built to solve people typing in the same paragraph at the exact same time, which is something that very rarely happens in my experience. (Talking about text based collaboration aspect)
You need OT or CRDT because otherwise you'd have LWW over the entire document. However, I agree that resolving conflicts at the character/letter level isn't the best idea. Doing it at the node level is a more reasonable balance.
is there a modern operational transformation implementation that would work in P2P environment? i've had my own, but it had some convergence issues in complex fuzzied scenarios, couldnt fix before my enthusiasm ran out. I still feel like OT is more elegant and easier to integrate than CRDT. But i'm not finding good lightweight implementations.
15 comments
[ 2.7 ms ] story [ 47.4 ms ] threadSpeaking of Riak, it's still around, in the form of https://github.com/OpenRiak!
Per the Automerge website:
> We are driven to build high performance, reliable software you can bet your project on. We develop rigorous academic proofs of our designs using theorem proving tools like Isabelle, and implement them using cutting edge performance techniques adopted from the database world. Our standard is to be both fast and correct.
While the time and storage-space performance of these new-generation CRDTs may not be ideal for all projects, their convergence characteristics are formalized, proven, and predictable.
If you're building a SaaS that benefits from team members editing structured and unstructured data, and seeing each others' changes in real time (as one would expect of Notion or Figma), you can reach for CRDTs that give you actionable "collaborative deep data structures" today, without understanding the entire history of the space that the article walks through. All you need for the backend is key-value storage with range/prefix queries; all you need for the frontend is a library and a dream.
It's interesting when you are working on something that:
1. Is essentially a logic problem.
2. That LLMs aren't trained on.
3. That can have dense character sequences when testing.
4. To see how completely useless an LLM is outside of pre-trained areas.
There needs to be some blackbox test based on pure but niche logic to see if an LLM model is capable of understanding and even noticing exposure to new logics.
The best current page I can find is https://tonyg.github.io/revctrl.org/MarkMerge.html . Boo link rot.
In the future, I plan to add a CRDT mode for scenarios where P2P is required.
You need OT or CRDT because otherwise you'd have LWW over the entire document. However, I agree that resolving conflicts at the character/letter level isn't the best idea. Doing it at the node level is a more reasonable balance.