to implement a couch you need a key/value store with transactions so you can atomically update the by-sequence index and the key/value storage simultaneously. pouch is neat because it does this in your browser (or on top of any leveldb) so that you can replicate and sync databases anywhere you can run pouch. i'd love to see more databases implement replication but sadly not many database developers think to store by-sequence indexes or expose their databases over http
That's the technical side of the battle. I think lots of people are just waking up to the idea that the database doesn't have to be a tier. It's not natural for data that needs to be addressed all at once but it does become useful for smaller datasets that individuals interact with (which might be millions, so we're not ruling out big-data, just monolithic-big-data).
I'd call it smalldata but you already defined it http://smalldata.org/... though your definition hits other more narrow points about small data. Still, an amazing concept after some research and prototyping.
The most important part of replication is to have an algorithm that's capable of merging two document histories that are not identical. This core bit of CouchDB is quite important and often overlooked in terms of its replication scheme. Randal Leeds is currently hacking this into PouchDB and I'm quite excited to see the algorithm in a non-functional language so that more people can see its simplicity without worrying about learning a new programming paradigm.
On the other hand the atomic update of the two indexes is quite important for efficient replication. Without the atomic update of a by-update-sequence index it would require a full table scan for each replication. Its definitely a necessary optimization, but not a sufficient optimization. The per-doc revision history merging is the special sauce that makes things work.
I would go so far as to "bypass" the spec of CouchDB -- at least for Pouch -- by doing a sort of auto-compaction, clearing out the database of the previous revision at storage time, and only storing the latest revision. This doesn't stop you from doing an effective merge, it simply stops the original notion of Couch having multiple branches of the same document. Again, in the browser, to sacrifice indexedDB space, I think that's an okay step to take.
The real win will be when a WebWorker is able to run through a view and automatically add the results of that to a separate dbspace. It looks like viewQuery is the beginnings of that.
I'm the one working on this right now and this is exactly what I'm doing. However, old document bodies are not the only way to store history. Documents have a revision history, possibly 'stemmed' to a maximum height, which is an array that traces a path from a certain depth back toward the root. All the leafs are conflicts. This is the algorithm davisp is talking about.
While I'm not going to leave the document bodies of superseded revisions accessible (by overwriting them in IndexedDB), I still need to store these history paths so that merging a new revision (either via interactive edit or replication) can properly generate a conflict when applicable.
Also, the version on master does what you suggest and makes no attempt to accommodate replication-based merging, thereby dealing implicitly with merging revision histories stemmed to a height of 2. "Real" CouchDB replication is what we aim for, though, and so this algorithm must be written.
I wanna see the leveldb stuff running as a node.js implementation of Couch. Online where I can replicate with it. I almost got there once in Ruby with Booth: https://github.com/jchris/booth
Is it odd that the first thing that jumped into my mind, was running this on top of rhino to power a distrubted group of jvm nodes with no external runtimes or jndi requirements.
Edit: and then I read that it acts as a translation/functionality layer on html5 storage.
This is great news. From the beginning the CouchDB folks have been saying one of their killer features is the ability to use a local Couch in the browser for offline use, and then syncing with a remote Couch as a way to solve the perennial you-have-to-code-your-data-model-twice problem.
I feel like this was always sort of conspicuous that their "killer feature" wasn't even possible. But this code seems like a major step in that direction... and if they can deliver on that promise it gives CouchDB a serious edge over other persistence solutions in terms of code simplicity for offline-capable web apps.
And I want offline-capable web apps to make native apps fade into obscurity, so I like that.
15 comments
[ 2.5 ms ] story [ 47.9 ms ] threadI'd call it smalldata but you already defined it http://smalldata.org/... though your definition hits other more narrow points about small data. Still, an amazing concept after some research and prototyping.
The most important part of replication is to have an algorithm that's capable of merging two document histories that are not identical. This core bit of CouchDB is quite important and often overlooked in terms of its replication scheme. Randal Leeds is currently hacking this into PouchDB and I'm quite excited to see the algorithm in a non-functional language so that more people can see its simplicity without worrying about learning a new programming paradigm.
On the other hand the atomic update of the two indexes is quite important for efficient replication. Without the atomic update of a by-update-sequence index it would require a full table scan for each replication. Its definitely a necessary optimization, but not a sufficient optimization. The per-doc revision history merging is the special sauce that makes things work.
In unrelated news:
http://twitter.com/vmische/status/100956289387077633
The real win will be when a WebWorker is able to run through a view and automatically add the results of that to a separate dbspace. It looks like viewQuery is the beginnings of that.
While I'm not going to leave the document bodies of superseded revisions accessible (by overwriting them in IndexedDB), I still need to store these history paths so that merging a new revision (either via interactive edit or replication) can properly generate a conflict when applicable.
Edit: and then I read that it acts as a translation/functionality layer on html5 storage.
I feel like this was always sort of conspicuous that their "killer feature" wasn't even possible. But this code seems like a major step in that direction... and if they can deliver on that promise it gives CouchDB a serious edge over other persistence solutions in terms of code simplicity for offline-capable web apps.
And I want offline-capable web apps to make native apps fade into obscurity, so I like that.