I went down this path over the last year or so while building an app. It was the first time I'd stumbled into the frontier of browser specs. I was really excited to use indexedDB and later Pouch, but got overwhelmed with how much of the spec and the various implementations are very much in flux. Eventually I just fell back to local storage because it was "good enough" for what I needed.
You're doing the lord's work, Dale. Somebody out there in the working world actually has to use and abuse these technologies that our standards-body overlords give to us to find out where the cracks and the useful parts really are. Some day it'll be as simple to pull down a bunch of objects over the network and query for subsets of them as it is to filter down a set of DOM nodes with jQuery, and it'll be partly because of the work that the Pouch team is doing. Thank you.
I had exactly this experience this summer, building a bunch of IndexedDB usage into my app, then adding a shim for places it didn't seem to work, and then finding more places that it didn't work (or I was doing something wrong, or there were other bugs), and finally ripping it all out and using synchronous localstorage calls instead. Bah.
To the credit of the standards bodies, I think speccing a database presents a totally different set of challenges than with the typical browser APIs. The fact that it took them three tries to get something everybody liked (LocalStorage -> WebSQL -> IndexedDB) is a real testament to that.
Also to the browser vendors' credit, the bar is way higher when you're implementing something like IndexedDB than the typical CSS/DOM stuff. A bug in CSS/DOM means the page looks weird; a bug in IndexedDB means your users lose data. And when you fix the bugs, you still have to migrate the existing user data, because it's sitting out there on people's hard drives. A page refresh isn't enough.
So given how insanely ambitious it is to have four independent implementations of the same database (Chromium/Firefox/IE/WebKit), I'd say the browser vendors have done a pretty great job. Firefox and Chrome in particular have shown it's possible to get it right with separate implementations; as of Chrome 38, I don't see any functional difference between the two.
I would like to take the credit, but Nolan was able to write about that stuff because it was a huge part of the work he has been doing. For my purposes supporting reasonably modern Idb and so so support for WebSQL was good enough but Nolan and others have been pushing to make sure it works great in IE, old versions of Android and iOS etc.
The bugs we have been finding have been making their way back upstream and a lot of those nightmares are down to legacy support, between modern versions of Firefox, Chrome and Android (kinda chrome) idb is pretty much conformant. hopefully IE and in a huge way Safari / iOS get themselves fixed soon.
Dale's humble, but he really is doing the Lord's work, because he set up all the Selenium/Saucelabs tests against the different browsers, fixed intermittent bugs and timeouts, etc. He's been doing all the unglamorous work that keeps us honest. No way I would have the patience for this stuff if there wasn't a big red X on my pull request to tell me I broke something in PhantomJS/IE/Safari...
And also I should point out it's not just PouchDB - the LocalForage [1] and ydn-db [2] devs have been really helpful - we all swap war stories and help file bugs on the browser vendors and such. Hopefully all this effort really will do for IndexedDB what jQuery did for the DOM. :)
If the author or someone interested in the realm of mobile DB read this : there is definitely a niche to be taken right now in p2p db syncing ( think iPad apps connected in a local network and replicating their local datasets between each others). I have been trying couchbase-lite recently which advertizes this feature, but it's clearly not their focus and it shows (dirt-slow replications, hidden timeout settings, buggy binary objects syncing, etc).
I wouldnt write couchbase-lite off entirely, sync is extremely hard and with Pouch we are still struggling with slow replications and bugs with binary objects.
The couchbase-lite devs have been contributing to PouchDB and couchbase-lite, PouchDB and CouchDB should all interoperatate in a p2p environment, if they dont then its a bug and I am sure they would love to hear it (as would we if the issue was with PouchDB)
Also Cloudant recently came out with their own libraries for iOS and Android (https://cloudant.com/blog/introducing-cloudant-sync), and in principle they should all be compatible with PouchDB/CouchDB. So I like to think we have options.
My approach is to store files in a (J)Git repo which then get indexed on each device for structured data, rather than replicating structured data directly. (i.e. imagine a contacts database where each contact is in the repo as a separate JSON doc or similar). Java only for now, thanks to the amount of libs it needs to integrate, and that it hasn't reached proof of concept yet.
CouchDB dev here. PouchDB is an amazing companion to the world of syncing databases and the CouchDB devs couldn’t be happier that the PouchDB devs are going to such great lengths pushing the sync vision of CouchDB to all sorts of browsers and devices. Thank you! <3
11 comments
[ 2.7 ms ] story [ 32.9 ms ] threadYou're doing the lord's work, Dale. Somebody out there in the working world actually has to use and abuse these technologies that our standards-body overlords give to us to find out where the cracks and the useful parts really are. Some day it'll be as simple to pull down a bunch of objects over the network and query for subsets of them as it is to filter down a set of DOM nodes with jQuery, and it'll be partly because of the work that the Pouch team is doing. Thank you.
To the credit of the standards bodies, I think speccing a database presents a totally different set of challenges than with the typical browser APIs. The fact that it took them three tries to get something everybody liked (LocalStorage -> WebSQL -> IndexedDB) is a real testament to that.
Also to the browser vendors' credit, the bar is way higher when you're implementing something like IndexedDB than the typical CSS/DOM stuff. A bug in CSS/DOM means the page looks weird; a bug in IndexedDB means your users lose data. And when you fix the bugs, you still have to migrate the existing user data, because it's sitting out there on people's hard drives. A page refresh isn't enough.
So given how insanely ambitious it is to have four independent implementations of the same database (Chromium/Firefox/IE/WebKit), I'd say the browser vendors have done a pretty great job. Firefox and Chrome in particular have shown it's possible to get it right with separate implementations; as of Chrome 38, I don't see any functional difference between the two.
The bugs we have been finding have been making their way back upstream and a lot of those nightmares are down to legacy support, between modern versions of Firefox, Chrome and Android (kinda chrome) idb is pretty much conformant. hopefully IE and in a huge way Safari / iOS get themselves fixed soon.
And also I should point out it's not just PouchDB - the LocalForage [1] and ydn-db [2] devs have been really helpful - we all swap war stories and help file bugs on the browser vendors and such. Hopefully all this effort really will do for IndexedDB what jQuery did for the DOM. :)
[1]: https://github.com/mozilla/localforage [2]: http://dev.yathit.com/ydn-db/index.html
If the author or someone interested in the realm of mobile DB read this : there is definitely a niche to be taken right now in p2p db syncing ( think iPad apps connected in a local network and replicating their local datasets between each others). I have been trying couchbase-lite recently which advertizes this feature, but it's clearly not their focus and it shows (dirt-slow replications, hidden timeout settings, buggy binary objects syncing, etc).
Just my 2 cents.
The couchbase-lite devs have been contributing to PouchDB and couchbase-lite, PouchDB and CouchDB should all interoperatate in a p2p environment, if they dont then its a bug and I am sure they would love to hear it (as would we if the issue was with PouchDB)
My approach is to store files in a (J)Git repo which then get indexed on each device for structured data, rather than replicating structured data directly. (i.e. imagine a contacts database where each contact is in the repo as a separate JSON doc or similar). Java only for now, thanks to the amount of libs it needs to integrate, and that it hasn't reached proof of concept yet.