Blog post author here. It definitely seems that I worded that part poorly. I hope the comments on the blog help clear this up, but it's a bit hard to talk about in the abstract.
If you have an app that's using the Sync API, feel free to reach out to talk specifics.
Deprecating an API is a bad way to go, especially if the API is fairly new. This post is telling me two things:
1) Dropbox's API was not designed correctly the first time. It should not just be a representation of their internal stuff exposed publicly. It needs to be designed first.
2) Dropbox is struggling to understand the difference between providing a consumer product (like their normal Dropbox product) and a developer product, like this API.
Maybe or maybe not. We don't know what the exact numbers were and what they needed to be to keep them in tact. They may have deprecated them anyway to keep a single SDK (at least that's kind of the impression I got though if the usage was high the timeline would probably be longer and more features would be in the core SDK).
It's easy to lambast someone because someone didn't want to rely on another's API but it's a very good point to bring up. Big companies such as Google regularly kill things people use and start-ups are encouraged to think fast, test fast; I have a hard time wanting to rely on either for my business.
Developers that build products on top of APIs controlled by other companies, particularly recent startups, are routinely abused. The upside of these products is unnaturally clipped due to moral hazard on the part of the API provider. That is, closed APIs incentivize their providers to farm out testing new products based on their data set to others, then reimplement the most successful products themselves, and finally cut off access to the developer because it's now a 'competitor'. This pattern has (allegedly) played out with several of the big closed API providers.
This behavior has a double benefit to the API provider, because it potentially turns would be direct competitors who would otherwise work on an alternative to its core technology if not provided with an API into unpaid new product idea validators that leave the API provider the option of crippling their product at will. That's a lot of competitive advantage, especially in markets dominated by network and first mover effects.
As a developer, it's prudent to be skeptical of closed API providers, because developer time is valuable, and the incentives of API providers and consumers are not aligned.
An API has to be extremely valuable in order to overcome these structural issues, and it appears that developers didn't regard these APIs as sufficiently valuable. So in my mind, the blame for this failure doesn't lie with the developers that failed to use the APIs, but with the company that didn't provide a sufficiently attractive API.
We don't know the right scale of desired adoption. For some companies 10k users is a lot but for others like Google, even with tenth of millions [1] they shutdown services like Google Reader.
Number of users using an API does not determine if it's deprecated or not. If they are making more money (directly or indirectly) than they are spending to run/maintain the API is what determines it most times. There are countless examples of heavily used API's being pulled, Twitter and Netflix spring to mind...
I actually wanted to use the Datastore API for my app. A lot of stuff was happening on the client side so it would have been a perfect match. Now I will probably manage the user's data on the server using RethinkDB.
I actually wanted to use the Datastore API for my app. A lot of stuff was happening on the client side so it would have been a perfect match. Now I will probably manage the user's data on the server using RethinkDB.
If you want an open source full featured alternative (with native libraries for most mobile platforms), we've put decades of engineer time into Couchbase Mobile.
Basic feature set: local storage and query, binary attachments, background sync, fine grained access control. Also goodies like webhooks and p2p capabilities.
Does it still work against directly CouchDB? Had some fun times with the mobile/CouchDB sync.
When Sync Gateway came along things got complicated (to me). Mostly because of another component to deploy, monitor etc. Did you consider bundling Sync Gateway into a Couchbase server, even if only for dev/staging environments?
Works great with Apache CouchDB, PouchDB, Cloudant, etc. The mobile libraries and Sync Gateway all use the CouchDB sync protocol.
We built Sync Gateway because the Apache CouchDB access control model wasn't fine grained enough for most apps, so it adds a data routing function to control sync access. (See link downthread).
Thanks for the feedback about simplicity. Our next release makes the Sync Gateway default config a lot more beginner friendly and even today we ship with a dev mode that uses a tiny embedded database instead of a Couchbase Server cluster for storage.
The problem is the datastore API worked directly in the browser via javascript. You could deploy an app on a CDN with that,just reference a few api keys and start writing apps using Dropbox as a datastore.
For that scenario PouchDB would run in the browser and connect to Sync Gateway using CORS. Of course you'd still have to install Sync Gateway somewhere first, which is more work than outsourcing to a provider like Dropbox, but also more suitable for enterprise deployments.
I haven't even used half of what Couchbase does, but it's been a great replacement for Core Data for me. I've very much loved not having to deal with schema migrations that historically have been very tedious with Core Data.
Be EXTREMELY cautious when using document-based auto-syncing DBs such as firebase. If you're not closely following the best-practice such as using flat document structure instead of nested ones, and dictionaries instead of arrays, you may enter a world of nasty disappointments as you struggle to query and update your local data structure.
Data syncing is hard and so far i've only seen two possibilities : roll your own solution using the fact that you have a knowledge of your business that will help you be smart and small, or use a generic solution but take some time to understand deeply its way of working before you build your real product.
I actually had to cancel what i thought would be a short angular/firebase project after two days struggling to sync one big nested data structure displayed on a page, and realizing i actually had to rethink my model from scratch to fit the technology (which ended up being completely overkill for my need anyway).
"The Datastore API is somewhat unique in that it, unlike the Core API, deals with non-file data, so we’re working directly with individual developers to help them migrate to an alternative."
I would be interested in what this help means concretely for developers. I imagine Dropbox has limited interest helping small developers who might have used this API.
Concretely, it means we emailed everyone by hand with a Datastore API app that had a non-trivial number of users and offered to help. (If anyone didn't get an email, you might only have a couple test users, but feel free to reach out to us yourself.)
With most developers who respond, we're digging into their data model and trying to suggest alternatives (possibly files in Dropbox or possibly alternative services like Parse and Firebase).
The nice thing about dropbox's mobile sdks is they have very cleanly solved the problem of how to get data synced to a server despite some users having a very spotty connection. The bad thing about dropbox's mobile sdks is they force all data to be stored in the end users dropbox folder and not in our enterprise folder.
I would gladly use their datastore apis if I could avoid having my users go through the hassle of authenticating to dropbox on their own and instead put it into my enterprise storage.
35 comments
[ 3.2 ms ] story [ 90.4 ms ] threadA few years later
"Nevermind, you'll need to build sync yourself. We believe in you!"
Is this a fair assessment?
If you have an app that's using the Sync API, feel free to reach out to talk specifics.
1) Dropbox's API was not designed correctly the first time. It should not just be a representation of their internal stuff exposed publicly. It needs to be designed first. 2) Dropbox is struggling to understand the difference between providing a consumer product (like their normal Dropbox product) and a developer product, like this API.
Dropbox reached out to me a while ago when my product got on HN to use their Sync APIs.
It's easy to lambast someone because someone didn't want to rely on another's API but it's a very good point to bring up. Big companies such as Google regularly kill things people use and start-ups are encouraged to think fast, test fast; I have a hard time wanting to rely on either for my business.
This behavior has a double benefit to the API provider, because it potentially turns would be direct competitors who would otherwise work on an alternative to its core technology if not provided with an API into unpaid new product idea validators that leave the API provider the option of crippling their product at will. That's a lot of competitive advantage, especially in markets dominated by network and first mover effects.
As a developer, it's prudent to be skeptical of closed API providers, because developer time is valuable, and the incentives of API providers and consumers are not aligned.
An API has to be extremely valuable in order to overcome these structural issues, and it appears that developers didn't regard these APIs as sufficiently valuable. So in my mind, the blame for this failure doesn't lie with the developers that failed to use the APIs, but with the company that didn't provide a sufficiently attractive API.
[1] http://www.lifehacker.com.au/2013/03/how-many-users-does-goo...
http://developer.couchbase.com/mobile/
Basic feature set: local storage and query, binary attachments, background sync, fine grained access control. Also goodies like webhooks and p2p capabilities.
When Sync Gateway came along things got complicated (to me). Mostly because of another component to deploy, monitor etc. Did you consider bundling Sync Gateway into a Couchbase server, even if only for dev/staging environments?
We built Sync Gateway because the Apache CouchDB access control model wasn't fine grained enough for most apps, so it adds a data routing function to control sync access. (See link downthread).
Thanks for the feedback about simplicity. Our next release makes the Sync Gateway default config a lot more beginner friendly and even today we ship with a dev mode that uses a tiny embedded database instead of a Couchbase Server cluster for storage.
Don't they use the sync api?
https://www.firebase.com/
Firebase does work offline if your app loses network connectivity temporarily (details here: https://www.firebase.com/docs/web/guide/offline-capabilities...). All data written while offline is stored in memory and is synced to the Firebase servers when your client reconnects. We’re working on improving offline support when your app goes into the background, and we currently have a beta version of this available for our iOS (https://groups.google.com/d/msg/firebase-talk/dOocEtjQz4w/ss...) and Android (https://groups.google.com/d/msg/firebase-talk/gYlnLgQ-Yhk/It...) SDKs.
Data syncing is hard and so far i've only seen two possibilities : roll your own solution using the fact that you have a knowledge of your business that will help you be smart and small, or use a generic solution but take some time to understand deeply its way of working before you build your real product.
I actually had to cancel what i thought would be a short angular/firebase project after two days struggling to sync one big nested data structure displayed on a page, and realizing i actually had to rethink my model from scratch to fit the technology (which ended up being completely overkill for my need anyway).
I would be interested in what this help means concretely for developers. I imagine Dropbox has limited interest helping small developers who might have used this API.
Concretely, it means we emailed everyone by hand with a Datastore API app that had a non-trivial number of users and offered to help. (If anyone didn't get an email, you might only have a couple test users, but feel free to reach out to us yourself.)
With most developers who respond, we're digging into their data model and trying to suggest alternatives (possibly files in Dropbox or possibly alternative services like Parse and Firebase).
I would gladly use their datastore apis if I could avoid having my users go through the hassle of authenticating to dropbox on their own and instead put it into my enterprise storage.
I can't believe they got Guido van Rossum (creator of python) to write javascript though! https://github.com/dropbox/datastore-js/commit/e75e67eccecfd...
Why not open sourced the mobile sdks for sync/datastore also?