The dark side of NoSQL (codemonkeyism.com)
There is a dark side to most of the current NoSQL databases. People rarely talk about it. They talk about performance, about how easy schemaless databases are to use. About nice APIs. They are mostly developers and not operation and system administrators. No-one asks those. But it’s there where rubber hits the road.
59 comments
[ 3.3 ms ] story [ 114 ms ] threadThere's a training obstacle. I know how SQL works and how to make SQL perform well almost by instinct. NoSQL requires me to relearn data querying. The lack of standardisation across offerings means that the knowledge I acquire will be less transferable what I've accumulated from Oracle/Postgresql/Openbase/Mysql/Sqlite.
I think you're talking of CouchApp, which allows you to create apps like that on top of CouchDB. Of course, if that was the maximum of what one was trying to do, one could get similar functionality from an RDBMS, some simple routing rules, and a CGI script that served content straight from the DB.
I guess my point is that there's not really anything intrinsic to CouchDB that makes this possible: it's a pretty short and inviting conceptual leap from "this DB does everything through an HTTP interface" to "I can run my entire application from this DB". My worry (and I'm not accusing the parent of this) is that that misses out on all of the other things that CouchDB offers (especially the ability to merge DBs that have been disconnected for long periods, which often gets lost behind the HTTP interface and MapReduce hand-waving).
What stops you from doing a one-liner from shell (or a short throw-away program) instead of a SQL query/update? (Well, except for the power of SQL compared to the APIs, of course.)
Disclaimer: I don't know enough about these newfangled things and their tradeoffs to have a serious opinion. They do look intriguing. :-)
Of course with key/value stores changing the structure of the database is harder compared to SQL databases, still designing a good data layout when building your application can help a lot to make it simple to add features to the application without to resort to scripts that rebuild the data in the key/value store.
Cassandra, for instance, provides an easy snapshot API, and you can back up the snapshots using whatever infrastructure you'd normally use to back up TB of data. (If you don't have such an infrastructure, yeah, that's a problem, but not Cassandra's fault. :)
Edit: I should point out that since Cassandra supports multiple data center replication already, I'm having trouble picturing a scenario where you want to do anything w/ the snapshots besides just leave them on the Cassandra nodes themselves (plan extra HDD capacity as necessary depending on how long you want to keep them). But some such scenarios probably do exist.
Backups are essential when you're doing major upgrades or data migrations/mangling that can fail.
OTOH, data export is not an inherent problem of distributed database either. If enough people want the feature, they can be built without too much fuss.
Obviously. The context was using snapshots as backups -- you can keep them around indefinitely, space permitting, and if you're using Cassandra's multi-datacenter features they're automatically "remote" as well.
> especially with Cassandra, where there is no builtin data versioning
At the risk of belaboring the obvious, versioning != backup, either. :)
Really the list of databases that do provide ad-hoc queries is much shorter. Neo4j (graph db), mongodb (single-master "document-oriented" db), the xml databases, probably some other minor projects.
TFA also mentions data export as a "dark side" but for projects that lack this it's really just a "nobody's needed it enough to write it yet" thing. So that's less interesting IMO than talking about ad hoc queries which is usually more of an architectural limitation.
Now if it were "the dark side of column-oriented databases," Bob's your uncle.
One powerful aspect of SQL is that the database can optimize the query. Some NoSQL databases enforces you to write joins manually in code. Unless you really know what you are doing and spend hours on it, the query will be slow. This is exactly what I see with many of our queries. Relational databases is not the answer to everything, but I would prefer to start with one rather than a NoSQL variant. You can always migrate when things begin to get too slow (ie, when you get mad success - which you probably never do anyway)
Most (?) encourage denormalization instead of joins-in-code.
> This is exactly what I see with many of our queries. Relational databases is not the answer to everything, but I would prefer to start with one rather than a NoSQL variant.
For most application domains, I agree -- but there are plenty of companies that have essentially outgrown the relational approach already.
/committer on Cassandra as well as SQLAlchemy
That's like saying "I live in a capitalist nation and capitalism is not what it's cracked up to be."
NoSQL is an ideology, and anyone you hear who uses that term is an ideologue. Please name the "NoSQL" you have been using and name the specific problems you have had with it so we can discuss matters more concretely.
Just as RDBMS users prefer one above all else, so do "NoSQL" users. I have seen my fair share of them and I thought a good chunk of them where "crap" (where crap means "does not fit my mental model of the solution"; mainly because I am too naive to study the field better and appreciate what the tools are capable of, so I dismiss anything that doesn't entirely model CL's Elephant.)
Last week I rolled out my own Memcached "replacement" in Common Lisp and I was spitting in bradfitz's general direction, that looney bloat monger! Yeah, my "memcached" replacement is an in-process memoizer for all the DB accessing functions. It worked until the server instance was overloaded and the load-balancer forked a second instance: then I had a problem. I wish I could suck-back all the spit I have sent in brad's general direction.
You're sounding a little ideologue-ish yourself. :)
NoSQL is a term of limited use since there is such a huge variety of non-relational database technologies, but that doesn't mean using it makes you an ideologue any more than someone using the term "dynamic languages," which is similarly broad.
SQL is a way to interact with a relational database, but it is not itself a database. But "NoRDBMS" still isn't as good as saying what the technology is - there are an infinite number of things that are not relational databases.
I would love to read a really detailed comparison of CouchDB (for example) to RDBMSs for common uses, by someone who thoroughly understands the relational model and is very familiar with (say) Postgres or Oracle. I like Postgres and SQLite, but I'm curious about CouchDB - it's a different kind of tool, and it probably has several complementary use cases.
Performance: Key value stores are fast. You lose features like queries by anything other than index and/or ACID guarantees. (Tokyo Cabinet or Redis).
Data structures: storing non-flat data (trees, graphs) in a SQL database == FAIL. Building a DB around a non-flat data structure lets you avoid doing O(N) queries to traverse a structure. (See Neo4j for graphs or CouchDB/Mongo for trees).
I should have said "trade-offs" to begin with...that was the heart of what I was getting at when I said "strengths".
Can you expand on this a bit? Or is this another MySQL-ism? Databases I use work just fine for trees/graphs.
You can store a tree using foreign keys for the children, but traversing the tree to level m requires m joins. Finding children at an arbitrary level requires O(tree depth) queries.
You can use preorder tree traversal which consists of mapping nodes to pairs of numbers representing intervals on the real line. Then "x is descended from y" is equivalent to "x's interval is contained in y's interval", and this at least lets you find the children of x. Going up a tree is harder. Inserting into the tree is tricky as well.
Another hack is to store on each node a string containing it's ancestors. I.e., "/2/4/27/35" indicates a node is descended can be reached from the root (node 2) by following the path 2->4->27->35. Then the regex search "2/4/27/35/.*" finds all children of node 35. This one is utter fail.
However, I freely admit I'm not a database expert. Is there some non-ugly solution I'm unaware of?
You're throwing away 90% of the advancements of the last 30 years and going back to PICK.
As for scalability: replication, sharding and backup are already problems that have been SOLVED over and over again.
It makes my head hurt.
1. Schemaless. This is a half-truth. In reality there is always some kind of schema, otherwise you'd get chaos. What works for us in Keyspace, where everything is a string, is to use "/" as a separator convention. So we have keys like: "/user/id:001" => <data>
More complicated example:
"/user/id:001/view/keyword:keyspace/"
2. Listing. Since listing happens on a prefix basis in Keyspace (give me everything that starts with "/user/id:001"), if you just want to get a list of some properties of the user, you have to create the list explicitly, otherwise you'll get all the keys under the user. Eg.
"/user/id:001/views/list/keyword:scalien" => null "/user/id:001/views/list/keyword:keyspace" => null
3. If you need to list by property X, you need to create a subtree for it, as above. This sucks if you figure this out at a later point, because you have to programatically parse your existing data and create it, besides adding it to the online code for new items. That's why RDBMSs are so good.
4. Backups. No good backup strategy at this point, hot or cold.
..
So why should anyone put up with this? You have to realize there's lots of hype right now, in reality KV stores should only be used by a small fraction of DB users. If an RDBMS is Java/C++, then a KV store is (can be) Assembly. You have much more control over what's going on. (You can of course get the same control by having a giant key-value table in your RDBMS.) You can put a cache layer in your stack more easily.
So you should think of a good KV store as something that you use under certain hi-performance cases, and even then, the KV store should offer some plus over having a giant key-value table in My/Postresql. Usually, that plus is some kind of distribution or replication that usually works. In case of Keyspace it's replication.
The other big plus is, as I learned, is that programming in the KV model is much much more natural then having embedded SQL statements in your code. A typical segment in my (Python) code is:
users = conn.listkeys("/user/id:%s/views/" % user_id) for (...):
This is much nicer then having SQL statements, and since you're using a KV store, it can't get more complicated than this. As I said, I assume you're doing hi-performance stuff, in which case some crappy ORM layer is out of the question anyway.
Here's a link about IOT for others:
http://www.oracle.com/technology/products/oracle9i/datasheet...
Personally I think there's a fairly large cultural disconnect between shops running open-source stacks and shops running Oracle, so such a comparison would not influence user choices much. But it would be very interesting for an implementor like me: if Oracle is faster, which is likely at this point, it would set a performance point to beat.
As a final fun fact, Keyspace is currently running on top of BerkeleyDB, which is an Oracle property =) But not for much longer, I'm very unsatisfied with BDB, it doesn't live up to its image of being an industrial strength btree store (it sucks).
Quite possibly. Our company does use a lot of open source in the Java & Perl layers, but we're pretty tightly locked in to Oracle on the backend. I think it may be true that the sheer vastness of Oracle tempts you into believing you can solve every problem the Oracle way, which is why IOTs came to mind. I'd love to experiment with other options, myself, but its difficult once everything you do is predicated on it.
Also --- what do you plan to replace it with? I know that AllegroCache started out based on BDB, and ended up not using it. The Franz guys said performance wasn't up to par, and I always wished for more details.
http://scalien.com