I'm looking pretty seriously at MongoDB and I've heard that Cassandra is worth considering. I do a lot of data warehousing / statistical analytics which generally means some sort of star schema-based reporting with lots of crosstabs, dimensions, etc.
If anyone can relate their experience with either of these two platforms, would either be a good choice for live querying for these types of applications? I know you can use MapReduce to eventually get the data you need, but I need to support queries that respond in (well) less than a second, even for very large data sets.
Like any other data store, MongoDB will do what you need in less than a second... * if the database fits in memory *. Once it's on disk, it doesn't matter as much how little overhead the storage engine has, it's going to be slow.
1) What kind of API do you need? It'll have to be simpler than SQL, likely key-value with secondary indices managed at either the API layer or application layer.
2) Which 2 of Consistency, Availability and Partition Tolerance do you need? Is "gets there eventually" enough or do you need "can read it right after I write it" guarantees?
For someone who knows nothing about NoSQL and decent with MySQL, can someone give a brief Idiot's overview of how NoSQL works? If I add a record in say a table named "news", where is the data stored? If I need to do a search by news id or description, what's the front-end api like and what happens at the backend when the api is called?
There isn't an answer that fits all of the different systems called 'NoSQL'. While you can count on any common RDBMS to be accesible through SQL, there is not a 'NoSQL' language or specification. In general, access is somewhat like using an ORM.
The data is stored in files of a special format, just like an SQL database.
I'm new to this myself but I'll attempt an explanation. Then someone can correct me if needed :)
The basic idea is that data is stored as key > value. You can sort of relate it to the MySQL practice of denormalization and storing a bunch of data from different tables as an array in one cell.
You don't want to (or can't) do joins because data is not normalized, but instead have to process the results you would want separately in a batch process called map/reduce.
It all boils down to this one fact: NoSQL is write heavy in order to improve read performance. Because your data is in multiple places and you are doing batch processing to get the right format/calculated data then you are saving time when the user requests it from you. This is one of the reasons its good for scalability. The other reason seems to be that denormalized data is easier to scale out then normalized.
PS: Why aren't we in the habit of citing papers like this? So many links are to websites rather than direct sources, and the info in papers is usually highly readable and extremely informative about the details.
I agree so much! Most blogs fail to sumarize such papers properly. And if I want a summary or introduction, I can get find those in the papers, too.
The same with RFCs and W3C documents. All that stuff is written very well, from the technical view as well as regarding their readability. Why putting a layer of blogs and websites around them, which actually lower the perceived quality?
Also, I'm frequently annoyed by articles that write about what some other people (e.g. RMS) wrote. These aren't much shorter than the source article, and aren't written nearly as clear as the original.
I appeciate it when an author just links to the source and elaborates on that topic, stating his own opinion, and perfers to quote rather than to paraphrase what's already clearly written in the source article.
I'd love to see an example of how people are redefining their schema with NoSQL databases (especially with document based databases like CouchDB). A common example you hear is "Your blog document can contain an array of comment nodes". Which is all great in theory but obviously won't scale.
If Digg are using Cassandra as a big key-value store then how do they look up comments for posts? If they're storing one entry per comment with an index on post ID then it's not really key based any more.
We hear a lot of case studies about performance increases, but I never seem to see any practical details on how the databases should be used correctly.
Comments are denormalized and stored two ways: One in a plain ColumnFamily (for random access) and one in a SuperCF (for sequential access).
The plain CF ("Comments") uses the comment ID (which is a timestamp+salt) as the row key. The fields of the comment are columns in the row (username, text, date_created etc).
The SuperCF ("StoryComments") uses the story ID as the row key. Each row contains one SuperColumn per comment. The SC name is the comment_id, and the columns are the fields of the comment.
So, say you want to get the first 50 comments for a given story, you'd do a get_slice on StoryComments, passing the story ID in as the row key, "" as the start column, and a count of 50. You get back 50 SuperColumns, each of which contains one comment.
Cassandra is extremely good for sequential reads, not just random lookups. Just about any list of things can be efficiently stored and retrieved with its batch insertion and slicing operations.
26 comments
[ 3.6 ms ] story [ 27.8 ms ] threadIf anyone can relate their experience with either of these two platforms, would either be a good choice for live querying for these types of applications? I know you can use MapReduce to eventually get the data you need, but I need to support queries that respond in (well) less than a second, even for very large data sets.
1) What kind of API do you need? It'll have to be simpler than SQL, likely key-value with secondary indices managed at either the API layer or application layer.
2) Which 2 of Consistency, Availability and Partition Tolerance do you need? Is "gets there eventually" enough or do you need "can read it right after I write it" guarantees?
Cassandra does ad-hoc queries quite well if you're willing to use mapreduce.
If you have a smaller number of frequently done queries and are willing to use MR for the others maybe that is a happy place for you.
http://try.mongodb.org/
http://try.redis-db.com/
The above site has been discussed today on HN: http://news.ycombinator.com/item?id=1181714
also no copy paste in the browser version AFAICT
The data is stored in files of a special format, just like an SQL database.
The basic idea is that data is stored as key > value. You can sort of relate it to the MySQL practice of denormalization and storing a bunch of data from different tables as an array in one cell.
You don't want to (or can't) do joins because data is not normalized, but instead have to process the results you would want separately in a batch process called map/reduce.
It all boils down to this one fact: NoSQL is write heavy in order to improve read performance. Because your data is in multiple places and you are doing batch processing to get the right format/calculated data then you are saving time when the user requests it from you. This is one of the reasons its good for scalability. The other reason seems to be that denormalized data is easier to scale out then normalized.
You should also look up CAP theorem and NoSQL.
A few links that I've been saving to teach myself: - http://stackoverflow.com/questions/1189911/non-relational-da... - http://stackoverflow.com/questions/2170152/nosql-best-practi... - http://www.linux-mag.com/cache/7579/1.html - http://blogs.computerworld.com/15510/the_end_of_sql_and_rela...
Overview: http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosyste...
Think BASE (Basically Available, Soft state, Eventual consistency) instead of ACID (Atomicity, Consistency, Isolation, Durability).
(I believe this is where the term originated)
PS: Why aren't we in the habit of citing papers like this? So many links are to websites rather than direct sources, and the info in papers is usually highly readable and extremely informative about the details.
The same with RFCs and W3C documents. All that stuff is written very well, from the technical view as well as regarding their readability. Why putting a layer of blogs and websites around them, which actually lower the perceived quality?
Also, I'm frequently annoyed by articles that write about what some other people (e.g. RMS) wrote. These aren't much shorter than the source article, and aren't written nearly as clear as the original.
I appeciate it when an author just links to the source and elaborates on that topic, stating his own opinion, and perfers to quote rather than to paraphrase what's already clearly written in the source article.
But yeah, Dynamo is the most important of those.
If Digg are using Cassandra as a big key-value store then how do they look up comments for posts? If they're storing one entry per comment with an index on post ID then it's not really key based any more.
We hear a lot of case studies about performance increases, but I never seem to see any practical details on how the databases should be used correctly.
The plain CF ("Comments") uses the comment ID (which is a timestamp+salt) as the row key. The fields of the comment are columns in the row (username, text, date_created etc).
The SuperCF ("StoryComments") uses the story ID as the row key. Each row contains one SuperColumn per comment. The SC name is the comment_id, and the columns are the fields of the comment.
So, say you want to get the first 50 comments for a given story, you'd do a get_slice on StoryComments, passing the story ID in as the row key, "" as the start column, and a count of 50. You get back 50 SuperColumns, each of which contains one comment.
Cassandra is extremely good for sequential reads, not just random lookups. Just about any list of things can be efficiently stored and retrieved with its batch insertion and slicing operations.