Not trying to take anything away from Cassandra (or any of the other products mentioned), but I would have liked to see the article focus on the actual data presented rather than a somewhat speculative discussion about 2 products that were not evaluated in the referenced study.
Aside from MongoDB not being a part of the subject study, it's also worth mentioning that they cherry picked the example that made Cassandra look particularly good. MySQL actually did extremely well on the non-scan tests, while offering consistency. It depends upon your usage.
When benchmarking HyperDex we found something similar. Cassandra is better at writes than reads by a surprising amount. Given Facebook's elaborate caching architecture (which likely pushed for such an inversion) this makes sense.
> A log-structured engine that avoids overwrites to turn updates into sequential i/o is essential both on hard disks (HDD) and solid-state disks (SSD). On HDD, because the seek penalty is so high; on SSD, to avoid write amplification and disk failure. This is why you see mongodb performance go through the floor as the dataset size exceeds RAM.
The structure of MongoDB's on-disk data has nothing to do with why its performance starts to falter when the dataset size exceeds RAM. It falters because each node mmap(2)s its dataset into MongoDB's process space and relies solely on the kernel's buffer caching algorithm to determine which pages to cache. The buffer cache is general-purpose, shared with every other process running on the node, and isn't finely tuned (or tuneable, for that matter) for database workloads, in which a basic LRU would be too naive. This is why MySQL, for example, doesn't mmap its tablespaces - instead, it's typically configured to manage its own buffer pool, and to avoid double buffering, O_DIRECT semantics are used for disk I/O.
There is a secondary problem with using mmap. You don't know if an access will take a page fault. When thrashing starts happening MongoDB doesn't throttle new queries. New queries coming in then add fuel to the fire making existing queries take longer and longer. This causes a huge and rapid performance dropoff.
Of course not all new queries will cause paging so they could be left unthrottled. There is a system call mincore that will tell you if pages will take faults but it doesn't support scatter/gather and has race conditions especially when there is lots of paging!
I'm not sure what's the point comparing benchmarks this way.
Choosing a database is not only about performance, it's about the type of application you are building, the stage it's in (prototype product doesn't have the same need as a product that has grown over 5 years).
It's also about the people that works on the project. Some projects are better handled in a specific language (ruby/java/php,asp.net, etc.)
For example, using MongoDB on a ruby stack to build a prototype is a pretty good choice. Moving some loads off mongo to redis would be a solution later on. And eventually, the need would arise to migrate your mongoDB stack to Cassandra.
I'm not sure switching to a document model with multiple indexes (MongoDB) to a key/value store (Redis) is something that can be done easily "later on".
Well structure evolves over time. Probably the structure would move and you would use redis as a memcache layer (to update counts, notifications, etc). What I'm saying is that databases needs evolves over time.
Comparing in-memory storage with SQL and NoSQL isn't a useful and misses the point.
This is terrible advice on how to build an application.
As everyone who has worked on a large enterprise type project before knows, the decisions you make at the start live on. It is very rare to completely switch major parts of your architecture especially today where the database you choose will affect your entire architecture.
Cassandra like Riak is multi master which means your deployment strategy would be very different to MongoDB which is still effectively master/slave.
Has anybody found / done a benchmark that would compare the scalability of a Cassandra cluster vs an ElasticSearch cluster the latter used as a NoSQL database (with stored fields)?
I am interested with 2 kinds of scalability:
- volume scalability with single concurrent user: average read / write query times vs stored-data and indexes size vs number of EC2 nodes
- concurrency scalability with a fixed size database: average read / write query times vs number of concurrent users vs number of EC2 nodes
Cassandra is ugly, hardcore and performant as hell. It's not meant for the casual user, it's really meant to be there for you at scales where MongoDB craps its pants. If you wrap your head around ColumnFamilies, tunable consistency and NetworkTopologySnitch strategies, you get rewarded by a database that can scale on a global level to millions of I/O operations per second. We at Trademob have chosen Cassy as the backbone of our tracking platform and couldn't be happier. It's pretty serious stuff though and nothing for a quick prototype or the first few iterations of a product IMHO.
Cassandra with PlayORM/Astyanax has been the easiest database for me to install, use and manage out of the 10+ I've tried. Far simpler to install/manage than MySQL Cluster or Riak, far easier to use than PostgreSQL and infinitely better to scale than MongoDB.
You don't have to understand ColumnFamilies, consistency or the different topology strategies. The defaults are fine and if you are a Java developer life couldn't possibly be simpler.
You would need to define "consistency" in order to have a more reasonable discussion about what each system provides, but Cassandra certainly isn't only eventually consistent. You can choose the number of replicas that must respond in order to consider the read/write operation a success per operation, which allows you to have quorum-based strong consistency guarantees.
Consistency is a safety property. HyperDex, Hbase and BigTable all provide linearizability, which has a well-defined meaning. Cassandra does not, and most of its descriptions of consistency only refer to the behavior of the system, and not the properties you can rely upon. Pointing to the number of replicas read or written only clouds the issue.
I think if you will write to Cassandra with consistency level ALL, you will get strong consistency. Or you can use write consistency = 1 and read = ALL, or write consistency = Quorum and read consistency = Quorum.
> You don't have to understand ColumnFamilies, consistency or the different topology strategies. The defaults are fine
Whaa?? Fine for what?
I agree with the OP. I used cassandra at a startup for a year, about a year ago, and haven't used it since last April. I'm sure things have progressed some (and we were using SuperColumns which a new user today might not) but the idea that it's a black box you don't need to understand? Not at the load we were putting it under.
But if you're a Java developer then PlayORM/Astyanax abstracts away almost all of the complexity and allow you to focus on your data model and not worry about the columns. Likewise for the preferences the defaults make sense and are pretty straightforward to change if you do need.
This is a recipe for disaster. Cassandra requires careful understanding of its claimed consistency guarantees to be used properly. Proper use of the ConsistencyLevel argument is pretty critical if you want to build a real site that actually works under load.
While Cassandra has some nice characteristics, there are a few things I've run into along the way.
Don't expect to run a 3-node Cassandra cluster and get much out of it in terms of availability, in the way you might run a master/slave failover setup. It's somewhat obvious, but your Cassandra deployment can't just start with a couple of nodes and scale up as you run into bottlenecks. The number of nodes needed starts to add up quickly with a replication factor of > 1 and quorum reads. And while you might say "I'm ok with eventual consistency, let's just read from a single node," if you're not reading from multiple nodes, the data may never become consistent, from what I can tell.
And counters should be marked with a big warning "not for production use". Their performance isn't great, and it nosedives as the dataset grows. (each counter update involves a read + a write) Having a node reboot can sometimes cause counters to double. They seem like basically an afterthought.
Your first paragraph is, bluntly, incorrect. Cassandra guarantees that data will always become consistent. This is automatic [1] for normal operation, including in the face of temporary failures. Permanent failures require running a "repair" process to rebuild the failed machine from other replicas [2].
I think you've also misunderstood how quorum works; it is a quorum of the replica count, which tends to stay constant over cluster lifetime, not machine count.
You are right that the current counters are an afterthought. I linked in my concluding paragraph, where I talk about improvements for Cassandra, "A new design for distributed counters." [3]
False -- if there are nodes being added or deleted from the system, Cassandra provides no guarantee of consistency. Two nodes might disagree on quorum membership and thus quorum accesses may fail to overlap, leading to inconsistency.
Cassandra uses a technique known as consistent hashing to allow each node to independently determine what nodes are replicas for a given row. The process really just involves hashing the row key and then comparing the result to the token of each node in the cluster. If the hash falls in between a node's token and the token of the previous node in the ring, then that node is a replica for that row. There's not really any way that nodes can disagree on this.
Given that nodes cannot disagree on the set of replicas for a row, quorums must (by definition, and the pigeon hole principal) overlap by at least one replica.
Imagine two nodes, both initiating updates to the same row. One has observed a number of node additions and deletions. The other has not. They will perform their updates on disjoint sets.
> There's not really any way that nodes can disagree on this.
Atomic, dynamic updates to a ring are a difficult thing to do. If I'm missing some additional mechanism that ensures agreement on ring membership, please explain.
Please enlighten us with a cluster database that enables guaranteed consistency with dynamic node membership. They all have quirks handling membership (unless you're looking at a shared disk setup).
Let's not revert to ad hominems so quickly. If you looked through my submissions, you'll see that I explained the problem elsewhere. Here it is again:
Imagine two nodes, both initiating updates to the same row. One has observed a number of node additions and deletions. The other has not. They will perform their updates on disjoint sets.
Additions to the ring are not atomic, therefore quorums aren't.
What is memory overhead? How much memory Java processes consume comparing to the amount of data a node could handle, assuming there must be no swap (otherwise we all know what happens to any Java process).
It seems that question has almost completely fallen off the radar today. I've experienced medium sized voldemort clusters eating up huge amounts of extra RAM (of the order of 100% overhead[1]) to avoid falling into pathological GC patterns over long runs. Actually I shouldn't really single out voldemort. The problem is java.
Java[2] is a terrible platform to write large in-memory caching servers on. The write and access patterns are a complete mismatch for the assumptions made in the generational GC algorithms that most current JVM's sport. Most caches will evict on an LRU basis, which means that almost all allocations will end up in the old generation heap before finally being evicted. Which is precisely the counter-optimal case for the basic assumptions that the generational GC model relies on (that most objects are short-lived and get swept while still in the "young" heap (which is ultra cheap).
Footnotes:
[1] "overhead" here means precisely how the parent post defines it.
[2] more precisely, the commonly used freely available JVM's that most shops use. There might be better GC implementations (e.g. as claimed by azul) but I don't have any direct experience with them.
We got bit by GC issues with our Cassandra cluster, and we had to completely re-design a column family to fix it.
It's pretty telling that the development community is moving as many memory structures as possible outside the java heap, each new major release has moved some piece or other.
The biggest threat I see to Cassandra is that java in the end won't cut it, that the JVM will limit its performance too much, allowing a competitor to surpass it. Stop-the-world GC pauses are not something you want in a high-performance database solution.
I was in a discussion with a member of the Go development team, bitterly complaining about their decision to go with a GC'd heap as the only way to access memory for something they intended as a "systems programming language". They suggested I link in C data structures for those heap-heavy caching applications :-( As I see it, C and C++ are the only practical options for writing high performance, memory efficient, cache heavy applications for production use in the current tech climate.
HBase looks bad in some of these benchmarks, because it is hard to setup and has many tuning knobs to be tuned correctly for the workload in question.
Due to its strictly consistent nature you have to think about key design, hotspotting of servers, etc, etc. In return you get correct atomic operations, row transactions, range scans by default (Cassandra uses a random partitioner by default not allowing range scans), etc, etc.
Some of the largest installations on this planet run on HBase. For example, FaceBooks HBase stats at HBaseCon (May 2012): Billions of msgs/day, 75Bn ops/day, 1.5M ops/sec peak. 250TB new data/mo and growing.
(Facebook also created Cassandra, but is not using it)
As usual you use the right tool for the job and isolated benchmarks usually do not bear this out.
And more usually, you'll use the tool the tool that your most familiar with or can become most familiar [quickest]. You say so yourself, HBase is relatively hard to quickly come to terms. We can observe the same when viewing the "SQL camp" with MySQL and PostgreSQL (and MSSQL).
It is also true of the Facebook HBase install! Completely unnecessary jab; the use of HBase was politically driven -- what the architects wanted to use. What are the proofs that it was the "right tool?" Well, it works, but you won't find it behind Google's Gmail / Talk, so who knows!?
First, kindly point at an Cassandra installation, which supports the size that HBase supports in this setting.
Can I assume you have first hand information about this decision from someone at Facebook? I was stating a fact... not a jab.
Re: Google. Nothing at Google is driven by an eventually consistent store. They got that part right from beginning. It's too hard to manage from the an application point of view (unless you store immutable data, in which eventual consistency is pretty awesome).
Checkout BigTable, Megastore, Percolator, Spanner. Except for the latter these are all based on BigTable. Note that BigTable is always consistent like HBase (and unlike Cassandra).
Are you saying the MySQL is better than PostgreSQL (or vice versa?) because one is easier to setup than the other. That is a terrible way to pick a technology.
Out of the box functioning is nice, easy to get started, makes the product look good in benchmarks, etc, but it just hides the complexity until you deploy into a production setting.
Lastly, I agree that if you store the odd few TB here or there, you certainly do not need to bother with the complexity of HBase.
Re: Facebook, as jbellis mentions in this comment thread, Facebook shards HBase, so it's not able to support what they need without some major additional complexity.
Re: eventual consistency. It's tunable to the need of the user. It's a trade-off. Tunable consistency becomes nice when you want a system to be always available. See the Amazon Dynamo paper, on which Cassandra is partially based.
I think many underestimate how many are using Cassandra in production. Spotify, eBay, Walmart, Netflix, Ooyala, NASA and others - one list is found here: http://www.datastax.com/cassandrausers.
It's worth noting that the FB HBase install is also sharding across multiple sub-clusters because of the HDFS namenode SPOF problems [1].
Personally, if I'm going to shard manually I'll stick with postgresql. One of the primary reasons to use something like Cassandra is that it solves that for you.
My understanding of facebooks Pod Architecture for HBase was not the name node but simply scaling HBase, HBase gets rather unpleasant at facebook scales. The facebook HBase fork has things like compactions disabled to improve performance.
I ran a HBase cluster with 1PB storage, it became very unwieldy at this scale, thousands of regions and lots of tricks to keep it happy. As for SPOF, the name node now has HA and it works very well.
HBase 90.4, problems with I/O we had a very heavy read load on top of a write load, write load bursting 14,000 TX per second, and an average of 8,000 per second - each record around 2k.
Because of the I/O the WAL had to be turned off, this introduced problems when Region Servers occasionally died. Implementation of large regions 10GB, and fairly large HBlocks 512MB, increasing flush sizes to reduce minor compactions. Use of MSLAB to virtually eliminate GC all together, use of large heap 12GB on RS.
Worst problems we experienced was META corruption, that really , really sucked.
Thanks. If there's a more detailed writeup you can point me to that'd be great. I would like to make sure then that all these issues are addressed in the current versions.
0.94+ has MSLAB by default, with HFileV2 (0.92+) we can support much larger regions (20G or bigger).
Curious about the 512M HBlocks, did you have scan-heavy read-load?
14k TX peak per regionserver? x 2k that's 28M/s (56M with WAL). Should be doable now even with WAL (definitely with deferred flush). Well, maybe not with concurrent very heavy read load, depending on disk configuration.
Probably on top of Hadoop 0.20-append?
Hadoop-2.x.x should be far better too.
Also they used an old version of HBase, 90.4 in the benchmark, HBase 94.0 has been out since May 2012, and has a lot of work to improve performance especially remove lock contention on the region server.
The big caveat in their usage of VoltDB is that they are apparently using a synchronous client, waiting for a response each time, instead of async streaming. They mention this briefly in passing at the bottom of the paper, and say the VoltDB people were able to see performance increase by using an async client.
I don't know if this is heretical to say, but when I think about services that people should look up to in terms of architecture, I don't think of Netflix.
See all the downtime they have despite the 1000 posts on their blog about how wonderfully available their architecture is.
I can point to 10 other sites running on a boring LAMP stack with similar availability.
You joined here two days ago. Please provide the list of these ten LAMP sites with similar scale and availability. Otherwise you just look like a troll account.
And if you examine the causes of Netflix' downtime, is it because of their usage of Cassandra? Would a LAMP service running on the same AWS ELB nodes have avoided said downtime?
As far as I know they still store all critical data in Oracle.
(That is actually true for many user for NoSQL data bases, not just Cassandra or Netflix).
My information is a bit dated, so that might no longer be the case.
Some of these are a bit dated, but it's still a reasonably accurate list of companies that have publicly disclosed their usage of Cassandra: http://www.datastax.com/cassandrausers.
Well it's a cassandra company so it's bound to exaggerate the throughput test that shows Cassandra winning but relegating its huge weakness in latency performance (Whoa!! 10ms average read latency from in-memory store...) to an "area for improvement" list at the bottom is a bit disingenuous. It's not a small issue. It's an order of magnitude difference from Voldemort, redis and even mysql at scale.
75 comments
[ 4.3 ms ] story [ 157 ms ] threadUnless I'm missing something.
The structure of MongoDB's on-disk data has nothing to do with why its performance starts to falter when the dataset size exceeds RAM. It falters because each node mmap(2)s its dataset into MongoDB's process space and relies solely on the kernel's buffer caching algorithm to determine which pages to cache. The buffer cache is general-purpose, shared with every other process running on the node, and isn't finely tuned (or tuneable, for that matter) for database workloads, in which a basic LRU would be too naive. This is why MySQL, for example, doesn't mmap its tablespaces - instead, it's typically configured to manage its own buffer pool, and to avoid double buffering, O_DIRECT semantics are used for disk I/O.
Of course not all new queries will cause paging so they could be left unthrottled. There is a system call mincore that will tell you if pages will take faults but it doesn't support scatter/gather and has race conditions especially when there is lots of paging!
I did report this at the beginning of 2010 - currently marked as major priority, planned but not scheduled: https://jira.mongodb.org/browse/SERVER-574
That said MongoDB is still my first database of choice. Nothing beats arbitrary JSON in, the same JSON back out.
Choosing a database is not only about performance, it's about the type of application you are building, the stage it's in (prototype product doesn't have the same need as a product that has grown over 5 years).
It's also about the people that works on the project. Some projects are better handled in a specific language (ruby/java/php,asp.net, etc.)
For example, using MongoDB on a ruby stack to build a prototype is a pretty good choice. Moving some loads off mongo to redis would be a solution later on. And eventually, the need would arise to migrate your mongoDB stack to Cassandra.
Comparing in-memory storage with SQL and NoSQL isn't a useful and misses the point.
As everyone who has worked on a large enterprise type project before knows, the decisions you make at the start live on. It is very rare to completely switch major parts of your architecture especially today where the database you choose will affect your entire architecture.
Cassandra like Riak is multi master which means your deployment strategy would be very different to MongoDB which is still effectively master/slave.
While you may disagree with this, there are many living examples.
I am interested with 2 kinds of scalability:
- volume scalability with single concurrent user: average read / write query times vs stored-data and indexes size vs number of EC2 nodes - concurrency scalability with a fixed size database: average read / write query times vs number of concurrent users vs number of EC2 nodes
Cassandra with PlayORM/Astyanax has been the easiest database for me to install, use and manage out of the 10+ I've tried. Far simpler to install/manage than MySQL Cluster or Riak, far easier to use than PostgreSQL and infinitely better to scale than MongoDB.
You don't have to understand ColumnFamilies, consistency or the different topology strategies. The defaults are fine and if you are a Java developer life couldn't possibly be simpler.
It's not a trivial topic and unfortunately "it appears to work as you'd expect" on a small dev cluster which can lead to statements like yours.
Your parent's post is actually very very accurate.
They ALL have issues and eventual consistency is a fundamental part of a distributed database so its something you have to learn either way.
There are more details on the options here: http://www.datastax.com/docs/1.2/dml/data_consistency
Whaa?? Fine for what?
I agree with the OP. I used cassandra at a startup for a year, about a year ago, and haven't used it since last April. I'm sure things have progressed some (and we were using SuperColumns which a new user today might not) but the idea that it's a black box you don't need to understand? Not at the load we were putting it under.
But if you're a Java developer then PlayORM/Astyanax abstracts away almost all of the complexity and allow you to focus on your data model and not worry about the columns. Likewise for the preferences the defaults make sense and are pretty straightforward to change if you do need.
This is a recipe for disaster. Cassandra requires careful understanding of its claimed consistency guarantees to be used properly. Proper use of the ConsistencyLevel argument is pretty critical if you want to build a real site that actually works under load.
[1] https://news.ycombinator.com/item?id=5076130
Don't expect to run a 3-node Cassandra cluster and get much out of it in terms of availability, in the way you might run a master/slave failover setup. It's somewhat obvious, but your Cassandra deployment can't just start with a couple of nodes and scale up as you run into bottlenecks. The number of nodes needed starts to add up quickly with a replication factor of > 1 and quorum reads. And while you might say "I'm ok with eventual consistency, let's just read from a single node," if you're not reading from multiple nodes, the data may never become consistent, from what I can tell.
And counters should be marked with a big warning "not for production use". Their performance isn't great, and it nosedives as the dataset grows. (each counter update involves a read + a write) Having a node reboot can sometimes cause counters to double. They seem like basically an afterthought.
Your first paragraph is, bluntly, incorrect. Cassandra guarantees that data will always become consistent. This is automatic [1] for normal operation, including in the face of temporary failures. Permanent failures require running a "repair" process to rebuild the failed machine from other replicas [2].
I think you've also misunderstood how quorum works; it is a quorum of the replica count, which tends to stay constant over cluster lifetime, not machine count.
You are right that the current counters are an afterthought. I linked in my concluding paragraph, where I talk about improvements for Cassandra, "A new design for distributed counters." [3]
[1] http://www.datastax.com/dev/blog/modern-hinted-handoff [2] http://www.datastax.com/docs/1.2/operations/node_repair [3] https://issues.apache.org/jira/browse/CASSANDRA-4775
The consistency claims are overblown.
Given that nodes cannot disagree on the set of replicas for a row, quorums must (by definition, and the pigeon hole principal) overlap by at least one replica.
> There's not really any way that nodes can disagree on this.
Atomic, dynamic updates to a ring are a difficult thing to do. If I'm missing some additional mechanism that ensures agreement on ring membership, please explain.
HBase for example is always consistent, but if you lose a node unexpectedly some data is not available for some time.
Have you actually experienced this issue or just making it up ?
Imagine two nodes, both initiating updates to the same row. One has observed a number of node additions and deletions. The other has not. They will perform their updates on disjoint sets.
Additions to the ring are not atomic, therefore quorums aren't.
Java[2] is a terrible platform to write large in-memory caching servers on. The write and access patterns are a complete mismatch for the assumptions made in the generational GC algorithms that most current JVM's sport. Most caches will evict on an LRU basis, which means that almost all allocations will end up in the old generation heap before finally being evicted. Which is precisely the counter-optimal case for the basic assumptions that the generational GC model relies on (that most objects are short-lived and get swept while still in the "young" heap (which is ultra cheap).
Footnotes: [1] "overhead" here means precisely how the parent post defines it. [2] more precisely, the commonly used freely available JVM's that most shops use. There might be better GC implementations (e.g. as claimed by azul) but I don't have any direct experience with them.
It's pretty telling that the development community is moving as many memory structures as possible outside the java heap, each new major release has moved some piece or other.
The biggest threat I see to Cassandra is that java in the end won't cut it, that the JVM will limit its performance too much, allowing a competitor to surpass it. Stop-the-world GC pauses are not something you want in a high-performance database solution.
Due to its strictly consistent nature you have to think about key design, hotspotting of servers, etc, etc. In return you get correct atomic operations, row transactions, range scans by default (Cassandra uses a random partitioner by default not allowing range scans), etc, etc.
Some of the largest installations on this planet run on HBase. For example, FaceBooks HBase stats at HBaseCon (May 2012): Billions of msgs/day, 75Bn ops/day, 1.5M ops/sec peak. 250TB new data/mo and growing. (Facebook also created Cassandra, but is not using it)
As usual you use the right tool for the job and isolated benchmarks usually do not bear this out.
It is also true of the Facebook HBase install! Completely unnecessary jab; the use of HBase was politically driven -- what the architects wanted to use. What are the proofs that it was the "right tool?" Well, it works, but you won't find it behind Google's Gmail / Talk, so who knows!?
First, kindly point at an Cassandra installation, which supports the size that HBase supports in this setting.
Can I assume you have first hand information about this decision from someone at Facebook? I was stating a fact... not a jab.
Re: Google. Nothing at Google is driven by an eventually consistent store. They got that part right from beginning. It's too hard to manage from the an application point of view (unless you store immutable data, in which eventual consistency is pretty awesome). Checkout BigTable, Megastore, Percolator, Spanner. Except for the latter these are all based on BigTable. Note that BigTable is always consistent like HBase (and unlike Cassandra).
Are you saying the MySQL is better than PostgreSQL (or vice versa?) because one is easier to setup than the other. That is a terrible way to pick a technology.
Out of the box functioning is nice, easy to get started, makes the product look good in benchmarks, etc, but it just hides the complexity until you deploy into a production setting.
Lastly, I agree that if you store the odd few TB here or there, you certainly do not need to bother with the complexity of HBase.
Re: eventual consistency. It's tunable to the need of the user. It's a trade-off. Tunable consistency becomes nice when you want a system to be always available. See the Amazon Dynamo paper, on which Cassandra is partially based.
I think many underestimate how many are using Cassandra in production. Spotify, eBay, Walmart, Netflix, Ooyala, NASA and others - one list is found here: http://www.datastax.com/cassandrausers.
Personally, if I'm going to shard manually I'll stick with postgresql. One of the primary reasons to use something like Cassandra is that it solves that for you.
[1] http://www.slideshare.net/brizzzdotcom/facebook-messages-hba...
I ran a HBase cluster with 1PB storage, it became very unwieldy at this scale, thousands of regions and lots of tricks to keep it happy. As for SPOF, the name node now has HA and it works very well.
Because of the I/O the WAL had to be turned off, this introduced problems when Region Servers occasionally died. Implementation of large regions 10GB, and fairly large HBlocks 512MB, increasing flush sizes to reduce minor compactions. Use of MSLAB to virtually eliminate GC all together, use of large heap 12GB on RS.
Worst problems we experienced was META corruption, that really , really sucked.
0.94+ has MSLAB by default, with HFileV2 (0.92+) we can support much larger regions (20G or bigger). Curious about the 512M HBlocks, did you have scan-heavy read-load?
14k TX peak per regionserver? x 2k that's 28M/s (56M with WAL). Should be doable now even with WAL (definitely with deferred flush). Well, maybe not with concurrent very heavy read load, depending on disk configuration.
Probably on top of Hadoop 0.20-append? Hadoop-2.x.x should be far better too.
[1] https://ccp.cloudera.com/display/CDH4DOC/Software+Configurat... [2] http://blog.cloudera.com/blog/2012/10/quorum-based-journalin...
I've only ever heard horror stories about big deployments, and the only posts about it come from DataStax.
See all the downtime they have despite the 1000 posts on their blog about how wonderfully available their architecture is.
I can point to 10 other sites running on a boring LAMP stack with similar availability.
My information is a bit dated, so that might no longer be the case.