Nobody wants it. It would be great if we could just run an arbitrarily large RDBMS as one node.
Sadly, hardware has limits, especially if you use cheap(er) virtual hardware. OTOH you can see Stackoverflow pulling just this trick: a single colossal SQL server handles most of the load [1]. They, of course, run two DB servers because they need a hot spare.
Yes, hardware has limits and you just cannot have a single machine with unlimited number of cores and disks.
But as the article mentioned, NewSQL is already there. DBs in this category has actually been awhile. I think Spanner's paper was 10 years ago. And it is ubiquitous across Google. So let's just accept that we should think things differently in 2022. Distributed RDBMS is already a thing, used in production, in many companies. Like TiDB(https://github.com/pingcap/tidb) mentioned in the article, Square already uses it to replace some MySQL's use cases (https://www.youtube.com/watch?v=TjqL50qzy3A).
> Yes, hardware has limits and you just cannot have a single machine with unlimited number of cores and disks.
Ok, but it still makes me wonder how far you can go today with a good old-fashioned RDBMS, running native on modern server hardware. I believe you can go a lot further than most developers seem to think, without having to go to more complex distributed setups.
In the environments I work in, it seems people often first check on scaling horizontally instead of checking how far they could scale vertically... my preference is to first scale vertically before trying to scale horizontally. (I am aware both have different advantages, but I'm mainly thinking from a performance point of view.)
Scaling horizontally or vertically, on-premise or with cloud service provider is always debatable.
IMHO for the scalability topic especially in the internet industry, China or Indian companies may have more say about it thanks to the largest population.
Searching through the mentioned TiDB, it's quite interesting to find through its affiliated site https://ossinsight.io/analyze/pingcap/tidb, that the popularity (stargazer) seems pretty high in China.
The US stars is not bad, comparable to Mongo's.
Honestly I've not heard about it much. How much of the load these databases can support up to, with a single cluster ?
Exactly, the scalability usually comes with "population", no matter it's population of China/India, or the volume of data. Actually I did a quick research on your last question and it seems that a single cluster with NewSQL architecture has no theoretical upper bound on it (and since you mentioned TiDB, I checked its document which says "unlimited"), however, in reality I believe there is such a bound due to implementation-based limitation and/or performance concern.
I don’t think sharding is a big issue for application development. It adds overhead to devops/sre teams though, especially in the case of resharding. I have seen worst issues and overhead to dev team caused by bad table partitioning schemes, not sharding.
Imo the biggest challenge of rdbms in modern saas is hybrid (oltp + olap) query pattern. Any cloud managed service that is going to nail this in a costly efficient manner will become the next mainstream thing.
11 comments
[ 5.0 ms ] story [ 36.3 ms ] threadSadly, hardware has limits, especially if you use cheap(er) virtual hardware. OTOH you can see Stackoverflow pulling just this trick: a single colossal SQL server handles most of the load [1]. They, of course, run two DB servers because they need a hot spare.
[1]: http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
But as the article mentioned, NewSQL is already there. DBs in this category has actually been awhile. I think Spanner's paper was 10 years ago. And it is ubiquitous across Google. So let's just accept that we should think things differently in 2022. Distributed RDBMS is already a thing, used in production, in many companies. Like TiDB(https://github.com/pingcap/tidb) mentioned in the article, Square already uses it to replace some MySQL's use cases (https://www.youtube.com/watch?v=TjqL50qzy3A).
Ok, but it still makes me wonder how far you can go today with a good old-fashioned RDBMS, running native on modern server hardware. I believe you can go a lot further than most developers seem to think, without having to go to more complex distributed setups.
In the environments I work in, it seems people often first check on scaling horizontally instead of checking how far they could scale vertically... my preference is to first scale vertically before trying to scale horizontally. (I am aware both have different advantages, but I'm mainly thinking from a performance point of view.)
Many services will not even approach the point where "unlimited cores and disks" becomes a concern.
> So let's just accept that we should think things differently in 2022. Distributed RDBMS is already a thing, used in production, in many companies.
I do not accept that. Why would I take a critical piece of infrastructure and layer on distributed systems problems if I don't absolutely have to?
Searching through the mentioned TiDB, it's quite interesting to find through its affiliated site https://ossinsight.io/analyze/pingcap/tidb, that the popularity (stargazer) seems pretty high in China. The US stars is not bad, comparable to Mongo's.
Honestly I've not heard about it much. How much of the load these databases can support up to, with a single cluster ?
"in this application, data increases by 8 billion records and 1.5 TB every day."
Imo the biggest challenge of rdbms in modern saas is hybrid (oltp + olap) query pattern. Any cloud managed service that is going to nail this in a costly efficient manner will become the next mainstream thing.