yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).
That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).
Someone posted a twitch conversation yesterday about this, I poked around on the page realised there was no open source version and noped out immediately.
I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.
I believe multigress is the similarly aged open equivalent from Supabase. Haven't used it myself and don't know what the differences are in usability, but I'm a bit more interested in that since it's open.
that's over 13k queries/sec per router core, about 50% of it is spent doing syscalls, the other 50%: parsing, doing grpc, tls, go gc, resolving the shards, waiting for the responses
neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming
87.3 % served from cache. Does that mean it returned a result existing in the cache because the very same query was executed before? Probably still a relevant result, if you have to process millions of queries every second, it seems not unlikely that you will see a lot of repeated queries. But at that point you are measuring cache performance more than query performance. But unless you run some standardized query benchmark, a single queries per second number is not that informative anyway because query complexity and therefore execution time can span many others of magnitude. Looking up a name by ID and aggregating across a billion rows from seventeen tables joined together are both a single query.
In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.
> all the signals from the offline files in mere seconds
That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.
Lots of EBS volumes mounted via 25GBPs or higher network, positioned carefully onto a single rack where possible and carefully tuned network/ip/os for both clients and servers. It is not feasible in most deployments as this would not have scaled to real production loads (RAID configurations requiring redundancy alone would slow you down, not to mention costs of using that many EBS volumes on extra high network/IO/provisioned IOPS nodes). This was only meant to prove what was possible in AWS at the time.
Your statement about SSD availability and performance does not fit with my personal recollections of history. Smoking fast SSDs were completely vanilla items you could order from anyone in 2015. For example, the Intel DC P35/6/700 series. Each of these was as good as 25 HDDs for linear reads and more like 5000 HDDs for random reads. The U.2 specification for putting SSDs in boxes was formalized in 2011. Outside of the mass market, cloud builders had private SSD designs long before 2015.
Less than 5% of commercial storage capacity was SSD of any flavor in 2015 (NVMe probably less than 1%), and the price-per-GB was anywhere from 5-15x. NVMe hardware RAID didn't happen until 2017, and AWS SSD via EBS was capped at 320 MB/s and ~500 MB/s with striped EBS volumes.
So the OPs claim of reading multiple terabytes of data in seconds in AWS in 2015 isn't possible.
SSD simply were not as ubiquitous in 2015 as they are today.
That's a good question, I was very high up in the leadership, couple levels below the CIOs and the biggest surprise for the team was that I could write even a single line of code. In large organizations, changing direction is not always easy so I could not get them off of using a data platform, which to be honest was acceptable because we could not ask the team to write such custom code for each job going forward, I did manage to shift them towards flink which I think was better suited to our actual real-time needs.
> The benchmark was very simple. A single-shard point select, one row fetched per-query by primary key. No writes, joins, or cross-shard queries. The workload that each shard receives is isolated, in that there are no single queries that span multiple shards.
> Worth being clear about this run: the shards were primary-only with no replicas, the workload is read-only across queries ranging in complexity, and we did not fail over during the measured window.
No HA either, or backups I assume.(single AZ?)
But think the point of it was not to benchmark PostgreSQL but to demonstrate the linear scalability of the shared-nothing/stateless routing architecture. I think this was already known from Vitess though, which shares the same architecture. Also, as part of a new product release benchmarking shit like this internally is cool/interesting, why not blog about it.
The more cynical side of me says it was a money burning exercise to give the marketing team a number to tweet about. :)
This is a mighty impressive feat and congratulations to the team. I hacked around a lot with database design 10 years ago; table sharding with multiple read-replicas on mysql accompanied by application server cache layer for reads, accompanied by elasticsearch and cassandra clusters - to manage a quite large graph of billions of requests per day.
The reason because I am saying all this, is because nowadays funny enough I advise people that a single postgres box is more than enough and to never overcomplicate. Yes, hardware scaled really well in the past 10 years, but also the reality is most apps won't ever in their lifecycle have that type of traffic and usage to begin with.
I am wondering if we will be entering an internet of unhinged agentic system activity where crazy scaling becomes a thing again. Or perhaps with AI the promise of systems that can consume and find meaningful signals on such vasts amounts of data makes storing and retrieving every tiny atom of information a winning strategy (big data v2). Let's see.
I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.
It’s cool to see the flip side of that argument. Planet scale is answering the question, “what does it look like when you DO fan out your workload to >100 nodes?”
There’s a place for both technologies. Very cool stuff.
> I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.
Do you have the timestamp where they are talking about this? The claim doesn't pass the smell test for me. If you're talking about latency, then perhaps. On throughput, I don't understand how a single-node system could deliver higher throughput than a three-node system
In 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [ref]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB has now become RonDB, but is still based on a non-blocking 2-phase commit protocol and is GPL-v2.
RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.
Its great to see scale and progress, but being closed source is HUGE DEALBREAKER.
Clickhouse is also on the right track of building some amazing opensource integrations with postgres, they have superior*[1] managed postgres looks like from their recent blog. I hope they do some OSS sharded postgres solution.
Most Citus workloads were 10s of TB with largest at around a few PB or so. Heap was a couple PB, back then, if I remember correctly. It is a brilliant piece of technology that supported mission critical workloads across mid/late stage startups to huge enterprises. The planner/executor are very advanced supporting a multitude of features and decade of intricate effort.
The biggest problem of Citus was migration effort, transition from single node to multi-node was not trivial. Here I’m not talking about single table use-cases, more classic relational, multi-tenant apps with 100s to 1000s of tables. This is partly expected with most sharding technologies, though.
Sharing some insights based on my decade of experience working with Citus!
Schema changes need locking... everywhere. Citus is no different. And they need proper design everywhere. If you mean that it requires distributed transactions, well, yes, again: expected and solved. Not even all sharding solutions support this.
> coordinator node
Not sure what the problem is here. If what you mean is that a single coordinator, even with an HA replica, can saturate, that's true, but you can add multiple "query routers" (that's our name in StackGres, see [1]).
> Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.
For example, we have a customer that ingests some 30TB/day, and it's ramping up towards 200TB/day of ingestion. On 24 worker nodes.
40 comments
[ 2.5 ms ] story [ 35.9 ms ] threadPresumably it took more than one attempt, or there's some humanpower in the budget.
I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.
512 shards, each with one Postgres primary each on an r8g.16xlarge
480 Neki routers, each on its own 8xlarge instance
That's ~250K queries/sec per router which seems lowish?
neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming
That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.
So the OPs claim of reading multiple terabytes of data in seconds in AWS in 2015 isn't possible.
SSD simply were not as ubiquitous in 2015 as they are today.
I mean... What's the point of this "benchmark"?
No HA either, or backups I assume.(single AZ?)
But think the point of it was not to benchmark PostgreSQL but to demonstrate the linear scalability of the shared-nothing/stateless routing architecture. I think this was already known from Vitess though, which shares the same architecture. Also, as part of a new product release benchmarking shit like this internally is cool/interesting, why not blog about it.
The more cynical side of me says it was a money burning exercise to give the marketing team a number to tweet about. :)
Neki 1 primary + 2 replicas: ~$5.0M/month (just the AWS bill)
Google Spanner w/ 3 replicas built in: ~$3.85M/month
The reason because I am saying all this, is because nowadays funny enough I advise people that a single postgres box is more than enough and to never overcomplicate. Yes, hardware scaled really well in the past 10 years, but also the reality is most apps won't ever in their lifecycle have that type of traffic and usage to begin with.
I am wondering if we will be entering an internet of unhinged agentic system activity where crazy scaling becomes a thing again. Or perhaps with AI the promise of systems that can consume and find meaningful signals on such vasts amounts of data makes storing and retrieving every tiny atom of information a winning strategy (big data v2). Let's see.
It’s cool to see the flip side of that argument. Planet scale is answering the question, “what does it look like when you DO fan out your workload to >100 nodes?”
There’s a place for both technologies. Very cool stuff.
[1] https://youtu.be/ONxwjqFjP3A?is=awlEJwGLxQmRE25i
Do you have the timestamp where they are talking about this? The claim doesn't pass the smell test for me. If you're talking about latency, then perhaps. On throughput, I don't understand how a single-node system could deliver higher throughput than a three-node system
RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.
[ref] https://www.slideshare.net/frazerClement/200-million-qps-on-...
Clickhouse is also on the right track of building some amazing opensource integrations with postgres, they have superior*[1] managed postgres looks like from their recent blog. I hope they do some OSS sharded postgres solution.
[1] - https://clickhouse.com/blog/benchmarking-nvme-managed-postgr...
Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.
The biggest problem of Citus was migration effort, transition from single node to multi-node was not trivial. Here I’m not talking about single table use-cases, more classic relational, multi-tenant apps with 100s to 1000s of tables. This is partly expected with most sharding technologies, though.
Sharing some insights based on my decade of experience working with Citus!
Schema changes need locking... everywhere. Citus is no different. And they need proper design everywhere. If you mean that it requires distributed transactions, well, yes, again: expected and solved. Not even all sharding solutions support this.
> coordinator node
Not sure what the problem is here. If what you mean is that a single coordinator, even with an HA replica, can saturate, that's true, but you can add multiple "query routers" (that's our name in StackGres, see [1]).
> Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.
For example, we have a customer that ingests some 30TB/day, and it's ramping up towards 200TB/day of ingestion. On 24 worker nodes.
[1]: https://stackgres.io/doc/latest/administration/sharded-clust...