I see no mention of foreign keys, or any other constraints, across shards. If, as I suspect, they're not implemented, it would still be useful but at the level of Spanner 10 years ago.
I love to see this but as a heavy Vitess/MySQL user I do fear the split focus from PlanetScale. Hoping to see continued improvements on the Vitess side as well.
Selfish doubts aside, congrats to Planetscale on the launch!
don't worry, we still give a lot of love to vitess. at the end of the day they are both databases. the beauty of doing both is we can take learnings from each product and apply it to the other.
The largest database to ever run on Aurora MySQL runs on PlanetScale, and the largest database to ever run on Aurora limitless also runs on PlanetScale.
The top concern I've gotten from dev teams when proposing HA distributed postgres (e.g through RDS Aurora global) is that eventual consistency is not suitable for many workloads.
Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.
Assuming it works the same way as their MySQL product, Vitess:
The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be.
I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)
Same here, first thing I need to know when considering a distributed system is how consistency is handled. If it's eventual consistency, what is the replication lag like? If it's strong consistency, can their network handle that? What happens when a node goes down?
The page matches their name, it's extreme hype. I finally figured out what I'm going to call them since their name doesn't sit well with me: PlummetScale.
> This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck.
Apart from being able to use any node as a coordinator (and you can load balance them to avoid having "multiple connection strings), there's a new pattern which effectively allows you to have as many coordinators as you want. They are effectively "data-less" nodes. We have devised and implemented this pattern in StackGres [1].
> Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus.
Adding nodes (infrastructure) is what operators solve. In StackGres, adding new nodes means editing one/two characters from your YAML file: the integer number that represents the number of workers that you have.
Shard rebalancing is fully built-into Citus as a UDF, which you can call (manually or in an automated manner) over Postgres protocol.
> Managing backups is also external to Citus, so operators still need to build the proper infrastructure
Agreed, but it's also solved (see distributed backups in StackGres [2]).
> PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially.
Unsubstantiated why. I assume it's because of the assumption that a proxy model is superior than Citus. To which I have to say that Citus model is also a proxy model, where the proxy just happens to be Postgres, which unsurprisingly, speaks Postgres protocol. Sure, there are nuances that we could debate in this area and we can say that Citus is not a "pure proxy", but that doesn't lead to concluding that a proxy model is better --it's arguably not.
29 comments
[ 0.23 ms ] story [ 3.0 ms ] threadThe pitch is compelling. I wonder how many teams will be able to operate sharded database setups in production as a result of this.
Selfish doubts aside, congrats to Planetscale on the launch!
Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.
The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be.
I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)
> This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck.
Apart from being able to use any node as a coordinator (and you can load balance them to avoid having "multiple connection strings), there's a new pattern which effectively allows you to have as many coordinators as you want. They are effectively "data-less" nodes. We have devised and implemented this pattern in StackGres [1].
> Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus.
Adding nodes (infrastructure) is what operators solve. In StackGres, adding new nodes means editing one/two characters from your YAML file: the integer number that represents the number of workers that you have.
Shard rebalancing is fully built-into Citus as a UDF, which you can call (manually or in an automated manner) over Postgres protocol.
> Managing backups is also external to Citus, so operators still need to build the proper infrastructure
Agreed, but it's also solved (see distributed backups in StackGres [2]).
> PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially.
Unsubstantiated why. I assume it's because of the assumption that a proxy model is superior than Citus. To which I have to say that Citus model is also a proxy model, where the proxy just happens to be Postgres, which unsurprisingly, speaks Postgres protocol. Sure, there are nuances that we could debate in this area and we can say that Citus is not a "pure proxy", but that doesn't lead to concluding that a proxy model is better --it's arguably not.
[1]: https://stackgres.io/doc/latest/administration/sharded-clust...
[2]: https://stackgres.io/doc/latest/administration/sharded-clust...
I am looking for interdimensional and interuniversal scale. Which of you trust fund babies has a startup which is working on this problem?