Postgres RDS is indeed different, yes it runs on EBS, but the Aurora in a sense isn't truly Postgres. What they've done is forked away from vanilla Postgres and tweaked the storage layer to work on the block based storage they've built just for it. It's why you see it advertised at "Postgres compatible" vs. just being will standard Postgres.
This article is definitely an interesting one as it highlights some of those trade-offs that they now need to account for.
Agreed, I haven't had the opportunity to experiment with Aurora yet, but I knew there would be issues. It's nice to get a field report, now I have a better idea what to expect.
The key difference between "classic" RDS and Aurora is that classic RDS really only automated the control plane. That is, RDS spins up an EC2 instance (or two, for multi-AZ) on your behalf, attaches an EBS volume of the appropriate specs, installs Postgres, sets up security and backups and replication etc.
Under classic RDS, when your application makes a SQL connection (the data plane) it's talking to a more or less stock Postgres instance, the same as you would have if you ran it locally.
Aurora, on the other hand, is involved in both the control plane and data plane. Your SQL connection is to a Postgres instance that's been forked/modified to work within Aurora.
"Aurora Postgres doesn't have a filesystem for its tablespace. It has a block store. Instead, Aurora Postgres instances that are doing large sorts or indexing large files use local storage, which is currently 2x the size of memory. That is, if an Aurora database instance has 72gb of memory, you only have 144gb of temporary space. Good luck sorting that 150gb table."
That seems like a problem that's easy to overcome. Allow people to configure the local storage size and you have a reasonably fair solution.
Still has its limitations for really large datasets, but that's also one you have with vanilla postgres.
As mentioned in the article, the ability seamless failover to replica significantly reduces MTTR when the primary/master node fails. Also, Performance Insights (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...) is really helpful chasing down slow/resource intensive queries.
That said, building a large, scalable Database still requires understanding the workloads and testing them out on production sized datasets. No easy button yet!
I'm not clear what you are trying to suggest with that link. The whole point of the claim in the article around sorting is that in Aurora, for using disk to sort, postgres only has access to a relatively small filesystem that runs out of space and can't use the Aurora storage that is actually backing the database.
> What's the point of Aurora? Aurora does have a couple of positives. You can create additional read replicas virtually instantly, since they're just pointed at the same shared block storage.
If I understand this correctly, they're using the same block store backing the primary database for the replicas. If that's the case, then wouldn't an issue with the blocking store hose both simultaneously? Or are they just jump starting the replicas copy of the data from the primary's block store until it's fully replicated?
> And from a management point of view, Aurora makes the database administrator's job far simpler since you no longer have to closely monitor your tablespaces and expand your block storage as needed (and reallocate tables and indexes across multiple tablespaces using pg_repack) in order to handle growing your dataset.
This is indeed very cool. Scaling from 1 MB, to 1 GB, to 1 TB, and beyond with no manual interaction is truly amazing. The storage pricing is a net win for Aurora as you only pay for what you use at the same price as gp2 EBS volumes ($.10/GB/month) whereas the latter has to be preallocated.
The block store is related to EBS, which is what AWS's other postgres option, RDS, also uses. An EBS system failure is rare and could potentially take down both services, and EC2 as well.
"""
These are each replicated 6 ways into Protection Groups (PGs) so that each PG consists of six 10 GB segments, organized across three AZs, with two segments in each AZ. A storage volume is a concatenated set of PGs, physically implemented using a large fleet of storage nodes that are provisioned as virtual hosts with attached SSDs using Amazon Elastic Compute Cloud (EC2).
"""
The storage nodes are using local SSD instance storage, not EBS.
>I'm currently managing about 2 billion rows in Postgres.
Not sure the type of data he's storing, but 2 billion seems a bit much for 1 or a few tables. Hopefully he's thought of sharding those into multiple tables by date or key.
17 comments
[ 2.7 ms ] story [ 40.3 ms ] threadThis article is definitely an interesting one as it highlights some of those trade-offs that they now need to account for.
Under classic RDS, when your application makes a SQL connection (the data plane) it's talking to a more or less stock Postgres instance, the same as you would have if you ran it locally.
Aurora, on the other hand, is involved in both the control plane and data plane. Your SQL connection is to a Postgres instance that's been forked/modified to work within Aurora.
That seems like a problem that's easy to overcome. Allow people to configure the local storage size and you have a reasonably fair solution.
Still has its limitations for really large datasets, but that's also one you have with vanilla postgres.
That said, building a large, scalable Database still requires understanding the workloads and testing them out on production sized datasets. No easy button yet!
If I understand this correctly, they're using the same block store backing the primary database for the replicas. If that's the case, then wouldn't an issue with the blocking store hose both simultaneously? Or are they just jump starting the replicas copy of the data from the primary's block store until it's fully replicated?
> And from a management point of view, Aurora makes the database administrator's job far simpler since you no longer have to closely monitor your tablespaces and expand your block storage as needed (and reallocate tables and indexes across multiple tablespaces using pg_repack) in order to handle growing your dataset.
This is indeed very cool. Scaling from 1 MB, to 1 GB, to 1 TB, and beyond with no manual interaction is truly amazing. The storage pricing is a net win for Aurora as you only pay for what you use at the same price as gp2 EBS volumes ($.10/GB/month) whereas the latter has to be preallocated.
""" These are each replicated 6 ways into Protection Groups (PGs) so that each PG consists of six 10 GB segments, organized across three AZs, with two segments in each AZ. A storage volume is a concatenated set of PGs, physically implemented using a large fleet of storage nodes that are provisioned as virtual hosts with attached SSDs using Amazon Elastic Compute Cloud (EC2). """
The storage nodes are using local SSD instance storage, not EBS.
Not sure the type of data he's storing, but 2 billion seems a bit much for 1 or a few tables. Hopefully he's thought of sharding those into multiple tables by date or key.