Pardon my ignorance but isn't dynamo just a document store itself? Most storage engines are designed and optimized for RDBs, so I'd expect inefficiency here unless there's a layer between them.
it's way better than vanilla RDS even without any clustering. it is far more stable under load / near capacity. the storage layer is amazing. people here who are underselling it relative to vanilla RDS simply lack operational experience under load and have a checklist-level understanding
+1 Aurora was the secret sauce for us to scale a db-heavy SaaS very quickly with no dedicated ops team. Got us an extra 9 in uptime, too. The cost difference from vanilla RDS was negligible for our revenue model. YMMV, of course on that.
Except... That's not always true. We were told the same, but we found the performance to be underwhelming, at best. Concurrent writes to the same row are orders of magnitude worse, certain read patterns are similarly slow, and you are severely limited in how many transactions you can have before the storage daemon gives up and you end up with double digit write latency for any write.
Yeah, for the most part it's better, but you lose the page cache buffer to stop you from running oom if you miscalculated your shared buffers, your cpu usage will go up significantly and sometimes things will be slower. While on paper the max IOPS are way higher, you do get more IOPS from migrating and then you run into other issues.
If you are or close to the 80k IOPS limit, or 65TB storage limit, there's no alternative, but the "aurora is always faster" line that AWS tries to sell you is bullshit.
Aurora Postgres is a fork of Postgres code, with modifications that allow it to leverage AWS' Aurora storage, and a few other things (replicas, etc.).
The goal is to remain 100% Postgres compatible. So the network protocol, SQL queries, and tools, all can be used without any change. There are (were?) some (read: very few) caveats/restrictions, though, because of the heritage of, and need to be compatible with, Amazon RDS Postgres.
Disclosure: Member of the founding-team of Aurora Postgres; left the team quite a while back, so my knowledge is quite outdated.
Edit: There's a lot of literature, and public information (articles, conference talks, etc.) published by the Aurora team over the years, which all allude to these facts, and you can use that material to build your confidence that it is in fact Postgres compatible.
For no particularly good reason, I was under the impression Postgres Aurora was just a Postgres compatibility interface layer on top of a common substrate shared with the MySQL Aurora implementation.
It's a long story, but not exactly. Elasticsearch was developed independently by Shay Banon in 2010 and a couple others who are the founders of Elastic (http://www.elastic.co) under the Apache2 open source license. In March of 2015, Elastic acquired Found, a company offering cloud-hosted managed Elasticsearch clusters which became their "Elastic Cloud" product. In October of 2015, AWS launched their own managed Elasticsearch service called "AWS Elasticsearch Service". Werner Vogels announced it in a now-deleted tweet saying that it was a "great partnership between @elastic and #AWS" when, in fact, there was no partnership at all (thing https://www.theregister.com/2021/01/21/aws_not_ok_says_elast...). In January of 2021, Elastic changed the license under which Elasticsearch was developed from Apache2 to the "Elastic" license which allowed essentially all the same things as Apache2 (free use, modification, etc) so long as it was not used to provide a managed offering of Elasticsearch (https://www.elastic.co/blog/licensing-change). This was done because Elastic viewed itself as developing the Elasticsearch product that AWS immediately took and used for its own "AWS Elasticsearch Service" in competition against Elastic's own Elastic Cloud service which was a significant source of revenue for Elastic. Additionally, Elastic sued AWS for trademark infringement for the use of the "Elasticsearch" trademark in the "AWS Elasticsearch Service" name. In response, AWS forked the last Apache2 release of Elasticsearch and named their fork "OpenSearch". They also changed the name of their managed offering to the "AWS OpenSearch Service".
During the same timeframe, many other companies have seen the same risk of AWS taking their open source products and providing competing managed offerings and have chosen various methods of protecting themselves from AWS competition. Many of them (including MongoDB, MariaDB, Confluent, CockroachDB, Sentry.io, Apollo, Graylog, Couchbase) have adopted licenses such as the SSPL, BSL, and even the Elastic license that prevent or strongly discourage AWS from using their products in a managed offering. Others such as Grafana Labs have partnered with AWS (under undisclosed terms) for a managed offering. It's an ongoing tension between companies that want to offer an open source product with a managed offering that monetizes it.
There's some deep dives on the aws YouTube channel that talk about their storage layer if you want some detail.
AWS happily share quite a lot more under NDA as well, if that's an option for you.
The short answer is the proprietary changes are very well done in a way that provides no meaningful vendor lock-in.
I’m really glad to see this. Historically Aurora has been VERY slow to support new versions of Postgres (despite promising this would not be the case). But lately they have really stepped up their pace to the point where I would consider migrating some databases to Aurora. I’ma big fan of the feature that allows you to clone a large database quickly (in 30 minutes maybe). It’s extremely useful for troubleshooting issues where you need production data.
My understanding is the earlier versions of postgres required more expensive patching to support their proprietary storage layer.
Some of the common hooks have since been upstreamed (whether aws gets any credit or if it was part of other work I couldn't say), but in theory aws doesn't have to do as much work to support new versions as they used to.
We have a database with a few TB of data. We can create an aurora clone in around 10 mins give or take, but what we’ve found is that this the clone loads data in lazily as needed. Meaning the first time you hit a given table in a query it can be very slow. Afterwards it works well though
Somewhat OT and forgive me if I'm missing something on your github, I don't see how to set up a cluster locally - am I thinking about Neon wrong and it only runs on one node at a time with shared storage if you want to migrate which node runs a given database at a given moment?
What about RDS Proxy? I believe it's still only supports 13 which means if you use both the Proxy and Aurora you will have to wait? RDS Proxy does still mention only support for v13[1].
RDS Proxy is very specific with which versions it supports. If it says it doesn't support it, it doesn't support it.
Postgres 14 is supposed to have built-in connection pooling. If you're on it, do you still feel the need for RDS Proxy? We're using pgbouncer and can't decide if we should switch to RDS Proxy or upgrade to Postgres 14 and drop the external connection pooler.
I work in an enterprise software ecosystem and on current version (something we call the converged platform), we are getting our SaaS customers upgraded to Aurora from previous SQL DB. It’s just what our DevOps calls a “life and shift upgrade” (whatever the heck that means coz the issues that entails it gives us a hard time)
Lift and Shift — move an application from one environment to another with minimal architecture changes. The difficulty comes when you have built-in assumptions that can’t be replicated to the new environment. Some good examples would be clusters that rely on floating ips, ARP, interface bonding, SCSI, or multicast.
Over the years cloud providers have added these features but they weren’t always available and definitely not something you would reach for in a cloud native setup.
Ignoring the need for a Synchronous Replica, you may see Aurora Postgres extremely costly and not too great in performance compared to PostgreSQL. This has been neatly documented in this Article
34 comments
[ 2.8 ms ] story [ 24.0 ms ] threadBut for many other database such as "Amazon DocumentDB" , "Time Series" humour suggest it's Dynamo.
We use Aurora for all apps. It costs more, but needs little to no maintenance from our side
Yeah, for the most part it's better, but you lose the page cache buffer to stop you from running oom if you miscalculated your shared buffers, your cpu usage will go up significantly and sometimes things will be slower. While on paper the max IOPS are way higher, you do get more IOPS from migrating and then you run into other issues.
If you are or close to the 80k IOPS limit, or 65TB storage limit, there's no alternative, but the "aurora is always faster" line that AWS tries to sell you is bullshit.
The goal is to remain 100% Postgres compatible. So the network protocol, SQL queries, and tools, all can be used without any change. There are (were?) some (read: very few) caveats/restrictions, though, because of the heritage of, and need to be compatible with, Amazon RDS Postgres.
Disclosure: Member of the founding-team of Aurora Postgres; left the team quite a while back, so my knowledge is quite outdated.
Edit: There's a lot of literature, and public information (articles, conference talks, etc.) published by the Aurora team over the years, which all allude to these facts, and you can use that material to build your confidence that it is in fact Postgres compatible.
For no particularly good reason, I was under the impression Postgres Aurora was just a Postgres compatibility interface layer on top of a common substrate shared with the MySQL Aurora implementation.
During the same timeframe, many other companies have seen the same risk of AWS taking their open source products and providing competing managed offerings and have chosen various methods of protecting themselves from AWS competition. Many of them (including MongoDB, MariaDB, Confluent, CockroachDB, Sentry.io, Apollo, Graylog, Couchbase) have adopted licenses such as the SSPL, BSL, and even the Elastic license that prevent or strongly discourage AWS from using their products in a managed offering. Others such as Grafana Labs have partnered with AWS (under undisclosed terms) for a managed offering. It's an ongoing tension between companies that want to offer an open source product with a managed offering that monetizes it.
https://cloud.google.com/blog/products/databases/introducing...
(I’m ceo of neon)
[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
Postgres 14 is supposed to have built-in connection pooling. If you're on it, do you still feel the need for RDS Proxy? We're using pgbouncer and can't decide if we should switch to RDS Proxy or upgrade to Postgres 14 and drop the external connection pooler.
Any references on this? I see a note somewhere about better connection handling, but nothing that would remove the need for connection pooling.
This is false. No PostgreSQL version has had a built-in connection pooler. There was a patch some years ago[0], but it didn't make it into PostgreSQL.
[0] https://www.postgresql.org/message-id/flat/ac873432-31cf-d5e...
Over the years cloud providers have added these features but they weren’t always available and definitely not something you would reach for in a cloud native setup.
https://www.migops.com/blog/is-aurora-postgresql-really-fast...