92 comments

[ 3.3 ms ] story [ 171 ms ] thread
So who's tested Graviton2, what were you testing it for, and what did you conclude?
We’ve been testing a few Graviton instances since before GA and have gone “all in” with our latest release targeted for EOY.

We went live with R6g (Redis) and R6gd/T4g (ES) instances at the beginning of the month. We were coming from an ancient cluster from AWS managed ES (2.3) and using their i3 elasticsearch instances so it’s been an Apples to Oranges comparison.

We are running T4g/C6g/R6g K8s clusters for web servers.

We do still have Postgres on R5 instances but I’m interested to check out R6g there since we can do a direct comparison

Regarding using c6g’s instead of i3’s to power Elasticsearch machines, could you tell me more? (We’re in the same boat, considering a switch)

Specifically around hyper threading, my understanding is c6 don’t have “vCPUs”, and just have “CPUs“, so the effective number of cores doubles. Did you find similar throughput (in terms of Elasticsearch Search/Write TPS) between a virtual and real core?

From my POV now, the i3 instances should never be used unless you absolutely need the dedicated local storage in those amounts. The per vCPU performance is horrid in comparison to any of the modern instances (R5/M5/C5) let alone comparing those vCPUs to the actual CPU of R6g/M6g/C6g like you said.

Another advantage of the Graviton processors is 50% more dedicated storage compared to equivalent Intel/AMD instances. To get enough storage we would have had to bump up to r5d/r5ad.24x or metal which when testing we also saw more “jitters” in latency on the long tail. Despite the x86 instances being larger they traded blows in different tests we had, aggregates were one thing that x86 easily beat out ARM but a lot of our aggregates come from another data source so it wasn’t a deal breaker. Overall we are happy with performance, compared to old stack we are at around 10% of the cost and I think our savings was more than 2x compared to x86 after locking in some rates. R6gd.metal (16x) vs R5d.metal (24x)

Interesting, we have an Elasticsearch cluster of around 300 i3s (and maybe soon some i3ens) I think mostly because of the NVMe storage. But yeah there's not much compute to go with all that disk space.

> aggregates were one thing that x86 easily beat out ARM

This is actually a lot (most?) of our ES workload, so that's a really interesting detail.

How are you deploying your web servers ? I assume that is where your custom application code lives.

What's the developer experience here - I have been considering cross building a docker vm for arm and deploying it. But I wasn't sure how comfortable is it.

We use python, if that matters

We have a base Debian image maintained by our Platform/Security team (compliance) that we build our Flask apps off. Other than them forgetting to push an ARM compatible image when they update we haven’t ran into issues, but I don’t know what went in behind the scenes to get the image working if anything.

I would say I’m the bridge between the infra/DevOps teams and our back end team of 12 people and as far as everyone else on the team is concerned it “just works”. Still waiting for the first big time it doesn’t

Thank you so much for replying! Could you go into some details?

So how do you bake your code in your base image ?

I'm referring to the developer experience here :

1. do you develop on x86 (mac, win, linux)?

2. Where do you build the arm docker images ? On your laptop - is it even possible.

3. Which build CI do you use to cross build arm?

1) most of the team is on MBP 16 inch models. I know there is one and may be two on 13 inch models.

2 and 3 I’ll answer to the “best of my knowledge” but on the professional level it’s something my team doesn’t handle (I do wish I knew more!). Features get merged to dev/stage/main and trigger blue-green rolling deploys based on whatever is configured. The deploy is handled through AWS CodeDeploy (not my choice, also not my teams jurisdiction) which handles ticket validation/testing/deploying to whatever k8s cluster/manual deployment rollover if needed.

I believe you can build ARM images from Docker, or at least I have a vague recollection of doing so for a Raspberry Pi, on a normal x86 machine

this is super interesting. it stands to reason that CodeDeploy would support docker crossbuild.

In my team we mandate testing docker images on local dev machines before rolling to production, so i was wondering how the cross-compiles, etc would work . But this is helpful. thanks!

We have. We are an egress firewall company and have been testing our product on Graviton2, privately, for some time now. The ARM builds will be GA in a couple of months but so far we've concluded that:

1. These offer an excellent price to performance ratio for an egress firewall like purpose because of low instance pricing and high network throughput.

2. For our stack written in C and Rust, any performance penalty isn't noticeable. It's basically a NAT gateway filtering by hostnames for TLS & SSH outbound.

Interesting point solution. Do you just license through the AWS Marketplace? Any issues with encrypted SNI?
Yes, just through AWS and GCP marketplaces.

ESNI (now Encrypted Client Hello: https://tools.ietf.org/html/draft-ietf-tls-esni-08) is still in draft.

v1 of the firewall that is currently published ignores the extension but will deny in case SNI is absent.

v2, due in a couple of weeks, will actively seek presence of any of ECH extensions and deny them. This is to prevent domain-fronting style of exfiltration attempts. We expect HTTP Clients in the enterprise to not have moved over to ECH any time soon. However, the firewall has a lot of checks and balances built-in to mitigate this in the future when it arrives.

Very cool! Thanks for the details. Totally agree on lag time for rolling out ECH.
Thank you!

Should you be interested in the v2 preview, drop us an email (from your work email) or follow our LinkedIn page.

Had a look at your website and it seems like a fantastic product.

Regarding V1 of your product that ignores the encrypted SNI (ESNI) extension, do you mean it just allows all traffic using ESNI? If so, have you considered blocking malicious domains via DNS instead?

(comment deleted)
Several defence-in-depth checks including DNS and protocol legitimacy are carried out in v1, so no it wouldn't just allow ESNI through.

Happy to invite you to the v2 preview if you drop us an email (from your work email) or follow our LinkedIn page.

It’s a fairly boring use case, but at QuasarDB we also target ARM (some of our customers run our database on embedded devices), and AWS’ graviton processors are a godsend for CI: we can just use on-demand EC2 instances as we do with all the other architectures and OS’es.
I'd really like to try this out on a postgres replica, however these instances don't appear in the web console at least when creating a replica from an existing x86 primary. Anyone have insight here?
Are you running 12.3 or higher?
Oof, no I missed that and had the version requirements for RDS IAM auth in my mind. Thanks!
Could be a region thing. us-west-1 for example is notoriously slow to get new features, including this one. us-east-1 has it.
These graviton2 instances are no joke. As soon as they released, it probably became the new lowest hanging fruit for many people (from a cost efficiency standpoint). The increased L1 cache has an incredible effect on most workloads.
ARM is smaller due to simpler instruction decoders mostly, which leaves more room for a huge cache in the transistor budget.

I am hoping Apple does this with their desktop chips: basically take the fastest phone/tablet processor, give it better thermals, and pump up the cache. That would be an easy path to beating Intel on a laptop.

My understanding was that large cache was mostly a cost tradeoff, and that there's no technical reason why x86 processors couldn't do this too. I believe Apple's A-series chips already have a huge cache which is a big part of why they are so fast.
Cost/die size is part of the consideration especially for L3 but with L1 and L2 it is much more about latency than cost: bigger caches will have higher latency than smaller ones. That's the entire reason for the multi-level cache hierarchy to exist in the first place.

Graviton2 doesn't even have a particularly large total amount of cache, the only part of the cache system that's bigger than both AMD and Intel is the L1D. L1I is the same as AMD, L2 is same as Intel, L3 is smaller than both (talking about current-gen server chips). That L1D can indeed make a big difference in certain workloads though.

The L3 is actually pretty small for how many cores there are, ARM recommends 1-2MB/core for the N1 cores in the Graviton2 and it has 512KB/core. AMD has 4MB per core in Zen2 albeit with a slightly weird setup where the L3 is localized to 4 core clusters.

(comment deleted)
Ryzen CPU's seem to have a decent amount of cache's built in: Level 1 cache size 12 x 32 KB 8-way set associative instruction caches 12 x 32 KB 8-way set associative data caches Level 2 cache size 12 x 512 KB 8-way set associative unified caches Level 3 cache size 4 x 16 MB 16-way set associative shared caches
How big are the L1 and L2 caches? I couldn't find it in the article.

  [ec2-user@ip-10-0-42-228 ~]$ lscpu
  ...
  L1d cache:           64K
  L1i cache:           64K
  L2 cache:            1024K
  L3 cache:            32768K
Find a deep dive on the cache and memory micro-architecture here: https://www.anandtech.com/show/15578/cloud-clash-amazon-grav...
Of course, you can't compare those directly with epyc/xeon because of the architectural/instruction set differences.

EDIT: meaning that a larger sized cache wouldn't necessarily deliver better performance (though I think in this case the architectural differences do favor Graviton making better use of cache)

I couldn't find anything on the differences between the two different L1 caches. What's the purpose for a second L1 cache?
The ARM architecture is a modified Harvard architecture, so the CPU has separate instruction and data L1 caches. x86 CPUs don't have this separation.

  $ lscpu
  Architecture:                    x86_64
  Model name:                      AMD Ryzen 7 2700X Eight-Core Processor
  L1d cache:                       256 KiB
  L1i cache:                       512 KiB
  L2 cache:                        4 MiB
  L3 cache:                        16 MiB
Agreed. We have already migrated 90% of our workloads to graviton2. After running our performance tests, it's a no-brainer price/performance boost.
RDS is ridiculously overpriced, I just cut my bill from $1,000 to $50 per month by switching back to EC2 for my database server.
I'm interested in hearing what your setup was like that cost $1,000 that you could replicate on your own for just $50.
I am trying to backup gmail emails my employees send and receive and make them searchable. This will enable a new account manager on an account to easily search and find previous discussions with clients and come up to speed to help them quickly.

We were pulling in millions of emails from the past few years, so we had to do a larger size RDS and we were killed on the IO charges.

We are now just on a medium sized EC2 server.

Do you have suggestions for better ways of doing this?

I can't give you direct recommendations, but it does sound like you might have used Provisioned IOPS storage instead of General Purpose storage which is simply pay by the byte.

If you're willing to post a bit more details, I'm sure people could give you some more detailed options. You basically switched from a fully-managed database solution to a fully DIY solution, and I honestly think you might have just over provisioned the RDS instance. I'm willing to bet you can switch back to RDS and get all the benefits there while remaining within budget. You can get a 1-year reserved t3.medium RDS instance with 50 GB of storage for $40/month.

Here is the rest of the details from my developer on the project-

What was our specific server size and setup and costs with RDS? We were using db.t3.medium (1 CPU, 2 vCPU, 4GB RAM), we were using a DB cluster that has 2 instances db.t3.medium overall cost (~$118 monthly) And now with EC2? We are using c5d.xlarge (4 CPU, 20 ECU, 8GB RAM), we are using one instance c5d.xlarge overall cost (~$140 monthly) What was the big cost savings? Our main issue was in I/O (Read and write), we have very high I/O demand so we was paying for I/O and storage (~$600) and the second issue is CPU credit (~$90 monthly) so the total save is ($600+$90+$118)-$140= $668 per month.

Also EC2 performance is better than RDS because we get a very good cheaper than RDS server.

Note: RDS is configuration less and has support while if you want to use EC2 then you need a person to do configuration and support for DB on it.

Still not clear what your major cost was. Were you using RDS Aurora, which has an IO fee? ($0.20 per 1 million requests?)
Yes, our major cost is IO and yes we are using Aurora and you are correct IO fee is $0.20 per 1 million requests.
So I suggest trying vanilla Postgres RDS, which doesn't have IO fees, but is still managed.

Also, if you're doing such heavy IO, you might want to also look at I3en instances, which have fantastic IO using local storage.

Or you know, just use a smaller RDS instance. The price difference is not 20x over an EC2 instance. It's been a while since I checked, but I remember the markup being less than 2x.
Yup, it's about 1.8x for the instance sizes I checked.
Yeah that’s only true if you aren’t using piops. It can be an order of magnitude in price difference vs i3 when you need lots of I/O.
PIOPS is the same price whether it's attached to an EC2 instance or an RDS instance.
Are there any gotchas with using these machines for general workloads? Are there compatibility issues you need to be aware of? Can I just run a Postgres instance on them and expect everything to work?
It kind of peeves me how often in RDS, the available machine instance types have vastly differing available RDS features.

Here, for example, Amazon's own Aurora MySQL is not available on Graviton processors.

Most multithreaded software has concurrency bugs and many of those bugs have been covered up for years by the relatively friendlier x86 memory model vs weaker models like ARM.

It is not surprising at all that supported products are going to be slow to support other architecture.

Ooooh interesting. Any more info about the difference in models and how it causes this effect?
Quick explanation: https://www.cs.utexas.edu/~bornholt/post/memory-models.html

Basically, although I'm sure someone will shout me down for this, x86 provides something close to sequential consistency among loads and stores even for concurrent programs, whereas ARM and POWER provide almost no kind of order for loads and stores without barriers. High-level languages like C++ do not really hide this problem. It is very easy to write a concurrent C++ program on x86 that appears to work in practice but never works on ARM.

You need proper memory model handling for the program to be compliant even for x86. If you do it otherwise and you see that it works for you, keep in mind that the compiler is allowed to reorder in that case, which can result in very odd breakage.

On x86, sequential consistency isn't the implemented memory model, but rather total store ordering. Stores are buffered inside the core and the guarantee provided is that they'll appear to other cores in the proper order. On SC designs, all the cores have the same view of coherent memory all the time. In such a design, barriers aren't required and are lowered down to NOPs.

(only general-purpose processors that I know of that implement sequential consistency are NVIDIA's Tegra K1 (64-bit variant) and Tegra Xavier)

> many of those bugs have been covered up for years by the relatively friendlier x86 memory model vs weaker models like ARM

I had to take a peek at a friend's Cavium's ThunderX2 kernel issue a while back, which stops having concurrency when release build is swapped with a debug build.

So the specific part that drives you crazy is "Atomic can be reordered with stores - Y" for ARM and N for X86_64.

And changing the compiler optimization flags can turn on/off this bug.

We previously did a product based on the Octeon3. I feel sympathy for anyone who has to deal with Cavium's SDKs.
It is not just ARM instances. Parallel Query, Multi-Master, and Performance Insights, are all locked to specific classes of x86 instances.

It's like Amazon has narrowly tested specific features on specific hardware sizes.

On one hand, it's incredibly annoying. On the other hand part of why you'd choose RDS over deploying MariaDB yourself to an EC2 instance is to shift some of the responsibility of testing & configuration over to AWS. If AWS is disabling features on instances that don't work, this is actually a benefit that they're providing to their users, as you can better trust that any feature enabled on a given product & instance type will "just work".
Also, this is what you're paying for. I have a couple of legacy RDS databases which have been running for a decade. When you're making a promise to a large number of customers that you're going to support something for the long term it's understandable that you're going to be conservative and probably run some sort of closed beta for long enough to get solid operational experience.
Is there somewhere that lists which features are enabled on which instances? We were thinking of switching to Graviton2 RDS instances but not being able to use Performance Insights would be an important factor in making or not making that switch.
Disclosure: I work at AWS on building cloud infrastructure

Amazon Aurora support for AWS Graviton2 powered instances is still in development. I'll share your comments with the team, since direct customer feedback helps us prioritize new launches. If you have an account manager, please also let them know to make a note on your specific interest.

Wouldn't product launches like these be tied to usage data which AWS already has?

Do specific customer requests change the development trajectory significantly?

You can't have usage data for a feature that doesn't exist.
From my perspective, we are fortunate to have a lot of customers and tons of feedback on what they want us to build for them. The most effective way for us to help deliver what customers want is to listen directly to their voluntary feedback. That helps us more deeply understand our customers through their individual stories and anecdotes, rather than looking at the averages that come from statistics and general surveys.

For Amazon Aurora, there are a lot of additional features in development, including offering Graviton2-powered instances. I expect that will deliver price and performance benefits for customers, but it's not the only benefit that's in the works.

It depends on the AWS product.

For the container systems e.g. ECS, EKS, Fargate they have an open roadmap where they drive much of the direction from the community:

https://github.com/aws/containers-roadmap

For others it's completely closed off. At my last company we spent tens of millions a year with AWS and they never listened to us. The account managers have zero pull.

How would known usage data help you decide how urgently customers want to use a new feature that doesn't exist yet?
if amazon is prioritizing postgres/mysql development over aurora development because there's a lot more users using the regular postgres/mssql instead of the aurora equivalent, it's a bit of a reach to suggest that there'd be a lot more demand for aurora on a new processor type.
(comment deleted)
Competitive benchmarks conspicuously absent. I would expect a vendor who controls both the hardware and the application build to easily show that graviton2+rds is beating the pants off percona builds on x86, but I don't see any claims here.
Would be nice. Raw power isn’t everything 20+ years of software optimization on x86 gives it a nice lead. Wonder how long it will take to close that gap. Or if AWS engineers are/have been submitting PRs to the popular open source dbs
AWS engineers most often fork the open source repositories and keep sll their patches only to them, they rarely submit those upstream.
Disclosure: I work at AWS where I build the cloud infrastructure that powers EC2.

One example of a patch that can help with MySQL performance on Graviton2 processors is enabling the Large System Extension atomic operations. You can find our pull request for enabling that here: https://github.com/mysql/mysql-server/pull/302

Unfortunately it seems that their CLA bot auto-closed the submission. Thank you for your comment here on HN, because it prompted me to go check on this PR and it led me to this discovery.

[edit] See also the discussion here about compiler flag defaults for PostgreSQL, to provide optimizations while retaining compatibility for Arm processors without LSE: https://www.postgresql.org/message-id/flat/099F69EE-51D3-421...

AWS provides the binary for RDS users, right? Why do you need to block on upstream for these improvements?
We aren't blocking on upstream for these improvements. You're right that it is easy to optimize only the engine builds that are executing via RDS deployments. But getting optimizations upstream as far as possible yields the maximum downstream effect. And we want open-source software to run exceptionally well on all Arm processors, including Graviton2, no matter who you get your binaries from, or if you build them yourself.
As an aside that conversation on your Github pull request is pretty funny from a sadlol perspective. Thanks for submitting the pull request even when the bot closed the conversation even though you said you submitted the OCA.

This is the automated future that's already here.

I can imagine in the future:

Automated appeals bot: "Thank you for submitting your appeals letter for appealing the Case decision number XYZ. We will need you to submit a signature"

Wrongfully incarcerated person: "OK I will need to find time to get to another computer so I can submit my digital signature"

Automated appeals bot: "Please submit your signature. This is mandatory for all appeals"

Wrongfully incarcerated person: "OK after negotiating with the guards I finally got a computer where I can sign. I've signed my appeal application"

Automated appeals bot: "As we didn't get a signature from you in the set amount of time, your appeals application has been closed. The rest of the sentence will be executed."

Wrongfully incarcerated person is executed.

> 20+ years of software optimization on x86 gives it a nice lead

Not as much as one would hope. x86 is a very complex architecture.

Note that most cellphones run ARM, there are many years of compiler optimizations as well.

> Note that most cellphones run ARM, there are many years of compiler optimizations as well.

Compilation, yes, runtime for DBs? Unlikely.

Personally speaking, I would prefer for folks at Percona to do that type of comparison. Percona provides additional deployment options to customers, as well as support and consulting for Amazon RDS [1]. I think they're in a better position to provide useful information to customers about the pros and cons, and help them make informed buying choices.

[1] https://www.percona.com/blog/2019/12/19/the-benefits-of-amaz...

How likely is it for AWS to come up with instance types offering more cores than x86? It seems like with Intel we'll be stuck at 96 cores for at least the next generation.

128 or more cores would spark my interest, especially on RDS. Graviton2 is built on Neoverse N1 which already has a reference design featuring 128 cores. N2 supports 192 cores.

omg. I tried to benchmark g2 on my dev/test workload and it was so much faster, I ended the test early and simply migrated over to g2.

Untold story: open source and CI is now sufficiently pervasive, that recompiling for ARM just isn't a big deal. Gone are the days of painful recompiles.

tl;dr: remember your first experience with SSD vs HD? that's what the g2 performance boost feels like.