It’s interesting that the author chose to use SHA256 hashing for the CPU intensive workload.
Given they run on hardware acceleration using AES NI, I wonder how generally applicable it is.
Still interesting either way though, especially since there were reports of earlier Graviton (pre v3) instances having mediocre AES NI performance.
Hardware-accelerated SHA support has a patchy history. I wrote an article some years ago about the prevalence of SHA instructions in x86 in x86_64 CPUs [0], like the current mess we see now with AVX-512, Intel invented something useful then declined to continue supporting it, while competitors that were late to the party became the real champions.
Would be interesting to add a cold start + "import boto3" benchmark for Python as importing boto3 takes forever on lambdas with little memory. For this scenario I only know this benchmark but it is from 2021 https://github.com/MauriceBrg/aws-blog.de-projects/tree/mast...
I don't really use Python, but most AWS SDKs seem to be autogenerated for each language, and they're pretty much just thin wrappers over REST calls to interal AWS endpoints.
I dunno why a Python impl would be particularly heavy.
> Node.js: Node.js 22 on arm64 was consistently faster than Node.js 20 on x86_64. There's essentially a “free” ~15-20% speedup just by switching architectures!
Not sure why this is phrased like that in the TL;DR, when ARM64 is just strictly faster when running the same nodejs workload and version.
How is the bootloader/peripheral compatibility on the non-SBC ARM systems these days? Can you plug in a boot disk on different machine and expect it to just work? My main problem with ARM is that many manufacturers act as if they're special little snowflakes and deserve to have their custom patched kernel/bootloader/whatever.
It is a pain to make any new platform useful enough for large adoption. Apple made a lot of effort to get MacBook M1 useable, same for AWS with Graviton.
Eventually it will be adopted for Linux laptops too, even without a specific vendor focusing on it, but it will take time.
One of the easiest hack to reduce your AWS bills is to migrate from x86 to arm64 CPU. Performance difference is negligible, and cost can be upto 50% lower for arm machines. This is for both RDS and general compute (EC2, ECS). Would recommend to all.
I'd say the best price/performance hack on AWS if you don't need web scale is just put your stuff on a tiny EC2 instance, like a t3.micro - it'll be likely faster and more flexible than lambda with much more predictable performance.
You can scale up by changing out to a bigger instance - it's surprising how far you can get with this strategy.
Would be interesting to see a benchmark with the rust binary with successively more “bloat” in the binary to separate out how much of the cold start is app start time vs app transfer time. It would also be useful to have the c++ lambda runtime there too; I expect it probably performs similarly to rust.
Tangent: when you have a lambda returning binary data, it’s pretty painful to have to encode it as base64 just so it can be serialized to json for the runtime. To add insult to injury, the base64 encoding is much more likely put you over the response size limits (6MB normally, 1MB via ALB). The c++ lambda runtime (and maybe rust?) lets you return non-JSON and do whatever you want, as it’s just POSTing to an endpoint within the lambda. So you can return a binary payload and just have your client handle the blob.
I would not be surprised that Rust be faster than Python, but looking at the code of his Benchmarks, I'm not sure that it really means anything.
For example, the "light" test will do calls to "dynamodb".
Maybe you benchmark python, or you benchmark the aws sdk implementation with different versions of python, or just you benchmark average dynamodb latencies at different times of the day, ...
And, at least for Python, his cpu and memory test code looks like to be especially specific. For example, if you use "sha256" or any other hashlib, you will not really benchmark python but the C implementation of the hashlib, that might depend on the crypto or gcc libraries or optimization that was used by AWS to build the python binaries.
Yikes, Node.js did really badly. If this holds up, my take-away would be ...
If I want to use TypeScript for Functions, I should write to the v8 runtimes (Deno, Cloudflare, Supabase, etc) which are much faster due to being much more lightweight.
This is benchmarking `hashlib.sha256` isn't that normally OpenSSL's heavily hand optimized assembly implementation, certainly isn't something written in Python?
Something not mentioned in this article is that which x86_64 and arm64 implementation you get are both relatively frozen in time. I haven't checked recently, but the last time I did, the arm64 implementation was stuck at something like Graviton2.
- How is Rust only one order of magnitude faster than Python?
- How is Python that much faster than Node.js?
So I looked at the benchmark repo.
These benchmarks mean nothing folks!
Each of these benchmarks is just a SHA256 hash. This is NOT a valid way to compare CPUs, except if the only thing you will ever do with the CPU is to execute SHA256 hashes.
Hash functions are not representative of the performance of:
- Compression or decompression (of text, video, or anything really)
- Parsing
- Business logic (which is often dominated by pointer chasing)
So, you can safely ignore the claims of this post. They mean nothing.
I found that Node 22 had ~50ms slower coldstarts than Node 20. This is because the AWS Javascript V3 SDK loads the http request library which became much heavier in Node 22. This happens on the newly released Node 24 as well.
I recommend that if you are trying to benchmark coldstarts on Lambda, you measure latency from a client as well. The Init Duration in the logs doesn't include things like decrypting environment variables which adds ~20ms and other overhead like pulling the function code from S3. The impact of this manifests when comparing runtimes like llrt to Node, the Init Duration is faster than Node, but the E2E time from the client is actually closer because the llrt bundle size is 4-5MB larger than Node.
I'd be much more interested in bare metal ARM64 vs X64 workloads on modern hardware, remember the article that compared Hetzner VPN + Bare metal to Amazon workloads a while back?
Amazon was so ridiculously gimped and expensive that it was almost unfair, thus comparing ARM and X64 on Amazon thus runs into whatever arbitrary "savings" AWS does.
I realise it's not a very substantial point, but it makes me wonder why ARM went with AArch64 rather than just officially naming it the obvious choice of ARM64. ARM64 is both snappier and much clearer as everyone immediately knows you're talking about ARM.
30 comments
[ 2.5 ms ] story [ 49.0 ms ] thread[0]: https://neosmart.net/blog/will-amds-ryzen-finally-bring-sha-...
I dunno why a Python impl would be particularly heavy.
Not sure why this is phrased like that in the TL;DR, when ARM64 is just strictly faster when running the same nodejs workload and version.
AFAIK ARM64 is around 20% cheaper, not sure where you got the 50%.
You can scale up by changing out to a bigger instance - it's surprising how far you can get with this strategy.
In terms of perf, Node has a pretty snappy JIT, and seems to perform OK in the browser, but this seems like something's not right here.
~200ms requests even in the light case are on the very brink of barely acceptable.
On the other hand, Python performs very well, but it's alarming to see it gets slower in every release by a significant margin.
This is in line with my experience using anything written in Node.js
Tangent: when you have a lambda returning binary data, it’s pretty painful to have to encode it as base64 just so it can be serialized to json for the runtime. To add insult to injury, the base64 encoding is much more likely put you over the response size limits (6MB normally, 1MB via ALB). The c++ lambda runtime (and maybe rust?) lets you return non-JSON and do whatever you want, as it’s just POSTing to an endpoint within the lambda. So you can return a binary payload and just have your client handle the blob.
For example, the "light" test will do calls to "dynamodb". Maybe you benchmark python, or you benchmark the aws sdk implementation with different versions of python, or just you benchmark average dynamodb latencies at different times of the day, ...
And, at least for Python, his cpu and memory test code looks like to be especially specific. For example, if you use "sha256" or any other hashlib, you will not really benchmark python but the C implementation of the hashlib, that might depend on the crypto or gcc libraries or optimization that was used by AWS to build the python binaries.
If I want to use TypeScript for Functions, I should write to the v8 runtimes (Deno, Cloudflare, Supabase, etc) which are much faster due to being much more lightweight.
Then I deoloyed an ECS task with ALB and got something like <5ms.
Has anybody gotten sub-10ms latencies with Lambda Https functions?
- How is Rust only one order of magnitude faster than Python?
- How is Python that much faster than Node.js?
So I looked at the benchmark repo.
These benchmarks mean nothing folks!
Each of these benchmarks is just a SHA256 hash. This is NOT a valid way to compare CPUs, except if the only thing you will ever do with the CPU is to execute SHA256 hashes.
Hash functions are not representative of the performance of:
- Compression or decompression (of text, video, or anything really)
- Parsing
- Business logic (which is often dominated by pointer chasing)
So, you can safely ignore the claims of this post. They mean nothing.
I found that Node 22 had ~50ms slower coldstarts than Node 20. This is because the AWS Javascript V3 SDK loads the http request library which became much heavier in Node 22. This happens on the newly released Node 24 as well.
I recommend that if you are trying to benchmark coldstarts on Lambda, you measure latency from a client as well. The Init Duration in the logs doesn't include things like decrypting environment variables which adds ~20ms and other overhead like pulling the function code from S3. The impact of this manifests when comparing runtimes like llrt to Node, the Init Duration is faster than Node, but the E2E time from the client is actually closer because the llrt bundle size is 4-5MB larger than Node.
Amazon was so ridiculously gimped and expensive that it was almost unfair, thus comparing ARM and X64 on Amazon thus runs into whatever arbitrary "savings" AWS does.