To the extent that this benchmark is valid, this is interesting:
"In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js.
Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages."
TLDR: You may be able to use a "slower" language for Lambda without necessarily losing much.
They are effectively measuring how fast AWS infrastructure can reach whatever it needs (VM?) for each language.
But these are not even cold start times. I'm assuming after the first request this is using the same VM (/backend job, whatever) to reply, so now it just depends on latency to get to that VM. (I'm assuming the runtimes' ability to return "200 OK" is similar and the latency of that is negligible compared to AWS infra that is involved.) Not completely meaningless, but not super useful either, esp without any details while making those bold statements. This is not "Go is equal to Java". This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java". Article is unclear on that.
Maybe this tells us AWS lambda backend infra is way different for .NET Core than Go and Java? ¯\_(ツ)_/¯
> This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java"
I wouldn't even go that far. They are not measuring anything that any real-world application would do. Heck, it's not even a todo app, it's just a "200 OK hello".
It's more like "AWS lambda function invocation overhead for Go is comparable to AWS lambda function invocation overhead for Java".
Ok, now being serious, they need to include a real example here. Whenever I run a Lambda function is to process something and make a call to at least one external service. I was a .NET developer for a long time and I seriously doubt .NET Core has been improved to the point where blocking http calls are handled better than in Go runtime. I may believe .NET Core may have some performance advantages in some CPU bound scenarios, but nothing more. Until then, this benchmark proves nothing.
Isn't the intent with a system like Lambda that you'd not do any kind of blocking action. Instead, you'd trigger some event that the AWS database solutions would receive. Once the DB has finished the query it sends the data back and triggers another lambda function.
EDIT: I write this as someone who does no web programming and has only toyed with AWS. This was my understanding from what I read, but not something I actually made.
The relevant methods in the AWS .Net SDK are all awaitable, i.e. not blocking. Even if you did block on waiting for it, an operation on DynamoDB such as writing a record should typically take under 10ms, which would be ok.
So there is no reason at all not to do it in a lambda.
No. "Once the DB has finished the query it sends the data back and triggers another lambda function"
What mechanism is the DB 'sending' data via? Is it just the response part of the initial query? Then it's executing in the context of the same lambda function. You -could- then call another lambda function from the running lambda function (but why?), but more, what was the function that executed the query doing while it waited?
If it had something it -could- be doing, it should be doing it, so non-blocking is important. If it had nothing it could be doing, it literally does not matter if it was a blocking call or not; it doesn't effect anything.
Now, if you meant you have a trigger in the DB that calls another lambda function, such that the original lambda could complete, yes, of course you could do that. However, if this was a GET style call, not a PUT, your options to get the data back to the user are limited if you don't just keep the REST request open and respond that way. Certainly, any other solution is more complex, probably more expensive, and may not be possible.
I have been doing a lot of experiments with Go and Lambda and have an open source boilerplate app here: https://github.com/nzoschke/gofaas
This benchmark leaves some room for improvement. With Lambda and X-ray you could really deep dive into where time is spent in a request. Here’s an example trace for a Golang function that does some actual work against KMS and DynamoDB:
One thing I found surprising about the Go support is that the ‘lambda.Start’ helper uses reflection to invoke your go func. This could contribute to some of the cold start time.
Lambdas bill at a 100ms minimum. What's the point of a few ms of difference here, unless you're running some computationally-intensive code? If that's what you're doing, the efficiency of the language itself is going to be what matters, not Lambda's speed of executing the runtime.
Why the hate? The methodology is sound, the results are interesting and they show that there is some room for improvement in language support. Also the source code has been provided in case anyone wants to alter the benchmark to be more comparable to their workloads.
Hopefully this sparks a race between the different teams within Amazon. Great job!
I am glad that the person who created this intended for the results to be reproducible. I will try my hand at what's in the git repo, and perhaps compare with some other languages too.
27 comments
[ 5.6 ms ] story [ 86.7 ms ] threadI feel like my world has just flipped upside down.
"In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js.
Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages."
TLDR: You may be able to use a "slower" language for Lambda without necessarily losing much.
at least for small projects
> Go performance is comparable to Java
> .Net Core 2.0 can execute up to 3x faster than Go.
> Consistent performance of compiled vs dynamic
But what are they measuring? How fast can a lambda function return "200 OK hello"? This is just sad.
But these are not even cold start times. I'm assuming after the first request this is using the same VM (/backend job, whatever) to reply, so now it just depends on latency to get to that VM. (I'm assuming the runtimes' ability to return "200 OK" is similar and the latency of that is negligible compared to AWS infra that is involved.) Not completely meaningless, but not super useful either, esp without any details while making those bold statements. This is not "Go is equal to Java". This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java". Article is unclear on that.
Maybe this tells us AWS lambda backend infra is way different for .NET Core than Go and Java? ¯\_(ツ)_/¯
I wouldn't even go that far. They are not measuring anything that any real-world application would do. Heck, it's not even a todo app, it's just a "200 OK hello".
It's more like "AWS lambda function invocation overhead for Go is comparable to AWS lambda function invocation overhead for Java".
I agree there's questionable value here but at least it's bring constrained to as few variables as possible.
EDIT: I write this as someone who does no web programming and has only toyed with AWS. This was my understanding from what I read, but not something I actually made.
So there is no reason at all not to do it in a lambda.
What mechanism is the DB 'sending' data via? Is it just the response part of the initial query? Then it's executing in the context of the same lambda function. You -could- then call another lambda function from the running lambda function (but why?), but more, what was the function that executed the query doing while it waited?
If it had something it -could- be doing, it should be doing it, so non-blocking is important. If it had nothing it could be doing, it literally does not matter if it was a blocking call or not; it doesn't effect anything.
Now, if you meant you have a trigger in the DB that calls another lambda function, such that the original lambda could complete, yes, of course you could do that. However, if this was a GET style call, not a PUT, your options to get the data back to the user are limited if you don't just keep the REST request open and respond that way. Certainly, any other solution is more complex, probably more expensive, and may not be possible.
Inefficient ? Yes. The quest for the $15 page refresh continues.
of course. Like e.g. dynamoDB?
Here are some examples of non-blocking calls from C# to dynamoDB that you could make from a lambda
https://matthiasshapiro.com/2017/03/21/tutorial-dynamodb-in-...
The rest of the AWS SDK for .Net core is like that too. All those Async methods.
Http should also be done in a non-blocking, awaitable way too.
The deleted comment is exactly right - Async/awaits operators are the .Net mechanism to avoid blocking calls, and they are available here.
<proceeds to show averages instead of mean and/or 90th percentile even though maximum spikes are easily orders of magnitude higher than average>
This benchmark leaves some room for improvement. With Lambda and X-ray you could really deep dive into where time is spent in a request. Here’s an example trace for a Golang function that does some actual work against KMS and DynamoDB:
https://twitter.com/nzoschke/status/969598630033162240?s=20
One thing I found surprising about the Go support is that the ‘lambda.Start’ helper uses reflection to invoke your go func. This could contribute to some of the cold start time.
Hopefully this sparks a race between the different teams within Amazon. Great job!
Oh! These test scenarios seem to have a pretty good acceptance in case you decide to do another benchmark in the future: https://www.techempower.com/benchmarks/#section=code&hw=ph&t....