It does, but the numbers are not too far. It's 3.5ms for Java versus 2.3ms for C++.
Even when running a lot of requests when 99% latency becomes the average one, that's like 1 millisecond difference, way under typical user's network latency.
These numbers are meaningless unless accompanies by specific usecases - don't just choose java or C++ just because the benchmark says its fast or the best.
High frequency trading needs that extra milli second, but a batch job or backend computation won't.
Clearly it is, since the C4 collector manages to do it. To be fair though, the "pauseless" part does come at the cost of a slight slowdown in normal code execution since it instates read barriers to work its magic. This means the lack of pauses is paid for by significantly higher overall CPU overhead for garbage collection compared to a collector design that does include pauses. It could be worth it for some workloads where it is very important to have low 99th percentile response times though.
This! Benchmarks that fully sustains the throughput are rarely useful. If you’re running at that capacity your service is melting.
Latency histogram for fixed throughout (e.g. 1k RPS, 10k RPS, 50k RPS) are far more useful. You want to know if going from 1k to 10k increases latency.
This is a classic issue for Java vs compiled to binary languages such as C. Java does its own memory management thus allocates a big chunk of memory from the OS at startup. It also has garbage collection which runs from time to time dependent on what gc implementation is used.
These issues can be mitigated by carefully tuning the parameters of the Java virtual machine, but in practice, for most projects, it is not an issue.
You are correct. Some of this has changed significantly somewhere between Java 8 and 16. For example the Java applications we're running against Hotspot 16 regularly gives memory back to the host. It's GC pauses are also quite insignificant now for a microservice regularly allocating and GCing.
You would? I wouldn't. It definitely looks like somewhat pathological case to me, at least in 2021. Maybe five years earlier the number would be appropriate, but there seems to be something wrong with Go slowing down this much at that small a heap. I'm wondering if it was tuned at all.
In 2021, Go is supposed to have GC pauses on the order of several ms at worst. Not 40 ms. So it kind of is surprising, something seems to be broken there. I'm wondering whether this isn't a limitation caused by forcing a single core operation, the runtime might not be designed for that.
EDIT: Someone else noted (https://news.ycombinator.com/item?id=27085507) a discussion on Reddit where a <5ms latency was achieved in 99.9% cases, so perhaps this is indeed a subpar result.
When you allocate against a running GC it will penalize you for this (literally sleep your thread) - hence garbage tail latency. The solution is to not do allocation which is what gogo library strives for
Most of the phases of GC can be done parallel, so allocation should stop only for a really small amount of time. For comparison, Java's low-latency GC (which optimize for latency but may reduce throughput), ZGC can do worst-case <1ms pause times, that is the OS scheduler will cost more than GC.
I’m talking about Go garbage collector specifically which does mark and sweep and will slow down functions doing frequent allocations on purpose in order to catch up. This is different from stw kind of scenario e.g shenandoah
Iirc, the official protobuf module for Go still uses
reflection in the generated code as opposed to fully
generating the encoding and decoding code, so maybe that
creates additional garbage or performance issues or lock
contention. I think I remember there being an alternative
module that fully generates the code, and it would be
interesting to see that in the table as well.
It's factual if you consider average latency important, which is debatable. On 90th, 95th and 99th percentile latency, it isn't the most performant. On memory usage, it isn't.
As a rust fanboy, I was very happy to see the 99% and memory numbers. Still, very impressive for Java, I wonder whether this is just trading off the 99% percentile peak vs higher common case throughput? I guess rust actually allocates/deallocates all the memory on each request while java batches the deallocation?
Looking at the memory column, it looks to me like Java benchmark uses a large heap to avoid deallocations for most of requests. This means less but larger garbage collections and reduces average time while increasing 99% where the GC shows up.
A long time ago I had to implement a real time service with Java. The best solution was to use whopping 16 megabytes of heap so that it would do a full GC multiple times per second but each of the GCs lasted less than a millisecond.
No idea. It was much faster than the service really needed to be, so we just set the heap to 128M in production and never heard about any latency problems.
In server applications you really shouldn't use the default allocator, but have one allocator per request which can then allocate and batch-deallocate very quickly. Unless you want to keep objects after the request ends, of course.
Kind of. You can use a bump allocator like bumpalo[0], but any code that allocates needs to be aware of it (or it won't be able to take advantage of it). It also needs to reinvent all collection types (bumpalo provides strings and vectors atm).
The plan is to parametrize all std collections on the allocator eventually, but that's not stable yet.
Hopefully this project could allow allocations inside memory-mapped files (which can then later be loaded at a different base address) in one quick load operation (instead of serializing everything).
"background_threads (disabled by default): enables background threads by default at run-time. When set to true, background threads are created on demand (the number of background threads will be no more than the number of CPUs or active arenas). Threads run periodically, and handle purging asynchronously. [...]"
I'd be curious about why the Java benchmarks are so much better than the Kotlin ones. I could buy hotspot JVM optimizations making it better than, say, Rust. But shouldn't Kotlin be leveraging the same speed ups?
The explanations could be: a) the bytecode generated by the Kotlin compiler is much worse and b) the benchmark Kotlin code is not as good or optimized as the Java one.
Java has the compiler in memory while running. Rust's compiler is a separate program so isn't in memory while it runs. That by itself is a lot of code.
Like all guest languages it needs to generate additional boilerplate to pretend to be Java, and support its additional features not available out of the box in JVM bytecodes.
Kotlin should be close-to-identical* in terms of bytecode it immits for straightforward code like this, it's designed to very closely match Java capabilities, among others to make sure interop is very good.
* Exception is supporting things like default methods on JVM 1.6 bytecode
It can only generate bytecode features that ART understands, and D8 is able to convert into DEX, thus is the price of Android marriage.
Going forward while for Java code there is no worry about using SIMD, JNI replacement, value types, Kotlin code will need to make use of KMM for code that is supposed to target both JVM and Android.
Those are bytecode level related. You can't consume Java 16 libraries on Android either. It's not related to Kotlin at all.
You are either being deliberately obtuse or you genuinely have difficulty to understand that Java version, bytecode version, Kotlin compiler output and what bytecode Android supports are totally independent concepts.
Optimizers can be finicky; kotlin and Java are going to generate different bytecode. The optimiser's been optimized for Java, and those differences will matter
Looking at the build files, different versions of the libraries appears to be used. The Java implementation is configurable but defaults to something called "direct executor".
Coroutines in Kotlin can be convenient, but they are overall about 20% slower than non-suspending code. I didn't spend too much time investigating the cause, but I believe it's because of the extra object that is passed along in all method calls.
In my case, I have a programming language interpreter implemented in Kotlin, and as an experiment I made the entire interpreter using suspending calls so that I could call asynchronous functions, and my performance tests dropped by about 20%.
As I understand it, Kotlin coroutines generate what is known as irreducible control flow. This means that loops have more than one entry point, or equivalently, there are loops that have a goto jumping into the body from the outside. (Java the language and I believe also Kotlin the language don't have goto, but the Java bytecode does.)
Irreducible loops make many optimizations much more complex. Bytecode generated by javac never contains irreducible loops, and since bytecode generated by javac is the number 1 use case targeted by JVM JIT compilers, they probably just don't bother trying to be that smart about irreducibility.
It's interesting that the java ones use an order of magnitude more ram than rust. Is that just the overhead of being bytecode, or a different choice in processing algorithm?
If I remember correctly the bytecode is interpreted and executed by the runtime. This means the bytecode is fully in memory as bytecode when the jvm loads .class files. The native code for the JVM interpreter and runtime execution is also present in memory.
At this point execution happens without "duplicate code" in memory.
Note: the JVM could compile the bytecode fully into native code before execution but that's an implementation detail for a specific JVM implementation.
Then we add the JIT. The JIT will notice hot paths during interpretation/execution and compile a method to native code and inject it so it calls the jit_native version instead of the bytecode_interpreted version. The JIT does not overwrite original bytecode but stores the compilation result separately.
So to come back to your statement "both bytecode and native code generated from bytecode in memory at the same time": yes, but only for the JIT-compiled hot paths.
This to me is the "power" of the JVM + JIT. Run interpreted bytecode and compile to native for the hot paths.
It's because Rust frees memory at the moment it's not needed while Java runtime maintains a pool of memory and does garbage collection every now and then. This is a huge performance boost for code that does a lot of small allocations, and if you want to play dirty, reserving a large enough heap may allow you to run through a benchmark without a single garbage collection.
Seriously, every now and then there's a story about some fintech company who use Java and use large enough heap that there's no garbage collection during stock market opening hours.
Possibly. I haven't tried running this benchmark with different heap sizes, but to me the 99% latency looks like it doesn't try to avoid GC. The JVM is really fast at memory management.
Indeed, but is relevant? If you have a machine with average memory installed, both are fine. If you have something with very little memory available, both would be too much.
Sure, it's up to you what you optimize for. Easy and fast to scale vertically or better numbers in terms of latency/through-output? With the former, go for cloud. For the latter, go for dedicated. Want best performance for each buck spent? Again, dedicated.
If I am already using Lambda because of infrastructure management costs, it is always nice to reduce my bill by using less memory and/or improving response time.
> If you're out after performance you wouldn't use AWS Lambda or even AWS at all.
That point makes as much sense as complaining that if you were after performance you'd use a Formula1 car and not a Tesla.
People who live in the real world and have to do real work need to use real world tools, and one of which is AWS Lambda.
Let's put things in perspective: would it make any sense at all to advise a company to not only rewrite a whole application stack from scratch in your pet performant language but also jump head on to some boutique service provider? I mean, who in their right mind would get accountants involved in a goal to shave a few milliseconds over a few gRPC calls? Is a suggestion to improve performance expected to be considered even sane if it requires rewriting everything and change shop?
> People who live in the real world and have to do real work need to use real world tools, and one of which is AWS Lambda.
Another real tool is dedicated servers, something people used before "cloud" and something that people who care about performance still uses. AWS even offers dedicated servers themselves, so not sure why you would need to involve any "boutique service provider". Otherwise you have OVH, Hetzner and a range of others who compete well with AWS on dedicated instances as well, neither I'd say are "boutique".
> rewrite a whole application stack from scratch in your pet performant language
Not sure where this comes from, which one of these languages are "pet performant (SIC) languages"?
> who in their right mind would get accountants involved in a goal to shave a few milliseconds over a few gRPC calls
Hmm, unless the accountants are involved somehow in the API design (not sure what you're building), I don't know what the accountants have to do with anything here.
In the end, AWS and Lambda absolutely does not fit every use case. Depending on your use case, and if it's important a few ms here and there, you chose different solutions. Since this benchmark is about throughoutput, I thought we were discussing the use case of needing the best throughoutput, otherwise this is all off-topic. And with that, I'm just sharing that if that is your focus, you would probably not be using Lambda in the first place, as you'll get very shitty throughoutput and you have to pay a lot, compared to other mature solutions for this that we already had for many many years.
Most instances on AWS get dedicated cpu and memory resources, so noisy neighbors isn't a problem. The difference with metal is that you get the whole box for yourself.
> Most instances on AWS get dedicated cpu and memory resources, so noisy neighbors isn't a problem
First time I heard of that and also doesn't match my own experience using AWS. AWS tends to have a lot of noise from neighbors but might be because of the region I was using. Could you link the official statement you got this from?
> The difference with metal is that you get the whole box for yourself.
Yes, + you normally avoid virtualization as that can have impact on your performance too. I'm glad we agree there is a difference that is worth mentioning when it comes to performance :)
At 15 minutes. Afaik it's all but T type instances that get reserved resources. Might have been different in early AWS, pre nitro etc.
Something that i just remembered is that in AWS dedicated means that the hardware is dedicated to you, so no other customer vm on it, but you can have multiple vm's on it.
If you have a large number of users, accounting is involved in virtually everything you do in the capacity of serving them. Engineering history is littered with projects to shave milliseconds or cents off a very small component which is used extremely frequently.
> If you have a large number of users, accounting is involved in virtually everything you do in the capacity of serving them
What do you count as "large number of users"? Worked on projects with millions of users, thousands of requests per second and when we refactored based on increasing throughoutput/decreasing latency we had exactly 0 accountants involved, even if the company had accountants in-house and full-time.
I think it depends more on the company size than the number of users you have.
> Engineering history is littered with projects to shave milliseconds or cents off a very small component which is used extremely frequently.
Yup, agree and been there myself, hence my comments about staying away from Lambda and VPS for this kind of focus and go for dedicated instances where this matter.
Maybe interacting with the finance department was above your pay grade. The bigger the operation, the more costs matter. That's a thumb-rule you can find anywhere.
> Maybe interacting with the finance department was above your pay grade.
It was not, we were a nimble and lightweight team with everyone doing everything they could possibly do. I had insight into accounting and helped them with implementing some stuff and some of the designers helped with frontend as some of them knew HTML and CSS and so on.
> The bigger the operation, the more costs matter. That's a thumb-rule you can find anywhere.
Yeah... I think I agree? "The bigger the operation" is referring to the employees working for the company, not the number of users right? If so, that was exactly my point. It's about the number of employees that dictate if accountants gets involved or not, not the number of users.
Being the SME on accounting for your team is not even a little bit comparable to working with the department with accountability to the company's cash flow. I don't know why you have tried that angle. It is a significant category error.
You should really rephrase that as "I was totally unaware that there were accountants involved" because it is simply inconceivable that a business activity involving allocating resources and changing operational needs would not be tracked. Either you are grossly misrepresenting your personal anecdote or you are filling in quite a few blindspots.
Yeah, it's completely impossible we both have two different experiences, I'm either lying or missing something, because every engineering-team waits for accountants before they do performance optimizations or capacity planning.
> Indeed, but is relevant? If you have a machine with average memory installed, both are fine.
The whole point is that you don't. You literally pay for the memory your application requires, proportionally to the amount of memory. The difference in the resources required to run is around two orders of magnitude.
Let's put things in perspective: in some platforms such as AWS Lambda, you are charged per memory used per second.
Yeah, sure, that's one use case, to run it on AWS Lambda. Everyone does not, and if you really care about latency/throughoutput, you won't be anywhere near Lambda, and probably not even on AWS or any "cloud" for that matter.
Paying for the amount of memory you use down to the MB, is hardly something everyone does, which makes it weird that you are now the third reply that assume this runs on AWS/Lambda.
> Yeah, sure, that's one use case, to run it on AWS Lambda.
It's not an edge case. It is a clear and irrefutable example that you pay for the memory you use.
Let's be very clear here: when you provision a VM anywhere in the world, you have to pick how much memory you require. You are charged for that memory, proportionally to the memory you require. If your app requires over 100x memory to run, that comes out of your wallet.
> when you provision a VM anywhere in the world, you have to pick how much memory you require
Yes, thank you! That's exactly my point! You create a VM (or a dedicated instance) somewhere, they ask you for the memory usage up front. Usually they start at 128MB and go their way up from there.
Even if you take the instance with the smallest amount of memory, you'll fit any of the benchmarked programs, effective making the "115.41 MiB Java vs 4.15 MiB Rust" statement not as important anymore.
> You are charged for that memory, proportionally to the memory you require. If your app requires over 100x memory to run, that comes out of your wallet
Hm, maybe on Lambda you pay more if the memory your app uses grows. But this is certainly not standard for "normal" hosting where you rent either a VM or proper instance. Then the memory grows until it cannot take any more memory, and the process either crashes, gets killed by OOM or does whatever operation you've designed it to take when running out of memory.
Giving further proof that if you actually care about increasing throughoutput + saving costs, you won't be anywhere near Lambda. Thanks for the additional resource and confirmation.
You're insisting in a red herring that tries to misrepresent the actual problem and the whole point of this discussion.
The whole point is that memory costs money, and the more memory you require, the more you pay.
AWS Lambdas is a clear example that demonstrates this, but is not an isolated case. This is the case for all cloud providers selling VM time. All of them.
Even bare metal providers charge you more for more memory.
Isn't it clear that if your app requires more and more memory, that comes out of your pocket? Is this something that really warrants a debate?
The problem is that the only way to get more CPU, you need to assign more memory to lambda. Most node.js lambdas would have stable idle/low memory usage around 200MB but you get ~1/9of vCPU core, this have severe impact performance. You are forced to overprovision lambdas with more memory, even then performance is still very much a disappointment. People that sold promise of serverless framework should have a special place in hell.
One project I worked on recently ran on containers with 256MB memory, on k8s. Main app would use ~80MB, plus five or six sidecars for metrics, logging, discovery, http proxy etc using 5-10MB each. Having a 100MB+ baseline makes it prohibitive for most of those applications.
The JVM's memory usage is based on a heuristic. It can get away with usually much less, but it errs on the bigger memory footprint side, because that way it has to do less actual work GC-wise.
For example, anecdotally a small, semi-complex JavaFX 2D game I wrote uses by default something like 250 MiB of RAM, but manually limiting it to max 80 was possible (but in the latter case, the GC had to run quite often)
It matters when you do other stuff than rust running a benchmark. If every piece of your software uses 25x the memory you might end up needing 25x the servers for your application.
Most of that is probably up front cost for a jvm instance. So the 25x only happens because it is a micro benchmark that isn't doing anything useful. Might as well complain that the size of a C executable with an empty main is infinitely larger than a python script that does nothing.
So you’re saying Java is a bad choice for something like a sidecar (most likely multiple) running along your main app, due to that JVM overhead. I think you’re all agreeing here.
You can probably cut down the default heap size, get it to use 32 bit pointers for a small heap, etc. . The JVM has quite a few startup options that you could use if the memory footprint of hundreds of tiny instances is a bottleneck for you. Might even speed things up a bit more.
I worked in a web scraping team for a couple of years and memory was always the bottleneck since we ran hundreds of thousands of instances. Our scrapers were written in python and with the interpreter + all the imports we had 40MB reserved before even starting work.
Not just that, but they were very sensitive to fluctuations.
How can people make generalized statements like this without anything about the use case or problem that would have to be solved?
You have exactly 0 information available and yet you make a comment like this? Is this why we have cargo-culting?
Not every solution to a problem needs to use as little memory as possible. And also not every solution can ignore memory usage. It depends, of course.
Since this thread is about throughoutput and optimizing for that, I could tell you that most times I've been put in charge for optimizing throughoutput, having low memory usage have been pretty far down the list.
Unless you talk about embedded, I have to disagree. RAM is the cheapest resource to scale and it will likely continue to grow without problems. There are server machines with 1TB of RAM available. And in the domain where Java is used, it uses totally apt amounts of RAM compared to other managed languages.
Not every solution to a problem needs to use as little memory as possible.
But that's not the point. People aren't suggesting micro managing memory. Java has a well deserved reputation of being a memory hog and its using 20x what is needed in this benchmark.
In optimizations, you focus on a few metrics, like throughput, latency, resource usage, but deciding to improve one can (and usually does) impact the others. Java (that is, OpenJDK) usually trades of RAM for throughput.
And compared to high-level languages doing comparable work, it doesn't use much memory. Of course a low-level language like Rust can be more lean, but that's beside the point. And in this case presumably, Rust could be made to have a better throughput by using more memory.
I haven't looked at the benchmark implementations but one particular area I've heard other languages to lag behind java is cost of allocation.
Since openjdk has had to cope with lack of (user defined) value types and the garbage heavy ecosystem it has a very well optimised GC for handling heap allocations.
I wonder if the results will be different if other language benchmarks (C++ or rust) use different allocators (eg: bump allocator, per request collectors) for cheaper allocation.
Most of the compared languages (except C#) are likely to lag behind Java on many things, performance included, since Java has been around since 1996, Golang only been around since 2009 and Rust 2010. More time = more engineering time to optimize.
This comparison is more about the different gRPC implementations rather than the languages. There is no inherent reason why Java (as a language) should be faster than C++ or Rust in a benchmark like this. A lot of time and money have been spent optimizing Java (or HotSpot), but only because Java is really hard to optimize compared to GC-less low level languages.
While this _could_ be correct if we are just talking about Go, which has its own backend and optimizer, it is definitely not the case for Rust, which uses LLVM. The differences seen in these benchmarks are all about the gRPC implementation used; languages are irrelevant. C++, Rust, Go and Java all have their own standalone implementation written from scratch, so it's kind of an apples and oranges situation here. You can write good or crummy code in any language for what it's worth.
I don't know, is it though? Properly done reference counting AFAIK requires using atomic increase/decrease in order to be safe, and that creates a bit of overhead every time you assign a reference, while assignment to a reference with a precise GC is basically a pointer assignment. It's much better for latency though, given that the overhead from rc is deterministic. It has been a few years though since I've looked into garbage collection techniques, so I could be a bit rusty about the current state of the art.
You only need atomic reference counting if you're sharing objects between multiple threads, but if you use an object from one thread at a time then non-atomic inc/dec is enough. Rust allows you to make the choice between the two kinds and the compiler can infer which kind you need to use.
Yeah - Rust can, because it also ensures you can't exchange data unsafely between threads, but C++ can't. That's why `std::shared_ptr` has to be thread-safe.
It is worth noting that the 90th, 95th, and 99th percentile latencies for Java are not the best ones.
Also, I believe the heavily editorialized title should be replaced with something that is close to the title of the original post, such as "2021-04-13 gRPC benchmark results".
AFAIK It is far from being usable benchmark in real world.
Why isn't teh SDK/Language major versions not mentioned in wiki page ?
Am I making mistake in reading this .... As the number of CPUs increases req/s and latency does not change, except for few java and rust readings. Network connections and IO isn't seem to scale as number of CPUs increases ?
There are probably better reasons to use GRPC in a language other than C++ than than this, like the annoying dependency on Abseil and the general disadvantages that come with using anything from Google.
Unfortunately I've never found a best-in-class modern RPC system for C++ that embraces integration with other libraries (for I/O, concurrency, etc)
Thrift (Apache not Facebook) and CapnProto are the only other contenders that I know of.
My hypothesis is that the Java implementation has simply received the most R&D and tuning. It’s developed mostly by Google, where Java has a large footprint.
Nice benchmark, but I think the results tell us more about the grpc+protobuf+http2 libraries in use than the languages. It's quite amazing though to see such a highly optimized Java libraries.
200 comments
[ 3.6 ms ] story [ 260 ms ] threadIt does, but the numbers are not too far. It's 3.5ms for Java versus 2.3ms for C++.
Even when running a lot of requests when 99% latency becomes the average one, that's like 1 millisecond difference, way under typical user's network latency.
These numbers are meaningless unless accompanies by specific usecases - don't just choose java or C++ just because the benchmark says its fast or the best.
High frequency trading needs that extra milli second, but a batch job or backend computation won't.
Latency histogram for fixed throughout (e.g. 1k RPS, 10k RPS, 50k RPS) are far more useful. You want to know if going from 1k to 10k increases latency.
These issues can be mitigated by carefully tuning the parameters of the Java virtual machine, but in practice, for most projects, it is not an issue.
No measurements are valuable all by themselves. None.
EDIT: Someone else noted (https://news.ycombinator.com/item?id=27085507) a discussion on Reddit where a <5ms latency was achieved in 99.9% cases, so perhaps this is indeed a subpar result.
pgc: ParallelGC sgc: SerialGC g1gc: G1GC she: ShenandoahGC zgc: ZGC
A long time ago I had to implement a real time service with Java. The best solution was to use whopping 16 megabytes of heap so that it would do a full GC multiple times per second but each of the GCs lasted less than a millisecond.
The plan is to parametrize all std collections on the allocator eventually, but that's not stable yet.
[0]: https://docs.rs/bumpalo/3.6.1/bumpalo/
https://crates.io/crates/jemalloc-sys
"background_threads (disabled by default): enables background threads by default at run-time. When set to true, background threads are created on demand (the number of background threads will be no more than the number of CPUs or active arenas). Threads run periodically, and handle purging asynchronously. [...]"
> keep objects after the request ends
Yes. Can that be made in a way compatible with per request allocators?
The explanations could be: a) the bytecode generated by the Kotlin compiler is much worse and b) the benchmark Kotlin code is not as good or optimized as the Java one.
* Exception is supporting things like default methods on JVM 1.6 bytecode
Additionally it is stuck on Java 8 view of the world, otherwise those .class files won't be usable on Android toolchain thanks Google.
I know for a fact it uses different bytecode features if the level is >= 1.8, not sure how smart it is above that.
Going forward while for Java code there is no worry about using SIMD, JNI replacement, value types, Kotlin code will need to make use of KMM for code that is supposed to target both JVM and Android.
It can do stuff android doesn't support if you set the bytecode target level to > 1.8
It's like using modern Javascript features but providing a polyfill for older browsers.
Also stuff like value classes will have different semantics in memory consumption and performance across targets.
Maybe some learning required?
You are either being deliberately obtuse or you genuinely have difficulty to understand that Java version, bytecode version, Kotlin compiler output and what bytecode Android supports are totally independent concepts.
It is going to be fun to port back stuff to Java.
https://github.com/LesnyRumcajs/grpc_bench/blob/master/kotli...
https://github.com/LesnyRumcajs/grpc_bench/blob/master/java_...
Looking at the build files, different versions of the libraries appears to be used. The Java implementation is configurable but defaults to something called "direct executor".
Maybe that explains the difference.
In my case, I have a programming language interpreter implemented in Kotlin, and as an experiment I made the entire interpreter using suspending calls so that I could call asynchronous functions, and my performance tests dropped by about 20%.
Irreducible loops make many optimizations much more complex. Bytecode generated by javac never contains irreducible loops, and since bytecode generated by javac is the number 1 use case targeted by JVM JIT compilers, they probably just don't bother trying to be that smart about irreducibility.
If I remember correctly the bytecode is interpreted and executed by the runtime. This means the bytecode is fully in memory as bytecode when the jvm loads .class files. The native code for the JVM interpreter and runtime execution is also present in memory.
At this point execution happens without "duplicate code" in memory.
Note: the JVM could compile the bytecode fully into native code before execution but that's an implementation detail for a specific JVM implementation.
Then we add the JIT. The JIT will notice hot paths during interpretation/execution and compile a method to native code and inject it so it calls the jit_native version instead of the bytecode_interpreted version. The JIT does not overwrite original bytecode but stores the compilation result separately.
So to come back to your statement "both bytecode and native code generated from bytecode in memory at the same time": yes, but only for the JIT-compiled hot paths.
This to me is the "power" of the JVM + JIT. Run interpreted bytecode and compile to native for the hot paths.
Seriously, every now and then there's a story about some fintech company who use Java and use large enough heap that there's no garbage collection during stock market opening hours.
Why would cloud provider be a business in the first place, do you think?
That point makes as much sense as complaining that if you were after performance you'd use a Formula1 car and not a Tesla.
People who live in the real world and have to do real work need to use real world tools, and one of which is AWS Lambda.
Let's put things in perspective: would it make any sense at all to advise a company to not only rewrite a whole application stack from scratch in your pet performant language but also jump head on to some boutique service provider? I mean, who in their right mind would get accountants involved in a goal to shave a few milliseconds over a few gRPC calls? Is a suggestion to improve performance expected to be considered even sane if it requires rewriting everything and change shop?
Another real tool is dedicated servers, something people used before "cloud" and something that people who care about performance still uses. AWS even offers dedicated servers themselves, so not sure why you would need to involve any "boutique service provider". Otherwise you have OVH, Hetzner and a range of others who compete well with AWS on dedicated instances as well, neither I'd say are "boutique".
> rewrite a whole application stack from scratch in your pet performant language
Not sure where this comes from, which one of these languages are "pet performant (SIC) languages"?
> who in their right mind would get accountants involved in a goal to shave a few milliseconds over a few gRPC calls
Hmm, unless the accountants are involved somehow in the API design (not sure what you're building), I don't know what the accountants have to do with anything here.
In the end, AWS and Lambda absolutely does not fit every use case. Depending on your use case, and if it's important a few ms here and there, you chose different solutions. Since this benchmark is about throughoutput, I thought we were discussing the use case of needing the best throughoutput, otherwise this is all off-topic. And with that, I'm just sharing that if that is your focus, you would probably not be using Lambda in the first place, as you'll get very shitty throughoutput and you have to pay a lot, compared to other mature solutions for this that we already had for many many years.
First time I heard of that and also doesn't match my own experience using AWS. AWS tends to have a lot of noise from neighbors but might be because of the region I was using. Could you link the official statement you got this from?
> The difference with metal is that you get the whole box for yourself.
Yes, + you normally avoid virtualization as that can have impact on your performance too. I'm glad we agree there is a difference that is worth mentioning when it comes to performance :)
At 15 minutes. Afaik it's all but T type instances that get reserved resources. Might have been different in early AWS, pre nitro etc.
Something that i just remembered is that in AWS dedicated means that the hardware is dedicated to you, so no other customer vm on it, but you can have multiple vm's on it.
What do you count as "large number of users"? Worked on projects with millions of users, thousands of requests per second and when we refactored based on increasing throughoutput/decreasing latency we had exactly 0 accountants involved, even if the company had accountants in-house and full-time.
I think it depends more on the company size than the number of users you have.
> Engineering history is littered with projects to shave milliseconds or cents off a very small component which is used extremely frequently.
Yup, agree and been there myself, hence my comments about staying away from Lambda and VPS for this kind of focus and go for dedicated instances where this matter.
Maybe interacting with the finance department was above your pay grade. The bigger the operation, the more costs matter. That's a thumb-rule you can find anywhere.
It was not, we were a nimble and lightweight team with everyone doing everything they could possibly do. I had insight into accounting and helped them with implementing some stuff and some of the designers helped with frontend as some of them knew HTML and CSS and so on.
> The bigger the operation, the more costs matter. That's a thumb-rule you can find anywhere.
Yeah... I think I agree? "The bigger the operation" is referring to the employees working for the company, not the number of users right? If so, that was exactly my point. It's about the number of employees that dictate if accountants gets involved or not, not the number of users.
You should really rephrase that as "I was totally unaware that there were accountants involved" because it is simply inconceivable that a business activity involving allocating resources and changing operational needs would not be tracked. Either you are grossly misrepresenting your personal anecdote or you are filling in quite a few blindspots.
Whatever happened to "Assume good faith"?
The whole point is that you don't. You literally pay for the memory your application requires, proportionally to the amount of memory. The difference in the resources required to run is around two orders of magnitude.
Let's put things in perspective: in some platforms such as AWS Lambda, you are charged per memory used per second.
Paying for the amount of memory you use down to the MB, is hardly something everyone does, which makes it weird that you are now the third reply that assume this runs on AWS/Lambda.
It's not an edge case. It is a clear and irrefutable example that you pay for the memory you use.
Let's be very clear here: when you provision a VM anywhere in the world, you have to pick how much memory you require. You are charged for that memory, proportionally to the memory you require. If your app requires over 100x memory to run, that comes out of your wallet.
Never said it was an edge case....
> when you provision a VM anywhere in the world, you have to pick how much memory you require
Yes, thank you! That's exactly my point! You create a VM (or a dedicated instance) somewhere, they ask you for the memory usage up front. Usually they start at 128MB and go their way up from there.
Even if you take the instance with the smallest amount of memory, you'll fit any of the benchmarked programs, effective making the "115.41 MiB Java vs 4.15 MiB Rust" statement not as important anymore.
> You are charged for that memory, proportionally to the memory you require. If your app requires over 100x memory to run, that comes out of your wallet
Hm, maybe on Lambda you pay more if the memory your app uses grows. But this is certainly not standard for "normal" hosting where you rent either a VM or proper instance. Then the memory grows until it cannot take any more memory, and the process either crashes, gets killed by OOM or does whatever operation you've designed it to take when running out of memory.
The whole point is that memory costs money, and the more memory you require, the more you pay.
AWS Lambdas is a clear example that demonstrates this, but is not an isolated case. This is the case for all cloud providers selling VM time. All of them.
Even bare metal providers charge you more for more memory.
Isn't it clear that if your app requires more and more memory, that comes out of your pocket? Is this something that really warrants a debate?
For example, anecdotally a small, semi-complex JavaFX 2D game I wrote uses by default something like 250 MiB of RAM, but manually limiting it to max 80 was possible (but in the latter case, the GC had to run quite often)
Not just that, but they were very sensitive to fluctuations.
You have exactly 0 information available and yet you make a comment like this? Is this why we have cargo-culting?
Not every solution to a problem needs to use as little memory as possible. And also not every solution can ignore memory usage. It depends, of course.
Since this thread is about throughoutput and optimizing for that, I could tell you that most times I've been put in charge for optimizing throughoutput, having low memory usage have been pretty far down the list.
Those problems are called "one-offs" relative to things I design for scale.
Yes, "relative to things I design"
You think that applies to everyone?
You think that comes close to applying to people focusing on improving throughoutput specifically?
But that's not the point. People aren't suggesting micro managing memory. Java has a well deserved reputation of being a memory hog and its using 20x what is needed in this benchmark.
Since openjdk has had to cope with lack of (user defined) value types and the garbage heavy ecosystem it has a very well optimised GC for handling heap allocations.
I wonder if the results will be different if other language benchmarks (C++ or rust) use different allocators (eg: bump allocator, per request collectors) for cheaper allocation.
Also, I believe the heavily editorialized title should be replaced with something that is close to the title of the original post, such as "2021-04-13 gRPC benchmark results".
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
[1] https://news.ycombinator.com/newsguidelines.html
Not really anything here:
https://github.com/LesnyRumcajs/grpc_bench/tree/master/java_...
https://github.com/LesnyRumcajs/grpc_bench/tree/master/java_...
Why isn't teh SDK/Language major versions not mentioned in wiki page ?
Am I making mistake in reading this .... As the number of CPUs increases req/s and latency does not change, except for few java and rust readings. Network connections and IO isn't seem to scale as number of CPUs increases ?
Unfortunately I've never found a best-in-class modern RPC system for C++ that embraces integration with other libraries (for I/O, concurrency, etc)
Thrift (Apache not Facebook) and CapnProto are the only other contenders that I know of.
Results for 2 and 3 CPUs are strange too. So little scaling, what's the point...