I was surprised to see that Java was slower than C++, but the Java code is run with `-XX:+UseSerialGC`, which is the slowest GC, meant to be used only on very small systems, and to optimise for memory footprint more than performance. Also, there's no heap size, which means it's hard to know what exactly is being measured. Java allows trading off CPU for RAM and vice-versa. It would be meaningful if an appropriate GC were used (Parallel, for this batch job) and with different heap sizes. If the rules say the program should take less than 8GB of RAM, then it's best to configure the heap to 8GB (or a little lower). Also, System.gc() shouldn't be invoked.
Don't know if that would make a difference, but that's how I'd run it, because in Java, the heap/GC configuration is an important part of the program and how it's actually executed.
Of course, the most recent JDK version should be used (I guess the most recent compiler version for all languages).
(Given credits to both sources in the description of this repo)
(Also fair disclosure but it was generated just out of curiosity of how this benchmark data might look if it was on benjdd's ui and I used LLM's for this use case for prototyping purposes. The result looks pretty simiar imo for visualization so full credits to benjdd's awesome visualization, I just wanted this to be in that to see for myself but ended up having it open source/on github pages)
I think benjdd's on hackernews too so hi ben! Your websites really cool!
D gets no respect. It's a solid language with a lot of great features and conveniences compared to C++ but it barely gets a passing mention (if that) when language discussions pop up. I'd argue a lot of the problems people have with C++ are addressed with D but they have no idea.
Tiny community, even more tinier than when Andrei Alexandrescu published the D book (he is now back to C++ at NVidia), lack of direction (it is always trying the next big thing that might atract users, leaving others behind not fully done), since 2010 other alternatives with big corp sponsoring came up, others like Java and C# gained the AOT and improved their low level programing capabilities.
Thus, it makes very little sense to adopt D versus other managed compiled languages.
The language and community are cool, sadly that is not enough.
C# is very fast (see multicore rating). Implementation based on simd (vector), memory spans, stackalloc, source generators and what have you — modern C# allows you go very low-level and very fast.
Probably even faster under .net 10.
Though using stopwatch for benchmark is killing me :-) Wonder if multiple runs via benchmarkdotnet would show better times (also due to jit optimizations). For example, Java code had more warm-up iterations before measuring
This entire benchmark is frankly a joke. As other commenters have pointed out, the compiler flags make no sense, they use pretty egregious ways to measure performance, and ancient versions are being used across the board. Worst of all, the code quality in each sample is extremely variable and some are _really_ bad.
Totally agree. I found the results surprising because a bunch of languages are faster than C++. Then I looked closer. The requirements are self-conflicting, No SIMD, but must be production-ready. No one would use the unoptimized version in production. Also looking at the C++ implementation, they are not optimized at all. This makes this benchmark literally pointless.
Genuine question: Are GitHub workflows stable enough to be used for benchmarking? Like CPU time quantum scheduling is guaranteed to be the same from run to run?
I was very surprised to see the results for common lisp. As I scrolled down I just figured that the language was not included until I saw it down there. I would have guessed SBCL to be much faster. I checked it out locally and got: Rust 9ms, D: 16ms, and CL: 80ms.
Looking at the implementation, only adding type annotations, there was a ~10% improvement. Then the tag-map using vectors as values which is more appropriate than lists (imo) gave a 40% improvement over the initial version. By additionally cutting a few allocations, the total time is halved. I'm guessing other languages will have similar easy improvements.
I wrote a script (now an app basically haha) to migrate data from EMR #1 to EMR #2 and I chose Nim because it feels like Python but it's fast as hell. Claude Code did a fine job understanding and writing Nim especially when I gave it more explicit instructions in the system prompt.
The fact that Julia “highly optimized” is 30x faster than the normal Julia implementation, yet still fails to reach for some pretty obvious optimizations, and uses a joke package called “SuperDataStructures” tells me that maybe this benchmark shouldn’t be taken all that seriously.
Benchmarks like this can still be fun and informative
The study seems to be “solve this the obvious way, don’t think too hard about it”. Then the systems languages (C, Zig, C++) are pretty close, the GC languages are around an order of magnitude slower (C#, Java
doing pretty good at ca. 3x), and the scripting languages around two orders of magnitude slower.
But note the HO-variants: with better algorithms, you can shave off two orders of magnitude.
So if you’re open to thinking a bit harder about the problem, maybe your badly benchmarking language is just fine after all.
That doesn’t require the strings that represent the tags to be the tag strings, So, one can bend the rules by representing tags by single-character strings or, alternatively, by using fixed strings of length 0 through 99, and then doing the tag comparisons only on the first character of each string or, alternatively, the length of the string (if obtaining that is fast)
Especially when tags have large common prefixes, that could speed up things tremendously.
If people don't find their preferred language on top, they will claim the benchmark is flawed. They will find a condition that is not satisfied by the benchmark. But if we operate outside of the benchmarks assumptions, all benchmarks are flawed since they cannot satisfy all possible conditions.
29 comments
[ 3.3 ms ] story [ 54.1 ms ] threadDon't know if that would make a difference, but that's how I'd run it, because in Java, the heap/GC configuration is an important part of the program and how it's actually executed.
Of course, the most recent JDK version should be used (I guess the most recent compiler version for all languages).
Nowadays whenever I see benchmarks of different languages. I really compare it to benjdd.com/languages or benjdd.com/languages2
Ended up creating a visualization of this data if anybody's interested
https://serjaimelannister.github.io/data-processing-benchmar...
(Given credits to both sources in the description of this repo)
(Also fair disclosure but it was generated just out of curiosity of how this benchmark data might look if it was on benjdd's ui and I used LLM's for this use case for prototyping purposes. The result looks pretty simiar imo for visualization so full credits to benjdd's awesome visualization, I just wanted this to be in that to see for myself but ended up having it open source/on github pages)
I think benjdd's on hackernews too so hi ben! Your websites really cool!
Thus, it makes very little sense to adopt D versus other managed compiled languages.
The language and community are cool, sadly that is not enough.
Probably even faster under .net 10.
Though using stopwatch for benchmark is killing me :-) Wonder if multiple runs via benchmarkdotnet would show better times (also due to jit optimizations). For example, Java code had more warm-up iterations before measuring
Looking at the implementation, only adding type annotations, there was a ~10% improvement. Then the tag-map using vectors as values which is more appropriate than lists (imo) gave a 40% improvement over the initial version. By additionally cutting a few allocations, the total time is halved. I'm guessing other languages will have similar easy improvements.
Benchmarks like this can still be fun and informative
https://niklas-heer.github.io/speed-comparison
Certainly does "look" very interesting.
But note the HO-variants: with better algorithms, you can shave off two orders of magnitude.
So if you’re open to thinking a bit harder about the problem, maybe your badly benchmarking language is just fine after all.
https://news.ycombinator.com/item?id=37848571
? unchanged from 7 months ago
> MUST
> Support up to 100 tags
> Represent tags as strings
That doesn’t require the strings that represent the tags to be the tag strings, So, one can bend the rules by representing tags by single-character strings or, alternatively, by using fixed strings of length 0 through 99, and then doing the tag comparisons only on the first character of each string or, alternatively, the length of the string (if obtaining that is fast)
Especially when tags have large common prefixes, that could speed up things tremendously.
In languages that support string interning (https://en.wikipedia.org/wiki/String_interning), I suspect that also could be used to bend the rules.