Just looking at some .NET programs, they all use unsafe code, which is always faster than using garbage collected memory management, so the results aren't surprising.
It's using features of the language. Is that not what this is meant to measure/compare? .NET has unsafe, SIMD, and value types which opens it to a whole world of programming that Java is currently unsuited for.
One would think that a benchmark described in reference to the performance of the virtual machines themselves, would be measuring the performance of pure bytecode (or a JITing thereof), not the performance of native code linked into the runtime.
Using a native regex extension lib is the norm for "scripting" languages so totally reasonable to benchmark like that. How do you define when it's OK to use a native extension and when it's cheating?
If you want performance, you need to refactor your code into a small performance-critical unsafe core (that has higher test coverage and audit standards), while keeping the rest of the business logic in the safe domain, and relying on the CLR to do the heavy lifting and guarantee memory consistency.
Bonus point: you can write the core in C++ and use COM-like marshaling for absolutely unbeatable performance (no actual COM runtime is needed).
And the fastest implementations aren't the fastest either, I was able to knock about 10% off one of the Java examples from 2010. I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow.
As others have pointed out, if you want to test C#'s ability to call C-functions in an unsafe way, then fair enough. Otherwise, with memory safety enabled, they are roughly comparable.
That said, I did port the C# version to Java and it added about 20% extra time, even with some of the cheats (algorithm optimizations) it has implemented.
But (without intending to spread unsubstantiated rumor) it seems like it's temporarily unmaintained at the moment:
- in the README.md there is some complaint about being difficult to reboot into a Linux dual-boot, maybe the measurements+website were being published manually
That issue appears to be someone who was upset with the methodology of the benchmarks and the delay in accepting his/her programs so they resorted to repeatedly changing the title of one their issues.
Only a few seems to be plain code vs plain code. For the C# code there's use of unsafe, SSE/AVX intrinsics and calling C code to do the job (PCRE for the regex).
Certainly it's nice that you have these capabilities in C#, but at least for me it limits the usefulness of the results.
Handicapping one language to the featureset of another is unreasonable. You'd also have to disable a host of JVM features (such as hotspot) for a "fair" comparison, which is ridiculous.
Calling C is unfair, though. JVM can do the same and it should be implemented the same way.
> Calling C is unfair, though. JVM can do the same and it should be implemented the same way.
If you want to benchmark C code you can just benchmark a C application. You run stuff in a VM like the JVM or CLR because you want things like memory safety. If you allow native code in something like a regex benchmark then all your benchmarks are basically about: How fast can the JVM/CLR call native code. And say absolutely nothing about how fast the VM itself is.
So using native code in these benchmarks is just silly and basically makes the whole thing meaningless.
How fast your managed code can call native code is an extremely relevant performance detail sometimes. It's far from the only thing you should benchmark, but it is a useful thing to benchmark.
Yes, C# is currently a more expressive language for performance than Java is, and can score much higher with tighter code. I wouldn't trust the opinion of someone who claims otherwise by ignoring the facts (as demonstrated by your link).
The specific benchmarks in the OP are not objective and fair. I personally want nothing to do with Java, its glacial progress and Oracle's lawyers. But fair is fair.
Using facts to prove a point (as you have) means that quality facts are needed. The OP is not quality facts.
This reminds me of the canonical scheme benchmarks: https://ecraven.github.io/r7rs-benchmarks/ . I don't think comparing schemes that guarantees safety to schemes that will happily run (car "hello") is very meaningful.
It is of course nice to be able to opt out of safety for critical code, but I don't think most code should run in a way where a type error might corrupt memory. Especially not in a dynamic language.
.NET Team member here. We don't look at these specific benchmarks. Our belief is that they are wild west w/limited moderation of what is reasonable/real-world and not. Not necessarily a reasonable comparison, but we focus on TechEmpower[1]. You can check out the latest .NET performance improvements in our latest blog post on the topic [2].
I am thoroughly impressed with you what your team has accomplished. Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject.
It's been exciting to watch the hard work you and your team are investing to make .NET (Core/Framework/etc) faster. In fact, virtually every single issue or point I raised in my blog post has been addressed to my sanctification and then some. The Techempower benchmarks show the massive strides you've all made and now I'm delighted to see .NET consistently performing at or near the top of the benchmark even without heavy customization/modification. To this day, I still really love C# as a language and there are a number of significant things in C# that I wish the Go ecosystem would adopt. If/when bootstrap a technology stack again, you can bet that C#/.NET will be the top contender on the list of options.
Thanks again for all of your effort. It is very much appreciated and admired, even if I'm not using that tech stack at this point.
> Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject.
After reading this quote, and as a former .NET developer, I was really interested in the performance issues/story your blog would go in depth about. Instead, I found it strange, as I skimmed your blog, that I didn't see any specific scenario or perf numbers/bottlenecks discussed (seeing as you specifically state your company moved away from .NET due to IIS perf issues) - it was very generalized/hand-wavey. Do you happen to remember anything specific? Or were the reasons your company moved away from .NET literally the contents of the blog post?
It seemed to me that your frustration was more surrounding the MS bloated tooling &/or windows.
Some quotes from your blog:
"Once I used TFS on a small job and it was a monster--just like everything else from Redmond. It infected my project files and contaminated my source directories. Yuck. No thank you. Give me command line git any day...or perhaps SourceTree if you need a little love from a GUI."
You could definitely have used git back then.
"As a .NET developer, when you start a new project there are a few things you'll typically do (by hand, of course):
- Create a new solution
- Add it to Team Foundation Server
- Create a corresponding website entry in IIS
- Create a new SQL Server database
- Associate the Entity Framework with the solution (typically for projects created after 2010)
Start designing your database and your ActiveRecord entities."
You absolutely did not have to use EF or TFS, even back in 2014.
The post ends with:
"And that's why I left the .NET Framework because it kept reasserting itself and wanting to be more than it was: an implementation detail."
> It's usually the company policy that use TFS for C#, not developers.
I agree, but remember we are specifically talking about a company that switched it's whole business off .NET. A company that makes such a decision is probably one that could also make the call to switch version control systems without changing stacks.
In my blog post from 2014 I was mostly illustrating a typical workflow that I've seen over and over and over at numerous Microsoft shops in 2000s all the way through 2014. I was always interested in messaging queuing systems and NoSQL and memcached but when I'd interact with devs from various organizations the answer was "We use SQL Server." I was intrigued because the use cases they were bending SQL Server to handle were more akin to map reduce kinds of scenarios. In other words, the mantra of 10+ years ago was "stay fully in the Microsoft ecosystem, it'll solve all your problems."
Also at that point as we started to use EC2 heavily and the workflow was so straightforward for Linux. The AWS docs would always have caveats for Windows, e.g. "This is how it works...but in Windows you also have to do X and you can't do Y." So that's why we jumped ship.
As mentioned in my earlier comment, I think they've completely tackled all of the perceived shortcomings to the point where the .NET stack is highly attractive and very competitive compared to other stacks.
Everyone just games these benchmarks by calling out to C libraries. It's better to actually compare the slowest programs for each language instead of the fastest ones, because those are usually idiomatic.
Seeing the debian.net domain name, I'm a bit saddened to be reminded that DotNet Core is open source, yet still not available on Debian without installing packages from external sources (like the deb packages MS provides):
Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Further, it installs a custom package repo and in the past I've had very negative experiences with PPAs so I tend to avoid them.
Because people in the SV/HN bubble hate Microsoft. Purely for cultural reasons of course (while running Facebook's Pytorch, Google's browser, Apple's laptop, etc.).
I think you’ll find plenty of complaints with Facebook, Google, and Apple mixed into their relevant conversations. In fact, I’m pretty sure many people on here would have issues installing a key from Facebook too.
Admittedly though, it’s hard to avoid a Google or Apple key, if you’re using an OS from them...
Personally, I'm sort of a purist so while I use Google's Android OS, I use the Lineage OS distro without gapps, which means I'm one of the few Android users without Google's keys (at least for the gapps).
And yeah, I'd have issues with installing pytorch too. I tried it once, but then gave up because I couldn't ensure it wouldn't mess up my system and I mainly wanted to try out some ML stuff on Github.
Not in SV. Farthest thing from it. I'm an old timer on this site, and I'll never trust MS. They got their lunch eaten and pretend to be open source friendly. They are not.
There is a difference between hate and "don't trust". MS has a long history of abusing open source, why should they trust them now? Why is MS always squeezing privacy abusing telemetry in their products and leaving it opt out via sometimes very obscure methods.
Because I don't want to trust Microsoft more than I have to. It'd be the only non-distro provided keys on my computer.
I would subject myself to them. Maybe in 5 years they decide that to download, you need a Microsoft account login?
This weakens one of the core strengths of GNU/Linux, that every software installation is only an apt get away. Also, Debian packaging does a level of vetting that the software complies by their packaging policies, which ensures that it is truly libre.
I know that Microsoft has their UEFI signing keys in all the IBM computers around the world and has signed the desktop Linux bootloaders as well, and I respect that they do this, but that's a different part of the stack. I understand that some entity needs to sign the bootloaders, and Microsoft has since shown itself to be a reasonable guardian.
.NET core was slower than mono which was slower than JVM for running real time multiplayer games (3d math, serialization, reading and writing from multiple streams) as of a couple of years ago, did something big change With the runtimes very recently or is this just coding to benchmarks?
Maybe I missed something, but there doesn't appear to be a benchmark between JNI, JNA, and pInvoke. Most my experience with .NET/Java has shown it is fast enough in most cases, but then run into areas that need to be optimized into Native Code. This calling of Native "C" code can be slow with JNI and generally just fast enough for .NET pInvoke. I haven't ever had a reason to test JNA.
84 comments
[ 2.3 ms ] story [ 71.5 ms ] threadhttps://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Bonus point: you can write the core in C++ and use COM-like marshaling for absolutely unbeatable performance (no actual COM runtime is needed).
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
It seems they have people submit solutions to the problem and compare the fastest implementation for each.
As others have pointed out, if you want to test C#'s ability to call C-functions in an unsafe way, then fair enough. Otherwise, with memory safety enabled, they are roughly comparable.
That said, I did port the C# version to Java and it added about 20% extra time, even with some of the cheats (algorithm optimizations) it has implemented.
One person became curious and was brave enough to contribute their programs.
But (without intending to spread unsubstantiated rumor) it seems like it's temporarily unmaintained at the moment:
- in the README.md there is some complaint about being difficult to reboot into a Linux dual-boot, maybe the measurements+website were being published manually
- CI yml seems incomplete
- Some very strange issues like https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
So don't :-)
> … difficult to reboot into a Linux dual-boot…
That's likely to be a problem soon after 1st September.
> CI yml seems incomplete
In what way? "This GitLab CI configuration is valid."
> Some very strange issues like…
What else specifically?
"It" is the benchmarks game and "they" are the maintainers of the benchmarks game.
Certainly it's nice that you have these capabilities in C#, but at least for me it limits the usefulness of the results.
Calling C is unfair, though. JVM can do the same and it should be implemented the same way.
A JVM is not Hotspot, and Hotspot is not a part of “the” JVM.
https://en.wikipedia.org/wiki/HotSpot
If you want to benchmark C code you can just benchmark a C application. You run stuff in a VM like the JVM or CLR because you want things like memory safety. If you allow native code in something like a regex benchmark then all your benchmarks are basically about: How fast can the JVM/CLR call native code. And say absolutely nothing about how fast the VM itself is.
So using native code in these benchmarks is just silly and basically makes the whole thing meaningless.
Same with intrinsics. Have a benchmark that tests how good are the intrinsics, are there any significant overhead etc.
When I look at generic names like "n-body sim" I'm thinking ok, let's see how generic mixed floating-point code compares between the languages.
Note the `C# .NET Core #4` and `Java #2` programs.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
The specific benchmarks in the OP are not objective and fair. I personally want nothing to do with Java, its glacial progress and Oracle's lawyers. But fair is fair.
Using facts to prove a point (as you have) means that quality facts are needed. The OP is not quality facts.
It is of course nice to be able to opt out of safety for critical code, but I don't think most code should run in a way where a type error might corrupt memory. Especially not in a dynamic language.
http://www.larcenists.org/Twobit/bmcrock.temp.html
"all C# .NET Core programs & measurements"
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
[1] https://www.techempower.com/benchmarks/
[2] https://devblogs.microsoft.com/dotnet/performance-improvemen...
It's been exciting to watch the hard work you and your team are investing to make .NET (Core/Framework/etc) faster. In fact, virtually every single issue or point I raised in my blog post has been addressed to my sanctification and then some. The Techempower benchmarks show the massive strides you've all made and now I'm delighted to see .NET consistently performing at or near the top of the benchmark even without heavy customization/modification. To this day, I still really love C# as a language and there are a number of significant things in C# that I wish the Go ecosystem would adopt. If/when bootstrap a technology stack again, you can bet that C#/.NET will be the top contender on the list of options.
Thanks again for all of your effort. It is very much appreciated and admired, even if I'm not using that tech stack at this point.
[1] https://blog.jonathanoliver.com/why-i-left-dot-net/
Go being only faster in one benchmark
After reading this quote, and as a former .NET developer, I was really interested in the performance issues/story your blog would go in depth about. Instead, I found it strange, as I skimmed your blog, that I didn't see any specific scenario or perf numbers/bottlenecks discussed (seeing as you specifically state your company moved away from .NET due to IIS perf issues) - it was very generalized/hand-wavey. Do you happen to remember anything specific? Or were the reasons your company moved away from .NET literally the contents of the blog post?
It seemed to me that your frustration was more surrounding the MS bloated tooling &/or windows.
Some quotes from your blog:
"Once I used TFS on a small job and it was a monster--just like everything else from Redmond. It infected my project files and contaminated my source directories. Yuck. No thank you. Give me command line git any day...or perhaps SourceTree if you need a little love from a GUI."
You could definitely have used git back then.
"As a .NET developer, when you start a new project there are a few things you'll typically do (by hand, of course):
You absolutely did not have to use EF or TFS, even back in 2014.The post ends with:
"And that's why I left the .NET Framework because it kept reasserting itself and wanting to be more than it was: an implementation detail."
I agree, but remember we are specifically talking about a company that switched it's whole business off .NET. A company that makes such a decision is probably one that could also make the call to switch version control systems without changing stacks.
Also at that point as we started to use EC2 heavily and the workflow was so straightforward for Linux. The AWS docs would always have caveats for Windows, e.g. "This is how it works...but in Windows you also have to do X and you can't do Y." So that's why we jumped ship.
As mentioned in my earlier comment, I think they've completely tackled all of the perceived shortcomings to the point where the .NET stack is highly attractive and very competitive compared to other stacks.
Really?
Please say which of the n-body programs do that.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Meanwhile, sort by program source code size —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969
Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Further, it installs a custom package repo and in the past I've had very negative experiences with PPAs so I tend to avoid them.
https://docs.microsoft.com/en-us/dotnet/core/install/linux-d...
Why?
Admittedly though, it’s hard to avoid a Google or Apple key, if you’re using an OS from them...
And yeah, I'd have issues with installing pytorch too. I tried it once, but then gave up because I couldn't ensure it wouldn't mess up my system and I mainly wanted to try out some ML stuff on Github.
I would subject myself to them. Maybe in 5 years they decide that to download, you need a Microsoft account login?
This weakens one of the core strengths of GNU/Linux, that every software installation is only an apt get away. Also, Debian packaging does a level of vetting that the software complies by their packaging policies, which ensures that it is truly libre.
I know that Microsoft has their UEFI signing keys in all the IBM computers around the world and has signed the desktop Linux bootloaders as well, and I respect that they do this, but that's a different part of the stack. I understand that some entity needs to sign the bootloaders, and Microsoft has since shown itself to be a reasonable guardian.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...