Quite true, although my (possibly naive) assumption was that with it being a statically typed language and these being "simple" algorithmic benchmarks the gap would be closer.
Not expecting it to match Rust or C in performance, just surprised at the size of the gap.
Would be interesting to see what the generated C is like from the Nim code.
If you don't use automatically managed types like seq/string and ref, there is no GC underneath. And even then, the GC is just reference counting so no difference from Rust Rc/Arc.
C is an easy intermediary, C is a bad intermediary.
The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable).
Rust and C both end up faster than Nim because they are talking directly to the compiler.
IDK enough about Nim to say if it could hit Rust/C speeds if it targeted the LLVM instead, but it would be a lot faster than it currently is targeting C.
> C is an easy intermediary, C is a bad intermediary.
> The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable).
The same issue exists in C-sourced compiler backends, which is both GCC and LLVM. It’s not exactly a secret that anything which is not strongly exercised by standard C(++) codebases tends to be rather broken for any non-trivial use and takes years to fix (restrict/noalias being the poster child for this, at least in the context of Rust, I think it hasn’t gotten re-disabled yet so it might make a year).
I think that has less to do with LLVM/GCC being C(++) implementations and more to do with the fact that the primary language compiled by these compilers is C and C++. Other languages that target these compilers aren't used nearly to the same level (maybe swift now?).
The JVM, for example, doesn't see the same issue so much with non-Java languages. Mainly because (IMO) it's been hosting non-java languages for several years now.
I'm not sure. The following claim is being made on Nim's website:
> Modern concepts like zero-overhead iterators and compile-time evaluation of user-defined functions, in combination with the preference of value-based datatypes allocated on the stack, lead to extremely performant code.
> Efficient, expressive, elegant
> Nim is a statically typed compiled systems programming language
A systems programming language need to have exceptionally good performance, so not sure it's to be expected that the performance is so different than Rust.
Apparently the Rust versions have received more optimization efforts compared to the Nim code as well, and it seems some people are working on optimizing the Nim versions so we'll see where it lands in the future I guess: https://news.ycombinator.com/item?id=30244188
I've been programming for like 30 years and I've seen dozens of language show up claiming "C like performance" and so far Rust is the only one where that has been true.
Indeed, probably the most overused selling pitch for programming languages has to be "C like performance", while only providing evidence for a few smaller use cases. I've only dabbled with Rust enough to grok the ownership parts, but I'm happy to hear you feel it lives up to that.
As an aside, I'm not expecting a lot of languages to live up to the "C like performance" claim when they make it, as C has been receiving optimizations for over 50 years by now, so it makes sense it's still the fastest around, at least until there is some major breakthrough in programming language design.
No. By that logic, all languages should have equivalent performance because they all ultimately turn into machine code.
C isn't inherently fast, commonly used constructs in C just tend to match up well with semantics that can be turned into high-performance machine code. For a language that compiles to C to be fast, its commonly used constructs would need to have semantics that match up well with what can be turned into high-performance C, which is often not the case. There are many examples of languages that compile to C, but are still very slow.
I would argue that C is inheritance fast. C puts the least amount of shit between the code and the CPU. There is no runtime, GC, expensive abstractions, etc. Your code more or less directly becomes machine code and is executed.
Maybe a better way to frame it is that most other languages are inherently slow. Garabge collection will always be slower than non-GC. JIT and interpreted languages will always be slower than ahead-of-time compiled languages. etc etc.
The point is that C isn't inherently fast ie: by simple virtue of translating a program from one language to C you are not producing a faster program. x86 has the least amount of shit between the code and the cpu, but we can see that many programs that compile to x86 are not fast.
That is a fair point, and looking back I misread the context of the comment I replied to. You should not expect free performance gains from transpiling to C.
Maybe I should have put a disclaimer in my comment to stave off this kind of reply.
I am well aware of the CRT. I don't think its worth mentioning because it is a
"run time" in name only. It literally doesn't do anything at run time. The only thing crt0 does is call your main(). From that point on, only your actual code is executing.
Contrast with Java or Python, where your code is executing inside a runtime which eats up cpu cycles at runtime.
I'm confused, if the CRT doesn't do anything at runtime except to call main(), how does argv get onto your program's stack? :P
Look, I get what you're saying, but it's not exactly relevant here. Yes, after your C program is loaded, execution follows by just moving the PC to the next instruction in the loaded binary. In C, adding two integers can be compiled down to a concise, machine supported operation; whereas in a language like Python, adding two numbers is a branch-heavy operation that cannot be compiled to a concise instruction ahead of time, so it must be interpreted at runtime. That's true. But that's really more about C being a compiled language than whether or not it has a runtime.
For an example of a runtime-heavy language that can compile highly performant code, see Julia. Thanks to dynamic dispatch, you get the best of both worlds. Yes you pay a heavy upfront cost (TTFP), but it's the same deal with compiling code -- pay once upfront, benefit forever with improved runtime performance.
You used Java as an example of a runtime-heavy language, but I think Java demonstrates why C should be considered as having a runtime, regardless of how small it is. Imagine a hardware Java Processing Unit (JPU), which is a CPU that has Java byte code for machine code [1]. I don't see why it wouldn't be possible to handle all of the stuff the JRE does in the OS, and everything the JVM does in the JPU. Then we could say that Java has no runtime. But really all we've done is taken the runtime and moved it to another place.
You could also imagine a version of C which compiles to CBC (C byte code) and runs on a CVM (C virtual machine), which itself runs on the JPU. You could write C code as usual, but the resulting binary would be compiled to CBC, and would have to run through a number of layers before it is executed on the JPU. Then could we say Java is an inherently fast language, and C is necessarily slower because it has a runtime? Maybe in this strange reality, but it's not necessarily the case. And it certainly has nothing to do with the languages per se, but their implementations on specific hardware/software platforms.
Or to get even crazier with it, we could imagine a version of Java which compiles directly to x86 machine code, because why not? There's no reason that should be slower than C.
Anyway, I think the point is that whether or not one considers a language to have a runtime depends very much on how a language is compiled, what hardware it's targeting, and how much you're able to leverage the OS (there's usually a lot of leverage available when the OS itself is written in the language you're using). So really, all languages have a runtime, it's just a matter of where it's hiding.
> So really, all languages have a runtime, it's just a matter of where it's hiding.
Your definitions are meaningless. You may as well also say that all languages are garbage collected, since the C programmer must call `free` one way or another (or not, and let the OS be the garbage collector at cleanup).
By everyone's understanding of the words, C does not have a runtime nor garbage collection. Java does have a runtime and garbage collection. You can play with definitions and hypotheticals; it does not change the facts.
> By everyone's understanding of the words, C does not have a runtime
What? But we both just agreed it did.
The way I understand these words are that C is a language, and a language is different from its implementation, and implementation is dependent on hardware. I understand that a runtime is code that runs when a program starts, in order to enable its execution. Is this not your understanding?
Let me put it this way: if C is inherently fast as you, why can you take the same C source, put it into two different compilers, and get two different binaries with different performance profiles?
The answer is because the language itself has no bearing on speed, and it all comes down to how a language is compiled. So I still wouldn't say C is inherently fast, but I would say that C's design lends itself to compiling fast binaries for Von Neumann style machines. That of course doesn't preclude compiling just-as-fast or even faster binaries from other languages, e.g. Rust or C++.
> You can play with definitions and hypotheticals; it does not change the facts.
I'm not playing with hypotheticals. JPUs are a thing; I linked to the Wiki page. Natively compiled Java is basically GraalVM: https://www.graalvm.org, which I think pretty much proves the point -- it all comes down to the compiler. Garbage checking is also an implementation detail. You can even run a garbage collector for C if you want.
I have to back up ModernMech here. C with -O0 is not very fast on any compiler that has it (by design) which is most. Nor is the output of barely optimizing compilers like tinycc/tcc. Saying "C is inherently fast" is very naive.
Moreover, it is also untrue on the other side of this axis - you can often go much faster. Even with -Ofast or -Omax or whatever there are still (often) opportunities for SIMD vectorization on modern machines that autovec cannot find. Most C compilers provide (not really standardized) escape hatches to assembly in various ways for that last 2..64X speed-up (single thread). (Yes, avx512/8-bits=64 wide vectors these days.)
Finally, whether it "has a runtime" in the sense of automatic memory management, well, that just depends upon how you link programs. Link with -lboehm and you can never free (and, yes, allocations become potentially expensive). These Rust conversations are unfortunately prone to absolutist statements that indicate very narrow exposure.
It's not inherently fast. It puts up few barriers that prevent the user from writing a fast program, but it's still on the user to write a program that is actually fast.
If your language has semantics that are difficult to translate into fast C, then your language isn't going to be fast just by going through C.
I wouldn't have down voted this comment, but I guess people are objecting because using C as the backend doesn't really tell us anything about performance. If the C code the Nim compiler generates is inefficient, the resulting program will still be slow. I imagine it is difficult to convert a Nim program into efficient C.
E.g. the Json interface of the Nim stdlib is not the quickest, but there are some third party libraries, 100% Nim, really fast (jsony). A look a both reveals that the problem is not the Nim compiler, but the algorithm used and intended usage.
I tried to qualify it "should put it in a good position" but I guess I wasn't handwavy enough. Indeed, the C code could be inefficient. But, for example, nim uses a C backend unlike Python which interprets bytecode.
It could, but at the moment it puts a garbage collector in there, though there are modes where you can turn that off. I haven't used Nim in a while so I don't know what the implications of disabling GC are at the moment.
Languages like Nim/Rust/D/etc. should not have significantly different speeds for equivalent implementations for these kinds of benchmarks, as they ultimately can be written to compile to equivalent LLVM IR and thus the same machine code.
What you're usually seeing when you see such large differences is that there are differences in the implementation of the algorithm or of a standard library function.
There's nothing about these kinds of languages that should result in inherently significant performance differences on such synthetic microbenchmarks.
A runtime isn't some sort of magic thing that inserts itself into hotspots. Several of these benchmarks are just looping over pointerfree arrays for almost all of their runtime and still show differences. Unless you call parts of the runtime, implicitly or explicitly, it shouldn't result in overhead.
Allocation-heavy or write-barrier-heavy code might make a difference, but you literally have performance differences while iterating over arrays of scalars. And even then, last I benchmarked it, Nim actually compared favorably with e.g. jemalloc.
It depends on the runtime. In the case of Go e.g. the compiler inserts potential "yield"-s at every function call[1].
And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not.
So it all depends on the language.
Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise.
> It depends on the runtime. In the case of Go e.g. the compiler inserts potential "yield"-s at every function call[1].
> And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not.
Yes, but Nim does neither. Nim does insert write barriers, but that's only if you (1) write pointers to heap-allocated memory and (2) don't use the mark-and-sweep GC, so that can't explain big performance differences for those benchmarks where you primarily iterate over arrays.
> Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise.
You don't seem to be aware of it, but D actually has an LLVM-based compiler (LDC). Nim can also utilize LLVM via choosing clang as its backend. You can definitely use them to generate equivalent LLVM IR and compare that. There's nothing weird here, this allows us to rule out differences related to the backend.
> There's nothing about these kinds of languages that should result in inherently significant performance differences on such synthetic microbenchmarks.
I guess it depends on what significant is but I certainly hope Rust can be faster than C in many scenarios due to optimizations llvm can make with the added type information such as inferring aliasing rules.
From my understanding, LLVM had or has many bugs around alias rules. I remember Rust emitted proper noalias IR to LLVM, only for it to be promptly ignored.
"compile to equivalent LLVM IR". This is not true at all, not in practice and not in theory so I don't understand why it is popping up constantly. Fortran was for example faster than C in some cases before C99 added the restrict-keyword, forever changing the language.
As we discussed on the Nim forum¹, the Nim programs could definitely be improved a lot. elcritch did some optimization and managed to make Nim's time roughly equal to Rust.
That speedup is accomplished by using a third-party LRU implementation. I can also speed up the Rust version by 30% by plugging in https://lib.rs/crates/lru.
(The thread points out that the Nim version uses a doubly-linked list from the standard library while you'd need unsafe in Rust, and the lru crate does indeed use unsafe. So take that into account.)
Unsafe is part of the rust language. Theres nothing wrong with using it if you know what you’re doing. The value of unsafe is that it boxes your unsafe code - so you can direct extra care to those functions when writing, reviewing and auditing. (As opposed to C where memory corruption could happen anywhere, so you need to be careful everywhere.)
I could imagine a separate language on the programming languages shootout which is “safe rust”. It’d be interesting how the best unsafe and the best safe programs compare in speed and size.
If you believe Rust zero cost abstraction - being safe doesn't mean it has to be compared to safe languages because Rust and C can translate into same assembler code. But in Rust the compiler gives you more guarantees.
I would say that a proper way to rephrase it would be to use idiomatic code vs unidiomatic. If you start using any high-level language in a non intended fashion (writing non idiomatic code) probably most of them would be faster.
Note that using the third party LRU makes the Nim version faster than the Rust one by 15%. Fixing the Deque/Table Nim version results in the same times as the Rust Dec/HashMap one.
I think people may not know or may not remember that ~6 years ago or so there was a fairly significant effort to increase rust's position in the benchmarks. I remember it coming up a lot on /r/rust when the language had stabilized. "C or C++ is first place? Isn't Rust supposed to be fast?" etc.
I assume nim has gotten a lot less attention in these benchmarks and there's probably huge wins available to it.
Is this a language benchmark or a programmer benchmark? Usually these differences come down to the specific implementation more often than the language used.
Those are related tho! If I had a tough problem to solve, I would tend to give it to the "average Rust" rather than the "average PHP" programmer. Language difficulty and the surrounding culture act as a filter.
Since much of the implementation is in available libraries, this is a comparison of those too, which is an important consideration. There are language features that slow things down too. The benchmark game makes it clear that no language with GC gets closer than a 2-3x the time of a non-GC language (julia is the closest) and that matches most real world experience (and also that we don't always care about that hit). It also makes it clear that being dynamically typed and interpreted is a huge penalty in performance. I.e. python. But again, development speed can be more important than that throughput hit. Benchmarks can help us decide which tools we like for which tasks.
You know I did see that the other day and forgot! C# is quite impressive in general. The intrinsic support really helped in that one. Thanks for keeping the thing going. Its cool to see someone follow your advice and benchmark a language not on the site.
> no language with GC gets closer than a 2-3x the time of a non-GC language
This is wrong. At least, it is not true generally. I wrote a 1:1 port of my C++ chess engine, and after a bit of fiddling with compiler options the Nim version was faster. In fact, when using a delayed garbage collector, the program was even faster than when using reference counting.
> The benchmark game makes it clear that no language with GC gets closer than a 2-3x the time of a non-GC language (julia is the closest) and that matches most real world experience (and also that we don't always care about that hit). It also makes it clear that being dynamically typed and interpreted is a huge penalty in performance.
It makes it clear that some people have most time to optimize for a benchmark than others.
This benchmark on state-of-the-art cryptography shows that GC-ed languages including Go and Nim can be 2x faster than Rust:
Furthermore, Nim GC/memory management is type-dependent, only seq/string (which use malloc/free) and `ref T` type (which use refcounting) involves some runtime.
So Nim devs can choose to opt-in in GC or use manual memory management depending on their priorities.
Why does hello world take 1.8ms to run? That's ~5,000,000 cycles. By my measurements with Actually Portable Executable the latency of vfork+execve+exit+wait4 on a physical computer running Linux should be 25µs. That's two orders of a magnitude difference. https://github.com/jart/cosmopolitan/blob/master/test/libc/s...
Sure, put a "roughly" in there. It's a simple multiplication by clock speed. This is Justine who designed APE. I think they know.
Edit, and also no, cycles are not "more about instruction set". They're pulses of electricity that make their way through a chip, they are about as far from instruction-set-specific as it gets.
Very simple: CPUs do not execute a constant number of instructions on each cycle. Some instructions are really slow, and their slowness is measured in cycles. You can usually do one XOR in one cycle, because it's a really simple piece of circuitry. Some instructions have complicated circuitry, which means that it takes multiple pulses of electricity to move the input signals through all the logic gates inside. XOR can do it in one pulse, but a division might take 20 pulses. Cycles are just pulses.
One important component of a program's performance, especially in numeric code, is how many instructions it manages to execute per cycle. If you have written your program's instructions in a 'good' order, then the CPU will e.g. not queue up many divisions in a row that depend on each other. It can do two different things at the same time. You can't do anything in less than one cycle, but you can do two of the same thing in the same cycle if you have two available execution units for that task.
The most direct answer to your question is that different instruction sets have different ideas about how much work a single instruction should do. The basic divide is between CISC (complex instructions, x86 qualifies) and RISC (basic instructions, ARM). It is a dramatic oversimplification but one instruction in a CISC architecture might involve 10 instructions in a RISC. So you can't compare instruction counts between architectures. But you can compare cycles and still make a bit of sense.
If this is interesting to you, you should try this game, where you can build a CPU purely out of the most basic logic gates: https://nandgame.com/
2.6 Ghz * 1.8ms = 4,680,000 CPU cycles. To reiterate, this is independent of instructions. That's the number of CPU cycles it took to execute hello world.
I've rounded the number and added a tilde since you're right. There's actually a lot of interesting depth behind this subject for x86 alone. RDTSC timestamps are guaranteed to be invariant across all models irrespective of clock speed (except k8) so a mapping exists between RDTSC reported clock cycles and nanoseconds; however, it usually isn't accessible on client grade hardware and has to be measured haphazardly by the operating system. In my experience on CPUs the last ten years it's usually in the ballpark of .323018 for both Intel and AMD. Multiply ticks by that and you get a pretty good nanosecond approximation. If you want a fixed point expression for your benchmarks:
static inline unsigned long ClocksToNanos(unsigned long x, unsigned long y) {
// approximation of round(x*.323018) which is usually
// the ratio between inva rdtsc ticks and nanoseconds
unsigned long difference = y >= x ? y - x : ~x + y + 1;
return (difference * 338709) >> 20;
}
That's just one quick and dirty way your approximation could be approximated. Should work great for any benchmark games. Obviously don't use it for like, an X-ray or something. You can get the RDTSC value as follows:
static inline unsigned long Rdtsc(void) {
unsigned long Rax, Rdx;
asm volatile("rdtsc" : "=a"(Rax), "=d"(Rdx) : /* no inputs */ : "memory");
return Rdx << 32 | Rax;
}
Intel also has recommendations for things like using CPUID and memory fences if you need to reign in speculative execution, but they can be costly. If you want to go even deeper then RDTSCP has a nice feature that lets you know if the operating system switched you to a different core during your measurement interval because it gives you TSC_AUX.
I'm not sure whether any of that timing precision will do any good on GitHub Actions, where this seems to be running. I don't think the machine is going to be quiet enough to get a good measurement of anything.
Noise and jitter in the VM should cancel out with repeated measurements.
Here's something interesting. If I run my test in a RHEL5 (Linux c. 2010) VirtualBox VM then it actually goes faster than a modern Linux kernel running on bare metal, taking only 14µs. If I run it in a RHEL7 VM then it needs 131µs.
So >1ms hello world execution should be a red flag. If that's actually the latency of GitHub's kernels and can't be explained by something like a shell script doing the time interval measurements, then it'd be interesting to learn what it's doing.
I disable Spectre mitigations on my Linux workstation since it doesn't host VMs or browsers. I'm sure if those were enabled it'd go much slower. But either way it's pretty clear that a lot of stuff got added to the kernel since 2.6.18 that slows down process launches dramatically, then some of it probably got optimized, and the rest stuck. Would be interesting to know what it is.
It appears helloworld is the only test with any repeats, and it only has 5 repeats. https://github.com/hanabi1224/Programming-Language-Benchmark... (Also, you can't average out the VM scheduling something else during the test, if it is constantly scheduling other stuff as it is a very busy build machine.)
Here's the measurement code, it appears to be significantly more complicated than a simple fork/exec/wait loop but that could just be all the C# getting in the way: https://github.com/hanabi1224/Programming-Language-Benchmark... Note that we are definitely measuring the C# async runtime to some degree. Nevertheless you are probably right that the bulk of this 1.8ms is in the executable under test, and it truly is just bloat. Running `hyperfine ./empty-main-function` from rustc on my Mac gives 0.8ms.
I'm guessing from a bit of googling that APE stands for the Actually Portable Executable[0] (which once I found I realised I had read about before, but I wasn't aware of the acronym).
Cycles are a function of time and clock rate. Most CPUs run at 1-4 GHz, so multiplying seconds by a billion gives you a cycle estimate within a factor of 10.
the file you linked to is doing file I/O. that's much more than fork/exec. Hopefully on the benchmark system it's tmpfs but it still has to go through VFS layer.
By default I think rust links libc and produces a dynamically linked executable. If you run strace I bet you'll find it's doing a ton of syscalls related to searching for and loading dynamic assets.
OK? As I said, it produces static executables that dynamically link with libc. I would not call that a dynamic executable, especially when already specifying that libc is loaded dynamically. I don't think most people would classify "loads libc dynamically" as a "dynamic executable".
Gentlemen there's no need to argue because you're both right. Yes, Rust linkage is predominantly static. However in order to link libc dynamically on System V, it officially speaking needs to be a dynamic object rather than an executable, in most cases. I just checked on my system with a Rust binary to confirm. The Elf64_Ehdr::e_type is set to ET_DYN rather than ET_EXEC.
How would you time this? I tried a simple C hello world and got a similar ~2ms run time. But I was just running "time ./hello" at a Bash prompt, which a) only has 1ms resolution and b) probably measures some shell overhead. Running it under "strace -tt" so I could see where the slow bits were made it 10x slower :-(
Edit: FWIW, building with "gcc foo.c -o foo -O2 -s -Wall --static" was slightly faster than without the --static. And "musl-gcc foo.c -o foo -O2 -s -Wall --static" was slightly better again. Maybe 0.5ms improvement in total. I'm on Ubuntu 20 x64, gcc 9, Intel Xeon W-2133.
You could write a non-shell launcher using fork+exec+wait and present the results with finer granularity than time(1). There may be something adequate available off-the-shelf but I’m not familiar with it.
$ git clone https://github.com/jart/cosmopolitan && cd cosmopolitan
$ make -j8 o//examples/rusage.com o//examples/hello.com
$ o//examples/rusage.com o//examples/hello.com
hello world 123
RL: took 90µs wall time
RL: ballooned to 108kb in size
RL: needed 106µs cpu (0% kernel)
RL: caused 13 page faults (100% memcpy)
This one has a tiny bit more overhead than the practical minimum you saw earlier in the benchmark. Note that APE binaries start off as shell scripts so the first run is going to be slower. It's something I'm working towards improving in a variety of ways. Here's the executive summary of the upcoming changes:
OK, I did "for i in {1..10}; do o//examples/rusage.com o//examples/hello.com; done" and the lowest wall time I got was 1,140us, which is a lot more than your 90us for some reason. There's probably something wrong with my setup. Maybe it takes my machine that long to come out of some P-state or C-state sleep mode I don't understand.
Then for my musl-gcc hello world, I did, "for i in {1..10}; do o//examples/rusage.com ../hello" and the lowest wall time I got was 120us. But most of the runs were closer to 350us.
So, my attempt to answer your, "Why does hello world take 1.8ms to run?" question is: a) they timed it with "time" which measures lots of overhead, b) there's something wrong with their setup that means they are mostly timing the time taken for a CPU core to wake up.
Because Rust is not competing with the HQ9 language. It's for writing programs that do more than just Hello World, so the startup initializes common things for non-toy programs.
How can such a benchmark be in any way informative? How much am I allowed to optimize the nim code? How much am I allowed to optimize the Rust code? Am I allowed to just change nim's GC to ARC/ORC? I'm sorry but this just does not make any sense to me.
It is more informative than nothing. It shows different languages implementing the same functions and the performance of those. It is something to compare to. Maybe they can all be optimized, etc. But you need to have some kind of starting point. I think people tend to poo-poo benchmarks (unless their fav language at top) and claim they don't represent real world. But I always argue that they at the very minimum give you some context for comparison. If you look at how they are implemented you can get further context. I only wish the benchmarks had some metric for how idiomatic the implementation is compared to normal developers.
They do, and its caused them a lot of headache, probably was a mistake. If they would have at least used something like C#/Java/Go (go didn't exist at the time) they might not have had to do time-slowdowns to make large battles possible.
Then again maybe it would have slowed down development and never gotten off the ground at all!
Does anyone know why nim seems to spend so much time in the kernel? The most obvious example being that last benchmark for spectral-norm where nim spends almost 3 seconds in the kernel, and rust spends 0.
Firstly, not interested in gamification; secondly, if you can't tell the difference between the languages then I'm not interested in participating in the conversation.
125 comments
[ 3.1 ms ] story [ 235 ms ] threadhttps://programming-language-benchmarks.vercel.app/rust-vs-c
https://programming-language-benchmarks.vercel.app/nim-vs-c
C has nothing to do with performance, runtimes and libraries do. Nim has runtime overhead whereas C and Rust do not.
Not expecting it to match Rust or C in performance, just surprised at the size of the gap.
Would be interesting to see what the generated C is like from the Nim code.
Just did a few quick tests. A hello world in nim is a one liner:
It compiles to a 102136 byte binary on Linux. A hello world c program consisting of a single printf is 16696 bytes.If you don't use automatically managed types like seq/string and ref, there is no GC underneath. And even then, the GC is just reference counting so no difference from Rust Rc/Arc.
The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable).
Rust and C both end up faster than Nim because they are talking directly to the compiler.
IDK enough about Nim to say if it could hit Rust/C speeds if it targeted the LLVM instead, but it would be a lot faster than it currently is targeting C.
> The issue with targeting C is you ended up losing a lot of information about intent. Compilers looking at C have to put a lot of trust that what the programmer wrote was intentional (and thus, unoptimizable).
The same issue exists in C-sourced compiler backends, which is both GCC and LLVM. It’s not exactly a secret that anything which is not strongly exercised by standard C(++) codebases tends to be rather broken for any non-trivial use and takes years to fix (restrict/noalias being the poster child for this, at least in the context of Rust, I think it hasn’t gotten re-disabled yet so it might make a year).
The JVM, for example, doesn't see the same issue so much with non-Java languages. Mainly because (IMO) it's been hosting non-java languages for several years now.
I'm not sure. The following claim is being made on Nim's website:
> Modern concepts like zero-overhead iterators and compile-time evaluation of user-defined functions, in combination with the preference of value-based datatypes allocated on the stack, lead to extremely performant code.
> Efficient, expressive, elegant
> Nim is a statically typed compiled systems programming language
A systems programming language need to have exceptionally good performance, so not sure it's to be expected that the performance is so different than Rust.
Apparently the Rust versions have received more optimization efforts compared to the Nim code as well, and it seems some people are working on optimizing the Nim versions so we'll see where it lands in the future I guess: https://news.ycombinator.com/item?id=30244188
Having programmed in both I think Rust often makes it easier to write fast code by making all the copying vs borrowing more explicit.
Nim is obviously still very fast and much easier to pick up, so there are trade-offs here.
As an aside, I'm not expecting a lot of languages to live up to the "C like performance" claim when they make it, as C has been receiving optimizations for over 50 years by now, so it makes sense it's still the fastest around, at least until there is some major breakthrough in programming language design.
EDIT: Downvoters - is that not the case?
C isn't inherently fast, commonly used constructs in C just tend to match up well with semantics that can be turned into high-performance machine code. For a language that compiles to C to be fast, its commonly used constructs would need to have semantics that match up well with what can be turned into high-performance C, which is often not the case. There are many examples of languages that compile to C, but are still very slow.
Maybe a better way to frame it is that most other languages are inherently slow. Garabge collection will always be slower than non-GC. JIT and interpreted languages will always be slower than ahead-of-time compiled languages. etc etc.
C has a runtime, it's called the CRT (C Runtime).
http://www.vishalchovatiya.com/crt-run-time-before-starting-...
It's quite small, but it's there.
I am well aware of the CRT. I don't think its worth mentioning because it is a "run time" in name only. It literally doesn't do anything at run time. The only thing crt0 does is call your main(). From that point on, only your actual code is executing.
Contrast with Java or Python, where your code is executing inside a runtime which eats up cpu cycles at runtime.
Look, I get what you're saying, but it's not exactly relevant here. Yes, after your C program is loaded, execution follows by just moving the PC to the next instruction in the loaded binary. In C, adding two integers can be compiled down to a concise, machine supported operation; whereas in a language like Python, adding two numbers is a branch-heavy operation that cannot be compiled to a concise instruction ahead of time, so it must be interpreted at runtime. That's true. But that's really more about C being a compiled language than whether or not it has a runtime.
For an example of a runtime-heavy language that can compile highly performant code, see Julia. Thanks to dynamic dispatch, you get the best of both worlds. Yes you pay a heavy upfront cost (TTFP), but it's the same deal with compiling code -- pay once upfront, benefit forever with improved runtime performance.
You used Java as an example of a runtime-heavy language, but I think Java demonstrates why C should be considered as having a runtime, regardless of how small it is. Imagine a hardware Java Processing Unit (JPU), which is a CPU that has Java byte code for machine code [1]. I don't see why it wouldn't be possible to handle all of the stuff the JRE does in the OS, and everything the JVM does in the JPU. Then we could say that Java has no runtime. But really all we've done is taken the runtime and moved it to another place.
You could also imagine a version of C which compiles to CBC (C byte code) and runs on a CVM (C virtual machine), which itself runs on the JPU. You could write C code as usual, but the resulting binary would be compiled to CBC, and would have to run through a number of layers before it is executed on the JPU. Then could we say Java is an inherently fast language, and C is necessarily slower because it has a runtime? Maybe in this strange reality, but it's not necessarily the case. And it certainly has nothing to do with the languages per se, but their implementations on specific hardware/software platforms.
Or to get even crazier with it, we could imagine a version of Java which compiles directly to x86 machine code, because why not? There's no reason that should be slower than C.
Anyway, I think the point is that whether or not one considers a language to have a runtime depends very much on how a language is compiled, what hardware it's targeting, and how much you're able to leverage the OS (there's usually a lot of leverage available when the OS itself is written in the language you're using). So really, all languages have a runtime, it's just a matter of where it's hiding.
[1] https://en.wikipedia.org/wiki/Java_processor
Your definitions are meaningless. You may as well also say that all languages are garbage collected, since the C programmer must call `free` one way or another (or not, and let the OS be the garbage collector at cleanup).
By everyone's understanding of the words, C does not have a runtime nor garbage collection. Java does have a runtime and garbage collection. You can play with definitions and hypotheticals; it does not change the facts.
What? But we both just agreed it did.
The way I understand these words are that C is a language, and a language is different from its implementation, and implementation is dependent on hardware. I understand that a runtime is code that runs when a program starts, in order to enable its execution. Is this not your understanding?
Let me put it this way: if C is inherently fast as you, why can you take the same C source, put it into two different compilers, and get two different binaries with different performance profiles?
The answer is because the language itself has no bearing on speed, and it all comes down to how a language is compiled. So I still wouldn't say C is inherently fast, but I would say that C's design lends itself to compiling fast binaries for Von Neumann style machines. That of course doesn't preclude compiling just-as-fast or even faster binaries from other languages, e.g. Rust or C++.
> You can play with definitions and hypotheticals; it does not change the facts.
I'm not playing with hypotheticals. JPUs are a thing; I linked to the Wiki page. Natively compiled Java is basically GraalVM: https://www.graalvm.org, which I think pretty much proves the point -- it all comes down to the compiler. Garbage checking is also an implementation detail. You can even run a garbage collector for C if you want.
Moreover, it is also untrue on the other side of this axis - you can often go much faster. Even with -Ofast or -Omax or whatever there are still (often) opportunities for SIMD vectorization on modern machines that autovec cannot find. Most C compilers provide (not really standardized) escape hatches to assembly in various ways for that last 2..64X speed-up (single thread). (Yes, avx512/8-bits=64 wide vectors these days.)
Finally, whether it "has a runtime" in the sense of automatic memory management, well, that just depends upon how you link programs. Link with -lboehm and you can never free (and, yes, allocations become potentially expensive). These Rust conversations are unfortunately prone to absolutist statements that indicate very narrow exposure.
[1] https://en.wikipedia.org/wiki/Boehm_garbage_collector
If your language has semantics that are difficult to translate into fast C, then your language isn't going to be fast just by going through C.
Even if you do use the GC, the compiler elides GC work if the type doesn't escape the scope.
But no, Fortran is not an alternative to C (und vice versa).
Languages like Nim/Rust/D/etc. should not have significantly different speeds for equivalent implementations for these kinds of benchmarks, as they ultimately can be written to compile to equivalent LLVM IR and thus the same machine code.
What you're usually seeing when you see such large differences is that there are differences in the implementation of the algorithm or of a standard library function.
There's nothing about these kinds of languages that should result in inherently significant performance differences on such synthetic microbenchmarks.
Allocation-heavy or write-barrier-heavy code might make a difference, but you literally have performance differences while iterating over arrays of scalars. And even then, last I benchmarked it, Nim actually compared favorably with e.g. jemalloc.
And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not.
So it all depends on the language.
Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise.
[1]: <https://golangbyexample.com/goroutines-golang/> (potentially stale information)
> And in the case of Swift the atomic "release" calls at ends of scopes will run regardless whether there are hot loops or not.
Yes, but Nim does neither. Nim does insert write barriers, but that's only if you (1) write pointers to heap-allocated memory and (2) don't use the mark-and-sweep GC, so that can't explain big performance differences for those benchmarks where you primarily iterate over arrays.
> Also, you pointed to 3 languages with 3 different memory management strategies and 3 different toolchains (Rust is LLVM-based; Nim compiles to C and then calls a C compiler, GCC recommended; D has its own non-GCC non-LLVM compiler). It feels weird to put them in the same bag implementation-wise.
You don't seem to be aware of it, but D actually has an LLVM-based compiler (LDC). Nim can also utilize LLVM via choosing clang as its backend. You can definitely use them to generate equivalent LLVM IR and compare that. There's nothing weird here, this allows us to rule out differences related to the backend.
I guess it depends on what significant is but I certainly hope Rust can be faster than C in many scenarios due to optimizations llvm can make with the added type information such as inferring aliasing rules.
https://github.com/rust-lang/rust/issues/54878
- multithreading runtime (i.e Rayon vs Weave https://github.com/mratsim/weave)
- Cryptography: https://hackmd.io/@gnark/eccbench#Pairing
- Scientific computing / matrix multiplication: https://github.com/bluss/matrixmultiply/issues/34#issuecomme...
There is no inherent reason why a Nim program would be slower than Rust.
¹ https://forum.nim-lang.org/t/8868
(The thread points out that the Nim version uses a doubly-linked list from the standard library while you'd need unsafe in Rust, and the lru crate does indeed use unsafe. So take that into account.)
Meanwhile: it may suggest the comparison is a work in progress, not finished or definitive.
Can data be "silly" or is it what people say about the data that is "silly"?
Unsafe is part of the rust language. Theres nothing wrong with using it if you know what you’re doing. The value of unsafe is that it boxes your unsafe code - so you can direct extra care to those functions when writing, reviewing and auditing. (As opposed to C where memory corruption could happen anywhere, so you need to be careful everywhere.)
I could imagine a separate language on the programming languages shootout which is “safe rust”. It’d be interesting how the best unsafe and the best safe programs compare in speed and size.
I would say that a proper way to rephrase it would be to use idiomatic code vs unidiomatic. If you start using any high-level language in a non intended fashion (writing non idiomatic code) probably most of them would be faster.
I assume nim has gotten a lot less attention in these benchmarks and there's probably huge wins available to it.
In the benchmarks game —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
This is not the benchmarks game, although "some of the benchmark problems and implementation are borrowed from it".
I think you're saying that no GC language gets withing 2-3x of the fastest non-GC language. There are slow non-GC languages.
No doubt I misunderstood what you meant, so
Is it just that I'm reading your "2-3x" literally, and you also meant 1.5x? :-)Finally! :-)
This is wrong. At least, it is not true generally. I wrote a 1:1 port of my C++ chess engine, and after a bit of fiddling with compiler options the Nim version was faster. In fact, when using a delayed garbage collector, the program was even faster than when using reference counting.
It makes it clear that some people have most time to optimize for a benchmark than others.
This benchmark on state-of-the-art cryptography shows that GC-ed languages including Go and Nim can be 2x faster than Rust:
- https://hackmd.io/@gnark/eccbench#Pairing (old benchmarks, in particular Nim had a parameter passing issue/bad copy behavior on large types)
Furthermore, Nim GC/memory management is type-dependent, only seq/string (which use malloc/free) and `ref T` type (which use refcounting) involves some runtime. So Nim devs can choose to opt-in in GC or use manual memory management depending on their priorities.
Edit, and also no, cycles are not "more about instruction set". They're pulses of electricity that make their way through a chip, they are about as far from instruction-set-specific as it gets.
Because instruction set is the interface and the architecture is the implementation? Or something else?
One important component of a program's performance, especially in numeric code, is how many instructions it manages to execute per cycle. If you have written your program's instructions in a 'good' order, then the CPU will e.g. not queue up many divisions in a row that depend on each other. It can do two different things at the same time. You can't do anything in less than one cycle, but you can do two of the same thing in the same cycle if you have two available execution units for that task.
The most direct answer to your question is that different instruction sets have different ideas about how much work a single instruction should do. The basic divide is between CISC (complex instructions, x86 qualifies) and RISC (basic instructions, ARM). It is a dramatic oversimplification but one instruction in a CISC architecture might involve 10 instructions in a RISC. So you can't compare instruction counts between architectures. But you can compare cycles and still make a bit of sense.
If this is interesting to you, you should try this game, where you can build a CPU purely out of the most basic logic gates: https://nandgame.com/
Here's something interesting. If I run my test in a RHEL5 (Linux c. 2010) VirtualBox VM then it actually goes faster than a modern Linux kernel running on bare metal, taking only 14µs. If I run it in a RHEL7 VM then it needs 131µs.
So >1ms hello world execution should be a red flag. If that's actually the latency of GitHub's kernels and can't be explained by something like a shell script doing the time interval measurements, then it'd be interesting to learn what it's doing.
Here's the measurement code, it appears to be significantly more complicated than a simple fork/exec/wait loop but that could just be all the C# getting in the way: https://github.com/hanabi1224/Programming-Language-Benchmark... Note that we are definitely measuring the C# async runtime to some degree. Nevertheless you are probably right that the bulk of this 1.8ms is in the executable under test, and it truly is just bloat. Running `hyperfine ./empty-main-function` from rustc on my Mac gives 0.8ms.
[0] https://justine.lol/ape.html
Edit: FWIW, building with "gcc foo.c -o foo -O2 -s -Wall --static" was slightly faster than without the --static. And "musl-gcc foo.c -o foo -O2 -s -Wall --static" was slightly better again. Maybe 0.5ms improvement in total. I'm on Ubuntu 20 x64, gcc 9, Intel Xeon W-2133.
Then for my musl-gcc hello world, I did, "for i in {1..10}; do o//examples/rusage.com ../hello" and the lowest wall time I got was 120us. But most of the runs were closer to 350us.
So, my attempt to answer your, "Why does hello world take 1.8ms to run?" question is: a) they timed it with "time" which measures lots of overhead, b) there's something wrong with their setup that means they are mostly timing the time taken for a CPU core to wake up.
Meanwhile —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
----
> I only wish the benchmarks had some metric for how idiomatic the implementation is compared to normal developers.
Maybe the benchmarks game website kind-of does — source code size.
Otherwise "I only wish…" so — wishful thinking.
Otherwise "how idiomatic" seems to mean — how I would implement the program today, if I didn't care [goldilocks] too much about performance?
(Incidentally "normal developers" doesn't seem to be stretching towards inclusive.)
However it is a mistake to then suppose that nothing can be known and just go ahead and use Python to make a AAA first person shooter.
And no language other than C++ has really found traction for stuff like engines; not even Rust...
Then again maybe it would have slowed down development and never gotten off the ground at all!
It's almost four times as slow as the otherwise identical single-threaded version.