The title is a bit misleading. Its not clear if the compiler times are included, but the numbers do include the benchmark. Thats why the exact same sbcl compiler has different times.
The benchmarks are mixed with some having simd and some using threads. It'd be helpful to be able to sort submissions by threaded/unthreaded as well.
Still kinda fun to read the different languages. I found Zig was very chatty and annoying to read even compared to the C versions. The common lisp was foreign. The Fortran 90 with openmp and Julia versions were pleasant to read. The Nim version was easy to read but lacks threading or simd.
unison cheats at this by being interpreted, if it weren’t it would have the same build times as everyone else (I suppose caching would be better in some cases, but i’m not sure if they’d be much better than what you get with nix?)
I've been meaning to write a new frontend for the Computer Language Benchmarks Game that allows comparison of arbitrary pairings for a few years now and never got 'round to it.
It's about time someone did. Thank you and nice work!
This is a really strange benchmark to use. Compiler speed is not an issue for these 100ish line standalone benchmarks. The question is how fast it is for a million-line project or worse. I doubt c or c++ do good there with their header file includes for dependencies.
On a single core processor, obviously not, but massive C++ codebase are compiled with multicore CPU, and often distributed. The advantage of the header inclusion model is you get an bunch of independent compilation units, and so it's straight forward to parallelize.
A common way to speed things up even more is to merge some cpp file together, so each compiler thread is active for longer and common headers are compiled once.
With the new module feature of C++20, compilation is transformed into a DAG, and it is still unclear if it's really faster than just throwing a lot of cores at the problem.
You show me that IDE for C++ who is using those multicore CPU's for compilation who's faster than Delphi, for same number of lines and/or external dependencies. Btw, Delphi is still using 32 bits for its IDE and a single core and still blows out of the water anything else in terms of millions of lines of code compilation time.
I wondered if Delphi supported LTO (as that's one of the main compile-time-eaters) and the only examples I could find of sentences containing both "Delphi" and "LTO" referred to Delphi support for TAPE DRIVES (https://wikipedia.org/wiki/Linear_Tape-Open). Of course prehistoric tech is faster, running a 1995 game on a i7 would also be fast. What matters is, is Delphi able to match C++ in run-time benchmarks ?
Also, here's a video of my edit-compile-run cycle ; cloc on that project gives me 506963 loc without counting dependencies. The software uses boost and fairly modern C++ with no particular feature restriction and templates going wild. It's not instant, but still, 2 seconds is somewhat tolerable (and definitely faster than various web stuff I've been involved with)
What I see there is a compilation of current module that you modify. Do a full project rebuild, then show. Here is the link to Delphi, official one, for a comparison between 2 of the previous version (10.4.1 vs 10.4.2):
I don't have Delphi 11, yet, so I can't verify their claim, but 10.4.2, which I have, is noticeable faster than previous versions I have scattered around in my virtual machines.
why would I ? I never have to do that during my dev cycle, that's not something really worth optimizing for. The whole build is ~4/5 minutes on that machine.
Then there you have it. I have a project, written by me around 150k lines. Has DevExpress components, ReportBuilder components and the whole build is around 2M lines in total in around 30k files, with full dependencies (I have sources for all). Takes 25 seconds, and is inside a virtual machine - as my philosophy is "each project inside their own VM".
Also the link I gave you for video, that's a full rebuild. The guy does a cleanup first, getting rid of existing builded components/modules and then does a full rebuild.
I think the title might be misleading you, when it mentions compiler speed, it means, for the same language there are different compilers. It's about runtime speed of different languages and different compilers for those languages
I think the C code in the benchmarks are not really efficient. An experienced C programmer does not simulate C++ style memory management if speed matters.
C++ is just a stricter C - if you write proper C, it will be a correct C++ program, therefore it would run at the same speed. There's no way C can be other than marginally faster than C++ - if you write C and compile it with a C++ compiler.
Then you have various optimizations that come from things like templates, constexpr. Look at the qsort method - to have a modicum of genericity you need to pass a function that will be invoked. Instead, a C++ version would be able to be optimized to compare the elements directly, without calling a function, avoiding a big number of function calls.
There's no way to make C faster than well written C++. But C++ is complex and a lot of people write bad C++, that's why it has developed a bad reputation. It's really undeserved.
> There's no way C can be other than marginally faster than C++
In theory `restrict` (to disallow pointer aliasing like in Fortran) could lead to optimization gains, same as not having exceptions (the possibility of exceptions prohibits certain optimizations of the C++ compiler).
I agree; of course, both clang and gcc extend C++ with __restrict__ if I remember correctly; from what I know, there are more things that can be done in C++ to improve over C than the other way around.
...I wish this were true. I'm back writing JavaScript/Typescript. The new build tools are great. Perhaps in the new year I'll convert some of these repos to build with esbuild or swc... or vite.
No matter how good you are at Python, if you need to build an SPA, it'll be far faster to go with Javascript. This is true even if you have to learn Javascript from scratch. If you're an expert at Pascal, good luck trying to use it on the Arduino.
The hardest part of learning a second and third programming language is overcoming your own frustration. Yes there are some pretty gnarly concepts in languages like Haskell. However, most mainstream programming languages are very similar to each other.
This is almost never the criteria people use as it is often not that important. Productivity is often a bigger factor. Security and Robustness is another factor. Libraries is another, portability, Platform limitations, Domain problem to be solved. Most software these days isn't really constrained by speed, if the software ran 10x faster, you'd only get marginal gains.
That the same logical steps expressed in one language results in less computation time spent - when compiled or interpreted - than when expressed in another language. Seems a resaonable question to ask, given there's more to a language than how you express things. Like toolchain, library availability and community.
Same idea as comparing different compilers for the same language. Some algorithms can be expressed in some languages in an essentially identical way. For example, various sorting algorithms look pretty much the same in Fortran, Pascal and C. Only surface syntax differs. Assuming this is done fairly with each implementation being properly equivalent, it's a way to compare the speed of the languages, at least for that algorithm.
Not sure how technically accurate it is, but "language" usually implies more than just the syntax. I hear and say things like "compiled language", "interpreted language", "language VM", "garbage collected language", "dynamically typed language" and so on. Those things all say something about development speed and runtime speed.
The phrases "compiled language" and "interpreted language", despite their widespread use, are not particularly sound concepts. There are plenty of languages that have both compilers and interpreters.
A lot. Semantics that can be made fast (e.g. C vs Python) with little effort.
(Some languages with bad semantics for that might still get fast, but the language itself isn't designed to help that. JS for example was made fast, but with tons of effort but with tons of money and man-centuries thrown in by Google, Apple, Mozilla, MS, not the kind of recources available to most languages. So, I'd call a language whose semantics afford optimization a fast language).
Runtime complexity (e.g. "zero overhead" vs "lotsa overhead" features).
Also, I find the "language vs compiler" dicthotomy pedantic and tedious. For most languages and most use cases, there's a single dominant canonical version, e.g. CPython, Matz's Ruby, Oracle's Java.
Just because there are niche other implementations all 10 people use doesn't change the fact that 90% of users will be on the canonical implementation, and for them, THAT's synonymous with the language, not just some abstract spec.
Strictly speaking it's meaningless, since a language is a set of syntax and semantics. Pragmatically though, when people talk about the speed of a language they actually mean the typical performance of processes defined by programs written in that language, which of course depends on the code generator and runtime, among other things.
There's nothing stopping anyone from writing a fully standards compliant C interpreter that will have considerably slower process execution after all. In fact, a quick search shows that it's been done many times, for example this[1].
Maybe we mean programs, not languages; maybe we mean programming language implementations, not languages —
"How many times slower, the fastest benchmark programs for selected programming language implementations are, compared to the fastest written in any of the programming languages."
Julia always seems to rank #1 or close to it when compared to other interpreted languages but I only ever see it used for research and math. Why doesn’t anyone use it for anything else?
Lack of integration with other massive libraries that Python, Java and C++ have. Lack of books, tutorials and extensive knowledge bases that Python, Java and C++ have.
Java doesn't integrate well with Julia, but Julia's interop with C++ and python are both pretty painless. pycall in particular is about as easy as you could imagine.
Julia is incredibly slow to compile. Far worse than C++ last time I tried. For interactive notebooks, I found it’s common to have a single line take a couple minutes to compile - it’s simply not interactive in the way Python is, despite having good runtime performance once compiled.
The deal breaker for me is that its compiler is JIT based whether you want it or not. Code will hang at runtime for multiple minutes as it compiles all the library dependencies and specializes them to the given data types.
Do you have an example for minutes of compile time for a single line? Especially since 1.6 (march 2021), I haven't run into that (and I'm using Julia pretty much full-time).
In comparison to what? I've found that it's very responsive in a repl/notebook for interactive work (especially with Revise). It's not good for running short scripts, but if you have something that's doing some real work, the compilation often isn't a big deal.
Personally, I think interactive usage has improved quite a bit. I don't "feel" it that badly in 1.7. It was bad enough earlier in the 1.0 series that I stopped using it for a while (I used it before <1.0 as well).
I think all of it has been improved, and can certainly be improved further. That also includes improving the speed of its interpreter (vs the JIT), which exists but was never optimized for speed afaik.
This is absolutely false in my experience. You pay for startup (100 ms on my system, I think) and for initial compilation, which has still been decently fast for me. Beyond that it's very fast.
In addition to the other replies, the comparatively long startup time makes it unsuitable for short scripts, and with a few dependencies it becomes unsuitable for even medium length programs. The startup time is always coming down and there are ways to compile dependencies ahead of time though. The JIT compilation can be annoying for real-time applications too.
These are what i'd say, as someone who has tinkered with it a lot but uses python/C++ for $day_job:
- immature web/network services stack. For better or worse the lang seemed to focus on more number crunching throughput early than reliable distributed data processing, but the latter is a much larger market and what most enterprises care about.
It's the difference between a lang only having Matlab or R ceiling vs python-like ceiling.
- poor static compilation or static analysis story. this is what a lot of people needing to put code in production need.
It's also standard expectation now with a lot of dynamic language code (python or js) that aims to be in production.
- for certain types of realtime uses, more direct control of memory is doable, but you have to jump through hoops. Again, determinism is very important.
- historically, compile times were quite horrid. part of this was llvm, part of it was julia. It's gotten way better in 1.6 and 1.7, but compilation results need to be cached more than just precompilation.
however, all being said, I love the lang! There is a lot going for it especially how the package ecosystem works. There is also a lot of innovative projects.
Julia has @code_warntype for type checking and Jet.jl for static analysis (and snoopcompile/cthulu for the really advanced stuff). Is there anything those are missing?
a lot of it is stuff its runtime loads by default, like huge BLAS buffers, the llvm orc and such.
I think there have been steps recently to make those optional recently. Julia was really optimized for "big iron"-styles of workloads, but that's less than ideal when you are not trying to just do linear algebra all day.
During it's development Julia was heavily promoted as a language for scientific computing.
Nowadays, the language is promoted as a general purpose language suitable for any task or domain.
However, the perception that Julia is primarily for scientific computing remains strong among many developers. Only time will tell if that perception changes.
I came here to ask the same. Pascal files are laid out in such a manner that the "implementation" section of the file is ready for compilation to bytes and any forward references just mean that it's code ends up later in the exe.
Typical runtime speed has only ever been fractionally slower than c/c++.
Common Lisp (sbcl) performance via native implementation of simd [0] is very impressive ! It is litteraly acheieving C/Cpp speeds (within few ms). Great work by Marco Heisig
Manual memory management a la C, and easy access to custom allocation (arenas, etc) that usually aren't used in most C code (and C doesn't offer as allocators by default).
Add compile time processing which can get pretty far into reducing all kinds of otherwise runtime ops...
I wonder if guile[1][2] would be competitive to the other Lisp (Scheme) implementations. It was significantly faster than Python at numerical tasks in my experience.
Whoah, I'm shocked by seeing Julia perform well. The compile times back in the day were abysmal. I mean even for very, very simple projects you had to wait for dozens of seconds. It looks like they made an enormous progress there.
89 comments
[ 3.4 ms ] story [ 94.2 ms ] threadI don't use Unison but I would love to for this reason.
Incremental: 700ms
Esbuild clean: 100ms
The benchmarks are mixed with some having simd and some using threads. It'd be helpful to be able to sort submissions by threaded/unthreaded as well.
Still kinda fun to read the different languages. I found Zig was very chatty and annoying to read even compared to the C versions. The common lisp was foreign. The Fortran 90 with openmp and Julia versions were pleasant to read. The Nim version was easy to read but lacks threading or simd.
It's about time someone did. Thank you and nice work!
(Ok it’s 5-10k lines rather than a million, but it’s non-trivial enough that the differences between languages are noticable)
A common way to speed things up even more is to merge some cpp file together, so each compiler thread is active for longer and common headers are compiled once.
With the new module feature of C++20, compilation is transformed into a DAG, and it is still unclear if it's really faster than just throwing a lot of cores at the problem.
Also, here's a video of my edit-compile-run cycle ; cloc on that project gives me 506963 loc without counting dependencies. The software uses boost and fairly modern C++ with no particular feature restriction and templates going wild. It's not instant, but still, 2 seconds is somewhat tolerable (and definitely faster than various web stuff I've been involved with)
https://streamable.com/az397y
https://www.youtube.com/watch?v=lBSpv94FDE4
And then here is a claim that latest one (Delphi 11 Alexandria) is even faster:
http://www.danieleteti.it/post/delphi-11-alexandria-compiler...
I don't have Delphi 11, yet, so I can't verify their claim, but 10.4.2, which I have, is noticeable faster than previous versions I have scattered around in my virtual machines.
why would I ? I never have to do that during my dev cycle, that's not something really worth optimizing for. The whole build is ~4/5 minutes on that machine.
Also the link I gave you for video, that's a full rebuild. The guy does a cleanup first, getting rid of existing builded components/modules and then does a full rebuild.
This uses the debian language benchmarks, in search of what it was that was different I falsely concluded this must be about compilation speed.
https://stackoverflow.com/questions/2513741/c-vs-c-for-perfo...
from which I gather that if you're using an allocator function that zero's out memory then it will be slower of course.
Then you have various optimizations that come from things like templates, constexpr. Look at the qsort method - to have a modicum of genericity you need to pass a function that will be invoked. Instead, a C++ version would be able to be optimized to compare the elements directly, without calling a function, avoiding a big number of function calls.
There's no way to make C faster than well written C++. But C++ is complex and a lot of people write bad C++, that's why it has developed a bad reputation. It's really undeserved.
In theory `restrict` (to disallow pointer aliasing like in Fortran) could lead to optimization gains, same as not having exceptions (the possibility of exceptions prohibits certain optimizations of the C++ compiler).
https://github.com/hanabi1224/Programming-Language-Benchmark...
https://github.com/hanabi1224/Programming-Language-Benchmark...
The hardest part of learning a second and third programming language is overcoming your own frustration. Yes there are some pretty gnarly concepts in languages like Haskell. However, most mainstream programming languages are very similar to each other.
It is supposed to work with FreePascal: https://wiki.freepascal.org/AVR_Programming
(Some languages with bad semantics for that might still get fast, but the language itself isn't designed to help that. JS for example was made fast, but with tons of effort but with tons of money and man-centuries thrown in by Google, Apple, Mozilla, MS, not the kind of recources available to most languages. So, I'd call a language whose semantics afford optimization a fast language).
Runtime complexity (e.g. "zero overhead" vs "lotsa overhead" features).
Also, I find the "language vs compiler" dicthotomy pedantic and tedious. For most languages and most use cases, there's a single dominant canonical version, e.g. CPython, Matz's Ruby, Oracle's Java.
Just because there are niche other implementations all 10 people use doesn't change the fact that 90% of users will be on the canonical implementation, and for them, THAT's synonymous with the language, not just some abstract spec.
Matz's Ruby MRI?
No — "Since Ruby 2.6 introduced MJIT in 2018, its performance greatly improved"
https://www.ruby-lang.org/en/news/2021/11/09/ruby-3-1-0-prev...
There's nothing stopping anyone from writing a fully standards compliant C interpreter that will have considerably slower process execution after all. In fact, a quick search shows that it's been done many times, for example this[1].
[1] https://github.com/jpoirier/picoc
"How many times slower, the fastest benchmark programs for selected programming language implementations are, compared to the fastest written in any of the programming languages."
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
The deal breaker for me is that its compiler is JIT based whether you want it or not. Code will hang at runtime for multiple minutes as it compiles all the library dependencies and specializes them to the given data types.
I think the problems were from:
1) llvm itself getting dramatically slower over time, before more focus upstream on compiler speed more recently: https://www.npopov.com/2020/05/10/Make-LLVM-fast-again.html
2) different types of invalidations
3) less than ideal precompilation of packages
I think all of it has been improved, and can certainly be improved further. That also includes improving the speed of its interpreter (vs the JIT), which exists but was never optimized for speed afaik.
Also, there's a solution to precompile binaries with no JIT penalty...
https://github.com/JuliaLang/PackageCompiler.jl
Enjoy!
- immature web/network services stack. For better or worse the lang seemed to focus on more number crunching throughput early than reliable distributed data processing, but the latter is a much larger market and what most enterprises care about.
It's the difference between a lang only having Matlab or R ceiling vs python-like ceiling.
- poor static compilation or static analysis story. this is what a lot of people needing to put code in production need.
It's also standard expectation now with a lot of dynamic language code (python or js) that aims to be in production.
- for certain types of realtime uses, more direct control of memory is doable, but you have to jump through hoops. Again, determinism is very important.
- historically, compile times were quite horrid. part of this was llvm, part of it was julia. It's gotten way better in 1.6 and 1.7, but compilation results need to be cached more than just precompilation.
however, all being said, I love the lang! There is a lot going for it especially how the package ecosystem works. There is also a lot of innovative projects.
You're comparing it with python.
https://github.com/typeddjango/awesome-python-typing
https://github.com/ethanhs/python-typecheckers
though i'd say they are worse than similar tooling in typescript/javascript.
julia's tooling is relatively nascent.
I think there have been steps recently to make those optional recently. Julia was really optimized for "big iron"-styles of workloads, but that's less than ideal when you are not trying to just do linear algebra all day.
Nowadays, the language is promoted as a general purpose language suitable for any task or domain.
However, the perception that Julia is primarily for scientific computing remains strong among many developers. Only time will tell if that perception changes.
[0] https://github.com/marcoheisig/sb-simd
http://shootout.alioth.debian.org
and it crashed and data was lost AFAR.
There's now
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://en.wikipedia.org/wiki/The_Computer_Language_Benchmar...
Just wanted to mention prior art.
The same benchmarks game project moved to new hosting at:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Add compile time processing which can get pretty far into reducing all kinds of otherwise runtime ops...
[1] https://www.gnu.org/software/guile/
[2] https://web-artanis.com/scheme.html
Meanwhile —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Don't you have the opportunity to compare user time as-well-as wall clock time?
( The benchmarks game does provide that sort --https://benchmarksgame-team.pages.debian.net/benchmarksgame/... )