So if I understand correctly, the pidigits test in this case this isn't a self-contained, fully ported microbenchmark, but rather a test of some random Rust code vs. an optimized library[1]?
If that's true, then what's the point of including pidigits if the entire algorithm isn't going to be ported into all target languages?
It's more of a "testing the platform" rather than the language. You can write a Rust program that runs at exactly the same speed as a C program (the difference here would be clang backend vs. gcc moreso than the choice of language).
As I understand it, the rule is "if your language comes with a bignum implementation, you must use it, otherwise you may use GMP". I've been seriously tempted to suggest removing our bignum implementation for this reason (though I won't).
A fix is in the queue. It was a tiny thing. Currently there is an Android buildbot issue that is preventing anything from landing, but hopefully that will be resolved today.
I actually have no idea where they come from. nbody in particular is very strange because we've been on par with C in that benchmark for years in my tests.
It's ridiculous and sad that they would include Rust, after banning LuaJIT a few years ago, citing "too much maintenance effort" (even though LuaJIT benchmarks were significantly more complete that Rust's)
The shootout has been utterly ridiculous for quite a long time. I remember when the maintainer(s?) threw a fit when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy) even though multiple programs per language had not been an issue before. Then they ended up banning multiple implementations, but their implementation pick is completely arbitrary.
You might think so, but keep in mind that it is a thankless volunteer job that's been on-going for something like 10 years. I'm pretty sure that Isaac would encourage your efforts to make that site.
Surely in today's world with elastic compute and robust virtual containers, it should be possible to replace thankless volunteer work with automated, self-service processes.
I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
Taking the measurements isn't really the thankless volunteer work involved here. The work is in ensuring new programs actually follow the intent of the benchmark and work across supported architectures.
This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
Except they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed.
The point of saying "send us a pull request" is just PR and it seems to be working.
AFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc.
I believe they have plans to add additional tests which will allow for caching and that kind of thing.
I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
>>when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy)<<
NOT TRUE.
Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted.
>>banning multiple implementations<<
NOT TRUE.
Measurements for both JRuby and CRuby are shown.
Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements.
The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.
igouy pretty much gets nothing but grief maintaining the shootout. He's told people for years that they should publish their own numbers for different language implementations. Never happens, just complaints.
LuaJIT, PyPy and HHVM are common and well know. If it is true that they devs contacted him and he refused to add it, that's sad.
I know maintaining a project like that is a lot of work. But adding superior implementations of already featured languages should be a piece of cake. Source code of Python, PHP and Lua is already in the repo, it's just a matter of installing the VM's on the computer and modify the scripts.
Beside, if the maintainer reads this, please keep up the good work, and maybe add LuaJIT, PyPy and HHVM. Thanks.
> it's just a matter of installing the VM's on the computer and modify the scripts.
That can be a pretty huge "it's just". In practice many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs. Since the alternate implementation, benchmark programs, and benchmark tools are all available, I'd recommend you put your theory to the test. It'll either prove eye opening or make for a nice contribution back to the game!
> many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs
HHVM is compatible with PHP 5.3, LuaJIT with Lua 5.1 and PyPy with Python 2.7.3. All benchmark code is written the languages code and sometimes relies on an language extension library (which is written in C or C++). But three above JIT VMs come with their own compatible language extension libraries or support the extension API.
"If you're interested in something not shown on the benchmarks game website then please take the program source code and the measurement scripts and publish your own measurements."
afaict we all feel the same way about this, we all feel that we should sit on our hands and wait for someone else to do the chores we don't wish to do. Of course, measurements of the "proggit popular" language implementations like PyPy and LuaJIT will attract attention and be the basis of yet another successful website (unlike more Fortran or Ada or Pascal or Lisp).
Anyone else a bit shocked by how well Javascript on V8 performs? I might need to rethink my assumptions but it actually runs faster than Python 3 according to this.
That benchmark isn't benchmarking what you think it is.
"This test exercises database writes. Each request is processed by fetching multiple rows from a simple database table, converting the rows to in-memory objects, modifying one attribute of each object in memory, updating each associated row in the database individually, and then serializing the list of objects as a JSON response."
You're actually just benchmarking the mysql driver for each of those languages, not the actual JIT capabilities.
The website design is a bit confusing, you probably missed the other tabs: "JSON serialization", "Single query", "Multiple queries", "Fortunes", "Data updates", "Plaintext". It's a micro benchmark but at least it covers a lot of languages and frameworks. (I saw that benchmark site on HN some months ago)
One thing that surprised me about that benchmark is how many languages fall into the "2 to 3 times slower than C" category. They all have various merits, but seemingly equal performance. I've also noticed each of the languages I'm familiar with have gone through a hopeful stage of "were as fast as C" only to fall short.
There's this strange idea still floating around that Java is somehow slow. Compared to hand crafted micro benchmark C code, Java is slower. But then so is everything else.
The simple facts are that the JVM is the most advanced and efficient VM available today, and the JVM is specifically targeted to increase the speed of Java. It's not much of a stretch to expect good speed out of that regardless of strange preconceptions you may have had.
The Go VM has a long long way to go be even remotely comparable to the JVM in garbage collection etc.
Go is anything but slow, it's just not as fast as Java. I was just addressing the idea that Java is somehow slow which seems to float around occasionally. Java is arguable the fastest language that isn't directly C-based and anybody thinking of it as slow is simply misinformed.
The biggest problem is that Go's garbage collector is conservative and slow. Add in that I don't think Go's optimizer is yet as good as HotSpot's, and you've got the recipe for those benchmarks.
Also, the default Go compiler essentially has no optimiser at all, just some weak inlining + a language designed to be easy to compile directly to non-horrible machine code without much effort. GCCGo will be much faster.
How do you expect Go to be faster than Java? Off the top of my head here's three reasons why I expect Go to be equal or slower in the general case:
* the type system doesn't provide any more information to optimize against
* lack of generics means you're going to incur runtime type checks to unpack data out of containers
* internal pointers force a choice between ignoring advanced garbage collection (which usually involves moving data in memory), making all object accesses a double pointer dereference (to allow moving), and forcing the user to specify at allocation time if the object should be eligible for movement. All of these choices suck.
Personally, I don't expect Go to ever be "faster than Java". Maybe (hopefully!) I'm wrong, but these are big roadblocks.
Maybe not. Rust is starting to mature and the code generation is already quite good (though there are still many opportunities for improvement). Hopefully this exposure will inspire people to improve these benchmarks. With a little effort Rust should be very competitive on them.
56 comments
[ 2.9 ms ] story [ 20.3 ms ] threadIf that's true, then what's the point of including pidigits if the entire algorithm isn't going to be ported into all target languages?
[1] GMP, The GNU Multiple Precision Arithemtic Library: https://gmplib.org/
Python: http://benchmarksgame.alioth.debian.org/u64/program.php?test...
Ruby: http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...
When someone contributes a Rust pi-digits program that uses GMP it will be shown (as long as it uses the same algorithm).
We get to see both programs that use GMP and programs that don't.
See this discussion: http://www.reddit.com/r/rust/comments/1xcq1q/c_gcc_vs_rust_w... There are also already some patches making rust a lot less worse: https://github.com/mozilla/rust/pull/12209
All the shootout* ones.
http://benchmarksgame.alioth.debian.org/play.php#faqs
I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks
I believe they have plans to add additional tests which will allow for caching and that kind of thing.
I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
http://alexgaynor.net/2011/apr/03/my-experience-computer-lan...
Sounds entirely reasonable to me.
NOT TRUE.
Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted.
>>banning multiple implementations<<
NOT TRUE.
Measurements for both JRuby and CRuby are shown.
Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements.
The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.
That's sad. Someone should fork it... would be great.
Another good benchmark that compares them all: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7... (found it on HN some month ago)
I know maintaining a project like that is a lot of work. But adding superior implementations of already featured languages should be a piece of cake. Source code of Python, PHP and Lua is already in the repo, it's just a matter of installing the VM's on the computer and modify the scripts.
Beside, if the maintainer reads this, please keep up the good work, and maybe add LuaJIT, PyPy and HHVM. Thanks.
That can be a pretty huge "it's just". In practice many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs. Since the alternate implementation, benchmark programs, and benchmark tools are all available, I'd recommend you put your theory to the test. It'll either prove eye opening or make for a nice contribution back to the game!
HHVM is compatible with PHP 5.3, LuaJIT with Lua 5.1 and PyPy with Python 2.7.3. All benchmark code is written the languages code and sometimes relies on an language extension library (which is written in C or C++). But three above JIT VMs come with their own compatible language extension libraries or support the extension API.
Look at the code, it should be easy: http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?t... and https://github.com/kragen/shootout/tree/master/bench
http://benchmarksgame.alioth.debian.org/play.php#languagex
PyPy was shown (on my initiative) and I asked for (and published) programs that would work with PyPy. For example check the comments here -- http://morepypy.blogspot.com/2010/03/introducing-pypy-12-rel...
LuaJIT was also shown; HHVM was never shown.
http://web.archive.org/web/20110125034347/http://shootout.al...
Otherwise people might get the impression that C is the "speed of light" as far as program speed is concerned.
http://benchmarksgame.alioth.debian.org/play.php#languagex
A company did that back in Dezember: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7... (found it on HN)
Revel uses LuaJIT (afaik), HHVM is HipHop PHP (Facebook) and you can find PyPy there too.
"This test exercises database writes. Each request is processed by fetching multiple rows from a simple database table, converting the rows to in-memory objects, modifying one attribute of each object in memory, updating each associated row in the database individually, and then serializing the list of objects as a JSON response."
You're actually just benchmarking the mysql driver for each of those languages, not the actual JIT capabilities.
One thing that surprised me about that benchmark is how many languages fall into the "2 to 3 times slower than C" category. They all have various merits, but seemingly equal performance. I've also noticed each of the languages I'm familiar with have gone through a hopeful stage of "were as fast as C" only to fall short.
The simple facts are that the JVM is the most advanced and efficient VM available today, and the JVM is specifically targeted to increase the speed of Java. It's not much of a stretch to expect good speed out of that regardless of strange preconceptions you may have had.
The Go VM has a long long way to go be even remotely comparable to the JVM in garbage collection etc.
Apparently, the cutoff is defined like this: as slow as Java is not slow, but any slower than Java is slow.
Also, the default Go compiler essentially has no optimiser at all, just some weak inlining + a language designed to be easy to compile directly to non-horrible machine code without much effort. GCCGo will be much faster.
* the type system doesn't provide any more information to optimize against * lack of generics means you're going to incur runtime type checks to unpack data out of containers * internal pointers force a choice between ignoring advanced garbage collection (which usually involves moving data in memory), making all object accesses a double pointer dereference (to allow moving), and forcing the user to specify at allocation time if the object should be eligible for movement. All of these choices suck.
Personally, I don't expect Go to ever be "faster than Java". Maybe (hopefully!) I'm wrong, but these are big roadblocks.
It's important to track this over time to make sure regressions don't get in.