The amount of work it takes to produce a really good benchmark is something more than what this guy did.
Exhibit A: his analysis of optimization impact.
I've been accused of biasing the results by using the -O2 option for GCC, supposedly because -O2 optimizes for space, thus slowing down the benchmark. This is not what -O2 does. According to the GCC -O documentation...
-O2: Optimize even more. GCC performs nearly all supported
optimizations that do not involve a space-speed tradeoff. The
compiler does not perform loop unrolling or function inlining
when you specify -O2. As compared to -O, this option increases
both compilation time and the performance of the generated code.
On the other hand, -O3 performs space-speed tradeoffs, and -O performs fewer optimizations. Thus, for these tests, I think -O2 was the best choice.
Uh...what about optimizations performed in the JVM (or, more specifically, the JIT)? How do they compare? How do you know that -O2 is the right choice, other than "it's kinda in the middle between lots of optimizations and not many, so that sounds good."? He could have started with this: http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html
Exhibit B: he may have introduced a ton of bias through test selection and doesn't even know it
Some of the C++ tests would not compile. I've never been very good at decoding GCC's error messages, so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks.
What?? If you can't read GCC's error messages, I don't get any warm fuzzies about your ability to decide which tests compare the things you want to compare or not.
There's plenty more to be critical of here, but the above two really were the sore thumbs in this article.
/me gets back to real work and plans to dissect this more tonight
The testing decisions feel very arbitrary. If he really wants to show that Java is better than C++, he must give C++ every opportunity to do better, then show that Java does even better despite this. The correct optimization flag to use is the one that gives C++ the best performance, not whichever he felt like using. Likewise for the choice of benchmarks: he should use the version of the C++ benchmark that runs the fastest, and he should have a C++ expert fix the broken benchmarks. Otherwise the only conclusion that can be drawn is: "Java experts are better at tuning Java programs than C++ programs".
While he's at it, he can try the new Shootout benchmarks, which clearly show C++ in the lead, even when using his vaunted "-server" flag:
...How old is this, anyway? There's no date in the article, but he's running gcc 3.3.1 and Java 1.4.2 on Fedora Test1, on a Pentium 4 laptop. The version of gcc places it in 2003, so I'd place more confidence the newer Shootout results.
4 comments
[ 4.8 ms ] story [ 22.3 ms ] threadThe amount of work it takes to produce a really good benchmark is something more than what this guy did.
Exhibit A: his analysis of optimization impact.
I've been accused of biasing the results by using the -O2 option for GCC, supposedly because -O2 optimizes for space, thus slowing down the benchmark. This is not what -O2 does. According to the GCC -O documentation...
-O2: Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff. The compiler does not perform loop unrolling or function inlining when you specify -O2. As compared to -O, this option increases both compilation time and the performance of the generated code.
On the other hand, -O3 performs space-speed tradeoffs, and -O performs fewer optimizations. Thus, for these tests, I think -O2 was the best choice.
Uh...what about optimizations performed in the JVM (or, more specifically, the JIT)? How do they compare? How do you know that -O2 is the right choice, other than "it's kinda in the middle between lots of optimizations and not many, so that sounds good."? He could have started with this: http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html
Exhibit B: he may have introduced a ton of bias through test selection and doesn't even know it
Some of the C++ tests would not compile. I've never been very good at decoding GCC's error messages, so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks.
What?? If you can't read GCC's error messages, I don't get any warm fuzzies about your ability to decide which tests compare the things you want to compare or not.
There's plenty more to be critical of here, but the above two really were the sore thumbs in this article.
/me gets back to real work and plans to dissect this more tonight
While he's at it, he can try the new Shootout benchmarks, which clearly show C++ in the lead, even when using his vaunted "-server" flag:
http://shootout.alioth.debian.org/gp4/benchmark.php?test=all...
...How old is this, anyway? There's no date in the article, but he's running gcc 3.3.1 and Java 1.4.2 on Fedora Test1, on a Pentium 4 laptop. The version of gcc places it in 2003, so I'd place more confidence the newer Shootout results.