You have to hit F5 to reload page, since otherwise the old page will be displayed from browser cache. Probably something wrong with page cache expiration settings.
With Fortran this is expected, since for language semantics the compiler can perform more assumptions compared to C, resulting into more optimizations.
For the shootout, all the source code is visible to the compiler (though at least GCC needs to be told so with a flag). So I don't think there's much more assumptions that can be made.
How does this matter when the compiler can see all the source code? It can see what the pointers are pointing to and determine if there is aliasing.
Feel free to point out the source code in the shootout where this would somehow not be the case. Shouldn't be hard to find if you focus on the benchmarks where Fortran beats C heavily.
Additionally, the argument about restrict is flawed because it can be added to the C code too, so it can't be an advantage of Fortran.
It is impossible write a compiler that reads a C source code and for every point in program decides if two names visible from that point cannot alias each other. You cannot do that even under assumption that every code path may be visited.
I agree it is impossible in the general case (it's obviously impossible when input data controls program flow, for example). But I don't think that has much meaning for the benchmarks under consideration, which, by the way, one would expect to sidestep the obvious pitfalls here.
The impossibility of solving aliasing in the general case doesn't stop C compiler writers from implementing alias analysis.
I assume xyzzyz was referring to the undecidability of the halting problem, not just user input. Comparisons to the halting problem are often made inappropriately, as would be the case here, since the compiler doesn't need to answer whether an arbitrary program contains aliasing, but only identify programs that admit arguments (often simple ones) demonstrating that they don't alias. It's still undecidable, though.
I pointed out specifically that all source code is available exactly for this reason. The original claim was that "for Fortran this is expected" which isn't true at all in this condition. So aliasing information can't explain the performance difference.
From a simple look, the Fortran code seems to have an additional optimization that allows it to avoid a lot of checking for overflows in the innermost loop.
binary-trees is rough to beat C on. It involves allocating and traversing some pretty gigantic data structures, and the C code is optimized by using region-style allocation to avoid all of the individual calls to free (or garbage collections in a GC'd language). From the CPU usage, it also looks like they also haven't parallelized the Fortan version yet, though even at ideal speedup it'll still lose.
I've been doing some tuning work on our parallel dialect of ML, and many of the C programs are fairly well-optimized, though mandelbrot still has some room for improvement.
These reports are worse than useless. Any person with good understanding of computing knows that implementation plays a huge part in it. Why people wasting their time comparing speed of different programming languages while they should focus on different implementations. I guess the latter one is difficult so people all try to avoid it, and the first one is just so easy to do and everybody is doing it.
That fact stands out immensely. The Scala code and java code should be equal as the first can be written exactly like the second. Don't even see Clojure. Hate to see what times they get there.
Except Scala has no 'break' (that isn't a hack using exceptions), this makes imitating imperative style difficult and some syntaxical sugar should no doubt ensure that Scala will always be slower than Java.
clang is waiting for someone like you, who wants to spend their time building language implementations on the LLVM toolchain and measuring program runtimes -
Sorry, but your Ruby code will never the same speed as C code, even if it is the same implementation. We'd all like to believe all programming languages are equal, but that's not the case.
Which of course the web site freely acknowledges. At the bottom of that page states the real question is "Which of these programming language implementations have the fastest benchmark programs?"
This is a really terribly done study, with a misleading title to boot.
First, even if we take the conclusion as given, "Fortran" isn't faster than "C". The median of one set of programs, compiled with one Fortran compiler with one set of options and run on one processor, is faster than the median of another set of programs compiled with one C compiler with one set of one and run on that processor. This tells us approximately nothing.
If you only expand your consideration to include the whole distribution of programs they tested, GCC actually looks better than iFort (eg. there is a single program which ran 2x faster with iFort, but multiple programs ran more than 2x slower). Median is a spectacularly poor choice of metric, especially with so few samples in the distribution.
But also: Intel also makes a C compiler. Why wasn't that used? How were these benchmark programs chosen? Etc.
This study says close to nothing about "Fortran" and "C", or about any of the other languages involved.
This is a really terrible criticism of the linked page, with a misleading first statement to boot. Where do you see the word study? The title at the top of the page says "Computer Language Benchmarks Game". How is calling it a benchmark game misleading? That's what it is, and it doesn't pretend to be anything else - it is a sorted list of benchmarks run on one particular configuration.
They conducted a series of experiments, and reported their results. If that's not a "study", I don't know what is. The fact that the authors acknowledge that it's a "game" doesn't change that. What is misleading is the title on the link: "Intel Fortran now 20% faster than C".
This study says close to nothing about "Fortran" and "C", or about any of the other languages involved.
Sorry, but I disagree here. It's one of the best resources out there for getting numbers on the relative performance of languages, despite the pitfalls. The implementations of popular languages are well tuned (sometimes extremely so), the used compilers and settings are quite sane, and the benchmark programs are fairly well distributed in functionality.
The median is a reasonable choice because it's less sensitive to outliers and was probably chosen for that reason. "Spectacularly poor metric" seems to be an opinion, not a fact.
I admit it would have been better to compare Intel C with Intel Fortran. Why wasn't this done? Perhaps some of the submitted programs don't compile correctly with Intel C, or perhaps Intel C doesn't actually perform well on them. Who knows.
But to say that this is a "terribly done study" because of that is doing it injustice.
Median is a bad choice, because it completely ignores just how bad the worst performance holes are for a compiler. Discarding outliers is bad methodology in this case.
"7 descriptive values" - Indeed that is true. But the median is used for the ranking and that is why I criticise it.
The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.
What is bad about median is that a compiler that performs well on most benchmarks but then falls off a cliff for many other, but less than half, performs well altogether.
I think the distance-weighted estimator would serve the purpose of the ranking better.
This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result.
I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey.
You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?
Again, that's what's bad about the median as the sole characterisation of the measurements - but the median is not presented as the sole characterisation of the measurements.
The outlier Fortran measurements are there for all to see.
Some variation on the distance-weighted estimator might work here -
The median really is a spectacularly bad choice when there are so few data points, and especially when thinking about performance. If A and B get 3x faster, but C gets 3x slower, then your overall workload gets slower, even though the "median performance improvement" is 3x. Obviously the effect is magnified with only 3 data points, but the number of points used in the examples on that page is not so much greater as to eliminate this effect.
Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C).
What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be).
If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.
Mean would be a better choice if a typical real-world application were split approximately equally between each of these tasks. Which is of course patently untrue, particularly so because writing number-crunching code in Fortran does not prevent you from writing your hash-table code in something else (note how the best C binary-trees code is actually calling an external library).
Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.
Fortran was designed to munch numbers. As most programs are written by people who will not spend much time getting that last 3x of performance out, a language like Fortran makes sense for the number crunchers. Its easier to screw up in C. These kinds of tests are not all that great, as the code is optimized. Its un-optimized code that matters, since that powers the world.
Is there some reason that gfortran from the GCC project is not on the list of compilers for this language shootout? Instead only the branded "Intel Fortran" compiler shows up. Hmm.
52 comments
[ 24.5 ms ] story [ 313 ms ] threadFeel free to point out the source code in the shootout where this would somehow not be the case. Shouldn't be hard to find if you focus on the benchmarks where Fortran beats C heavily.
Additionally, the argument about restrict is flawed because it can be added to the C code too, so it can't be an advantage of Fortran.
The impossibility of solving aliasing in the general case doesn't stop C compiler writers from implementing alias analysis.
p2 = function(p1);
Do p1 and p2 alias? Note: function() is in a *.so file, no source available.
How many times do you have all of the source code available for analysis?
I've been doing some tuning work on our parallel dialect of ML, and many of the C programs are fairly well-optimized, though mandelbrot still has some room for improvement.
http://shootout.alioth.debian.org/demo2/which-programming-la...
http://shootout.alioth.debian.org/help.php#llvm
Hint: If you want to check for some text on a web page, use page search.
How about Intel Fortran vs. Intel C ? How about gcc C vs. gcc Fortran ?
Hyper-optimized compiler for Intel architecture beats compiler designed to be portable to many, many target architectures? Not very shocking.
First, even if we take the conclusion as given, "Fortran" isn't faster than "C". The median of one set of programs, compiled with one Fortran compiler with one set of options and run on one processor, is faster than the median of another set of programs compiled with one C compiler with one set of one and run on that processor. This tells us approximately nothing.
If you only expand your consideration to include the whole distribution of programs they tested, GCC actually looks better than iFort (eg. there is a single program which ran 2x faster with iFort, but multiple programs ran more than 2x slower). Median is a spectacularly poor choice of metric, especially with so few samples in the distribution.
But also: Intel also makes a C compiler. Why wasn't that used? How were these benchmark programs chosen? Etc.
This study says close to nothing about "Fortran" and "C", or about any of the other languages involved.
Can you find the statement "Intel Fortran now 20% faster than C" anywhere on the linked website?
> the authors acknowledge that it's a "game"
See http://shootout.alioth.debian.org/dont-jump-to-conclusions.p...
Sorry, but I disagree here. It's one of the best resources out there for getting numbers on the relative performance of languages, despite the pitfalls. The implementations of popular languages are well tuned (sometimes extremely so), the used compilers and settings are quite sane, and the benchmark programs are fairly well distributed in functionality.
The median is a reasonable choice because it's less sensitive to outliers and was probably chosen for that reason. "Spectacularly poor metric" seems to be an opinion, not a fact.
I admit it would have been better to compare Intel C with Intel Fortran. Why wasn't this done? Perhaps some of the submitted programs don't compile correctly with Intel C, or perhaps Intel C doesn't actually perform well on them. Who knows.
But to say that this is a "terribly done study" because of that is doing it injustice.
http://shootout.alioth.debian.org/u64q/which-programming-lan...
For each programming language implementation, that web page shows 7 descriptive values - not just the median!
As I've already said - The outliers are not discarded. The outliers are shown both in the chart and in the table.
The worst value for Fortran, one of the included values, is more than five times worse than that for C. This suggests (not proves) that the Intel Fortran compiler has worse performance holes than gcc C that show up for some, but not all, of these benchmarks.
You criticise but don't even suggest an alternative, let alone evaluate how well or badly that alternative compares to the median.
That's empty criticism.
I think the distance-weighted estimator would serve the purpose of the ranking better.
http://en.wikipedia.org/wiki/Distance-weighted_estimator
This measure finds a measure of the centre of the distribution that gives less weight to items further away from that centre without discarding them. In this way, the measure is not dominated by outliers, but they still contribute to the final result.
I don't say that this is the best measure, only one that is better. A factor analysis might get at the extent to which the different benchmarks are doing the same thing. Some attempt to find measures of how these activities are represented in larger pieces of code might suggest weights for the benchmarks (there's a literature on analysing loads that might be relevant here). But this would make their analysis more complex and more prone to bias leaking into their analysis, so I don't say they ought to do this; my point is that median is simply a bad population measure for the purposes of their survey.
You have argued against me without trying to defend the choice of median as the measure used for the ranking. Is that denialism?
It isn't any kind of ism.
> What is bad about median is that...
Again, that's what's bad about the median as the sole characterisation of the measurements - but the median is not presented as the sole characterisation of the measurements.
The outlier Fortran measurements are there for all to see.
Some variation on the distance-weighted estimator might work here -
http://shootout.alioth.debian.org/u64q/which-language-is-bes...
Adding to this is that significant slowdowns, the "outliers" that you're keen to disregard, tend to dwarf performance improvements that the median might show (because of the multiplicative nature of these values: if A/fortran is 10x faster than A/C, and B/fortran is 10x slower than B/C, then A+B/fortran is 5x slower than A+B/C).
What's really right is to show the complete distribution of speedups and slowdowns (as one of the detail pages on that site does). If you look at the actual distribution for the comparison of C and Fortran in question, C actually comes out looking much better than Fortran does (which shows precisely how misleading using the median can be).
If you must to boil it down to a single number, there are several choices better than the median. I would be happier with a mean, despite its imperfections.
The page the OP linked DOES show all the distribution - that's the beauty of box plots.
Comparison of medians is nice because it shows that for approximately half of fairly diverse tasks one language is faster than another, which imo is a much more interesting statistic.