Where are the links to the sources and is there a chance to compare the versions (Fortran vs Rust doing the same calculations)? I wasn't able to find them.
Where is the discussion of the library availability? The libraries for Fortran are maintained and improved during the decades?
You can find the code for the simple N-Body implemented in different languages here (the more advanced N-Body version with tides has not been released yet):
const N_PARTICLES: usize = 2;
...
fn main() {
...
while time <= time_limit {
integrator_leapfrog_part1(N_PARTICLES,
&mut x, &v, half_time_step);
...
fn integrator_leapfrog_part1(n_particles: usize,
x: &mut [[f64; 3]; N_PARTICLES],
v: &[[f64; 3]; N_PARTICLES],
half_time_step: f64) {
for i in 0..n_particles {
x[i][0] += half_time_step * v[i][0];
The way I understand it, with the C code the compiler during the compilation of the function doesn't know the size of the array, and with the Rust code it does, it is explicitly written? I refer to the difference between the capital letter constant and the plain variable. What would happen if the C compiler only knew that much too? that is, having the presence of the N_PARTICLES in all declarations? I can also imagine that just adding the proper compiler and linker options for C, not used in the makefile, can maybe give the benefit of that constant propagation in this particular case? I mean what happens when the "-flto" is added?
The N-Body on this site, with other implementations, has clearly faster C than Rust:
As noted elsewhere in here, the C implementation is using SSE, whereas the Rust implementation isn't. It's just as much of an unfair comparison as the one you're describing. :P
The OP doesn't even specify the rules for its own benchmark, as far as I understand doesn't verify the results? People here get NaNs?
If the NaNs are produced as results, then the OP code is not measuring the speed of calculations at all but the speed of the failed calculations. Which is especially problematic as the main argument is "attractive for the scientific community" "it guarantees memory safety." Which is presented as good because not having it "can produce random behaviors and affect the scientific interpretation of the results." If the result here is NaN the calculation doesn't even have to be performed after the first NaN that affects the result appears, as anything + NaN is NaN etc.
Edit: kibwen, did I write somewhere that I "refute" something? I gave a link to the rationale behind the "benchmarksgame" site. The rest is about the OP, surely not about your post.
> The programs on the site fit the rules defined on it, and the rules include the verification of the results
I'm not sure what you're refuting?
Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers).
The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC).
(And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.)
If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).
> The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C.
Today the benchmarks game is referred to from the Rust FAQ. What is one to do?
Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…"
> …repeated massive disclaimers that people should not take the performance results as a means of serious comparison.
Actually one small disclaimer --
"Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages."
Does current Rust stable provide equivalent SSE support?
If so then an equivalent Rust program would be welcome.
If not then it's hardly unfair to compare what actually is provided in current Rust stable to what actually is provided in an old C compiler (gcc 5.4.0).
It would be great if somebody have a fresh look at the C version, since I completely agree that it is not normal that its execution time is so far away from Fortran or Rust.
Edit: Looking into this a bit more, I think 'rustc -C opt-level=3' is optimizing out the actual integration. If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Not sure what 'gcc -O3 -ffast-math' is doing; I haven't looked at the disassembly.
Rust is able to avoid allocations and de-allocations in many areas where C programmers would manually use alloc and free. I must admit though, that's less relevant to scientific programming, where the heaviest work is mainly matrix matrix multiplication and the sizes of everything is known in advance.
Without reading through the paper, I expect work on the Rust/Fortran side is being vectorized and/or being kept in L1/L2 caches, whereas the C and Go compilers don't use SIMD instructions and/or are generating code that triggers cache misses.
Compiler optimisations can be very nuanced, especially in C. You might find that changing the code very slightly, or even just using a different combination of compiler options, can speed up the code a surprising amount.
Or rather, Rust is slower on those because SIMD is currently only supported on nightly, and the benchmarks game chooses to use stable Rust exclusively.
That is a slightly different thing: stable is a perfectly good choice, but one can still debate the fairness or not of being compared to C-with-language-extensions i.e. not standard C (a joke which I think you missed, in one of kibwen's replies).
There are places on the website that abbreviate to "C" or shorten to "programming language", but whenever practical it's more specific - "C gcc" & "programming language implementation".
We do not claim that Rust is so much performant than C, we just showed that Rust can be as fast as Fortran or C. And indeed, there must be something wrong with the C implementation. Pull requests with improvements are welcome:
The Intel compiler is relatively old, but mostly it seems to be a case of IEEE 754 conformance -- if enabled, things take longer, if disabled, they’re (probably) as fast in C as in Rust.
In my reading of the comment it seemed to imply that the commenter already had both clang and gcc installed on their computer since they were talking about the amount of instructions produced by each. Perhaps I misunderstood?
Interesting, the code looks more similar than I expected, yet the results are very different! Could it be that the difference is due to aliasing rules? It would be interesting to see what happens after adding a few restrict keywords here and there...
You would think that the presence of any unsafe block in the program's text, or even linkage with libc could result in (legitimate, non UB) aliasing in lots of code. Though I suppose that the private-linkage-by-default could help drastically reduce the scope it has to consider, maybe in some cases it can safely say that data is not aliased.
%xmm0 begins at 0, %xmm1 is 0.04, and %xmm2 is 3.6525E+8.
In other words, it's measuring how long it takes to count to 3.6525E+8 by adding 0.04 repeatedly. LLVM (which both Rust and clang use as a backend) optimizes away the actual work of the program.
GCC doesn't seem to optimize things out as aggressively, so that probably explains the difference: https://godbolt.org/g/lBRIIW
>It's crazy for me to see that C is much slower than rust.
It's not in the general case, but can be for particular code.
Unless the C program was written badly, the fact that Fortran is also faster seems to point to compiler optimizations not easy/possible in C (e.g. involving aliasing assurances).
For those who are looking at doing data analysis with Rust in astrophysics, I have a crate fitsio (https://crates.io/crates/fitsio) which wraps cfitsio into rust, allowing the reading and writing of .fits files.
I don't know the astrophysics domain at all but shouldn't there be more arguments for Go?
* Safer than C
* Almost as fast as C
* Good concurrency support
* High productivity due to simple abstractions and good tooling
Is the downside of a GC language really relevant? Does astrophysics suffer from "stop the world interrupts" or is it just because of the performance? The Go GC is already really fast.
During the past years of coding in C++ in the context of tensor networks in condensed matter physics, I have never encountered the case where a programming error which would have been caught in Rust or Go lead to incorrect results. Yes, there were plenty of crashes and plenty of hard to debug issues, but it was always obvious that something was wrong way before any results were calculated/printed/output.
Yes: Safety does not get you correct results where non-safety would have given you wrong results. Safety gives you correct results in some cases where non-safety would have given you no result. Those cases are easy to identify and can be solved using standard debugging techniques.
So safety gets you faster development: less debugging, easier parallelisation etc., but it does not give you correct results, nor does it give you those results by spending less computational time.
You legitimately can not imagine a scenario where a program is written which could give a result that appears correct but isn't due to a bug?
And you legitimately can't imagine how reducing the likelihood of making bugs by using a language+compiler which can identify/prevent more of them would help?
I definitely encountered such issues in solid state physics. But they are very rare. The likelihood of a crash or an obviously wrong result is perhaps three orders of magnitude higher.
I was thinking the same thing when I read this quote:
> For instance, a common source of error is the access to invalid memory regions, which produces random execution behaviors and affects the scientific interpretation of the results.
Such errors are astronomically more likely to cause crashes than silent errors in calculations.
Arithmetic in these contexts is done with floats. Any errors that would result in signed arithmetic issues would thrash so many other things about the problem that protections against UB would rarely be an issue. It's a legitimate possible use case, but not one to justify a choice in one language over another.
And your point about Fortran is great. I used that and C++ for my research and never once encountered an issue that Rust's guarantees would have helped with.
That's the key point. Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
A grad student (if paid properly) costs maybe 60k €/year. In comparison, we regularly spend more than 250k on new and faster computers, not including maintenance, the electricity bill etc. If you can make software even 50% faster by increasing development time, this is nearly always worth it in an academic setting. Note that the benchmark implementation shown here is a very simple example, normally computing jobs take days to weeks (multiplied by however many cores you can sensibly use) of CPU time.
Decreasing runtime also increases productivity a lot, since the turnaround time becomes shorter; waiting 3 versus 6 weeks for results is a noticeable difference.
Additionally, these tools rarely have safety concerns: In my own code, there is no "untrusted user input". There is correct user input (good) and incorrect user input (mostly it will crash, with common mistakes it will try to notify the user). Safety is handled by the operating system (such that you can’t overwrite someone else’s data, for example).
This means that the only advantage Go could have over C/C++ in academia is the "good concurrency support". However, concurrency in HPC is usually handled via OpenMP (shared memory) or MPI (distributed memory) parallelisation. This takes a while to get used to, but is very different (and in a sense much easier) than e.g. the typical case for a web server, where you wish to serve as many users as possible at the same time using as little CPU time as possible – a busily waiting loop is horrible in the latter case but perfectly fine in the former (under some circumstances).
So overall, languages are not interesting in the academic HPC crowd if they sacrifice speed for anything, which, incidentally, makes Rust also interesting, because that is precisely not the case (apparently, under some circumstances, etc.pp.).
> Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
I think this is what is most flawed in the paper. For (some) concurrent problems Go is about as fast as C [1].
But then again, I don't know what kind of computing requirements they have. Is there a reason why GRP computing isn't mentioned? They are really efficient and CUDA isn't that hard to learn.
> Rust allows the user to avoid common mistakes such as the access to invalid memory regions and race conditions
> I think this is what is most flawed in the paper. For (some) concurrent problems Go is about as fast as C [1].
Certainly, this was really only taken as a ballpark estimate of the performance difference. Looking at your link, it seems to have been slightly overestimating the difference, though the general point still stands: C is (in most cases) faster and there is (nearly) no case in which Go beats C.
> Is there a reason why GRP computing isn't mentioned?
> They are really efficient and CUDA isn't that hard to learn.
Sorry, not sure if you mean GPU instead of GRP in the first sentence. CUDA helps to some degree, but not always, e.g. when you are memory bandwidth bound (common in tensor networks in physics). I have no experience with Monte Carlo methods and can’t comment on whether they substantially benefit from CUDA; I know of at least one (Quantum Monte Carlo) code which runs much faster on a standard Xeon than on the Xeon Phi, though.
> They seem to care about safety.
Yes of course safety is nice to have and I’ll gladly learn Rust when I have some free time to get that safety at hopefully zero cost. But sacrificing performance is simply not competitive, if you can throw someone with gdb at the problem and get essentially the same "safety".
> if you can throw someone with gdb at the problem and get essentially the same "safety".
That's an odd tradeoff. That time of debugging could be significantly longer than just writing the software in a safe language. Seems like a bad tradeoff, and in many scenarios, bugs can lead to very bad things that you can't recover from. I've experienced all of these, having to fix them over long periods of time (not all my code, but sadly some was): data loss, concurrency (tough to debug in gdb), major memory leaks (even from std libs), corrupted data because of misused non null terminated c strings, array out of bound issues. Each one of these took weeks to track down, maybe because I'm not smart, which is a valid criticism; with Rust I've never had issues with any of those (2 years and running), and given that I'm not smart, it helps me by telling me where I got something wrong.
I think you misunderstand what "safety" refers to in the context of programming. It's not about flaws in your program being attacked, it's about writing correct programs.
I think you are completely missing the point here. The HPC world is mostly concerned with the speed at which you get results, safety means almost nothing in this domain. What people typically do is to test their code against known input/output and check the relative error.
It is easier to debug and improve a fast program when you can have some result in 1 day versus same algorithm implemented in a slower language that takes one week.
> languages are not interesting in the academic HPC crowd if they sacrifice speed for anything,
It depends. If it takes 8 weeks for the program to run using a high-level language, but 1 week for it to run using a faster language, dropping down to C to cut the running time down to 6.5 days doesn't help if it take several weeks to do it, and you are not sure that you won't have to rerun the program due to hard-to-find bugs.
Of course it's just a recommendation. Why would you think otherwise? To Rust, module name is just a bunch of characters, it doesn't recognize words anyhow.
It's kinda both. Since module names are identifiers, you can't use -s in them. You can still make it work, I believe, with the path attribute, but you'd need to do it on every mod declaration.
It's a recommendation that you'll find everywhere in rust. By default camelCasing will output compiler warnings. It's my only complaint about the language so far, underscores aren't very typing friendly.
Is the code in gravity_calculate_acceleration correct?
I think 'j' will always be 0 in the inner-loop, which seems to defeat the purpose of having a j variable.
> Compile C version with better optimization flags or ask an expert to tweak it for faster performance.
Part of the point they are trying to make is that you don't need to be an expert to reap performance benefits in certain languages. They explicitly state that they did not do things that C experts would know how to do. This is pretty reasonable in the context of astrophysics. The whole idea of the thing is that if there is a language that allows you to write faster code than C without having to "ask an expert to tweak it", you will be more productive using that language.
That said, your point 2 is entirely valid; the paper fails to demonstrate pretty much all of its claims, and it never makes a case for needing Rust's safe dynamic memory allocation features in a program that doesn't use dynamic allocation at all.
For whatever it's worth, the astrophysicists I most recently talked to wrote their simulations in Python (with SciPy or similar) or even something called IDL: https://en.wikipedia.org/wiki/IDL_(programming_language) and they also tend to have mad Bash skills for glueing different data-processing programs together.
You have a good criticism on my #1. I should drop "ask the expert" part but still maintain that when writing numerical computation software checking for compile flags is a reasonable request. Leave code as is, but still try -O3 / -Ofast.
At this point it is not the language expertise, but knowing about your tools.
Rust's safety features aren't tied at all to dynamic memory allocation. You can write a Rust program with pointers (into the stack, for example) with no dynamic memory allocation at all and still benefit immensely from the safety features.
This are not valid benchmarks at all.
What is the point to run N-Body sim with all particles set to 0.0 ?
Where is the result of the sim ?
Why it will be not validated at all ?
To make this look like real benchmark one will need to use the same start condition and then validate that result for all languages is the same.
It would be great to have C++ code using Eigen library for example ? To show the difference .
Also this is naive O(N^2) sim that could be speed up using Fast multipole method (FMM), but of course this would be much more complicated to do this in Rust or Go as in C++.
For me Rust has a lot of potential to write more secure code in places where it is needed but astrophysics is may be not one of them.
Not sure about language it self, my knowledge about Rust is limited.
But st is at least harder because we already have optimized libraries that can do this in C++.
Do some one know such library for Rust ?
It would be great too look at it.
As others have pointed out, adding insult to injury, in the Rust version the bulk of the actual calculation is optimized out completely! Adding a print at the end increases the runtime by an order of magnitude.
The Rust community is actively hurt by people doing things like this, no-one takes this type of work seriously.
Edit: please come back when you've replicated and validated at least something like a basic, well-understood finite difference solver of single-phase incompressible Navier-Stokes, and shown that it's either equivalently fast and much easier code, or significantly faster (very unlikely). Do note that a readable Fortran version is going to be ~400 lines, so there's not a terribly large room for improvement. Also, we already have a good language that gives us slower and much easier code, namely Python.
I just had a look at the Fortran code, and oh boy, it's not pretty, and far from "canonical" Fortran. I will submit a PR (hopefully later tonight) that makes it less ugly and slow.
(Protip: don't use if's inside tight loops. Just don't.)
A little less than a year ago I worked on a bioinformatics project in rust, and I think it came out very well. My perception was that bio is a field where most practitioners rely on high speed code written by someone else that can be called from their perl or python code. I see something very empowering with rust, where people can write high performance analysis without having to shoehorn their workload into an existing toolkit or framework.
That said, while I'm really excited to see rust go more places, astrophysics seems like one of very few fields where practitioners learn a high performance language as a matter of course, and so rust may have a less empowering effect for them. I would wager that after paying the upfront cost to learn it, students and researchers may see improved iteration time from fewer bugs. But it seems like rust may have an incremental improvement to offer which is not as powerful as the transformative nature of its potential in other computationally intensive fields.
a) access to invalid memory regions, b) dangling pointers and attempts to free already freed memory, c) memory leaks
and, d) race conditions.
The first three benefits also come with essentially any GC'ed language. d) is interesting. What are the costs that come with such a guarantee? Presumably it prohibits certain kinds of parallelism? Are the ownership/borrowing mechanics interesting programming tools, or are they a hindrance?
People talk a lot about static typing, and I can see the benefits for critical/user-facing applications, but not for numerical code. The nightmare with numerical code is finding out that I forgot a minus sign somewhere, and that it invalidates my last 6 months of published results.
> Are the ownership/borrowing mechanics interesting programming tools, or are they a hindrance?
Like any sort of static analysis, some see them as a tool, some see them as a hindrance ;)
> also come with essentially any GC'ed language.
Ownership and borrowing can extend to arbitrary resources, not just memory.
Also, C and D aren't correct; while Rust helps with memory leaks, it absolutely does not prevent them, and Rust prevents _data races_, a specific form of race condition, but cannot prevent race conditions generally.
114 comments
[ 10.7 ms ] story [ 646 ms ] threadWhere is the discussion of the library availability? The libraries for Fortran are maintained and improved during the decades?
https://github.com/marblestation/benchmark-leapfrog
And indeed, Fortran has decades of advantage in terms of libraries.
The C code:
The Rust code: The way I understand it, with the C code the compiler during the compilation of the function doesn't know the size of the array, and with the Rust code it does, it is explicitly written? I refer to the difference between the capital letter constant and the plain variable. What would happen if the C compiler only knew that much too? that is, having the presence of the N_PARTICLES in all declarations? I can also imagine that just adding the proper compiler and linker options for C, not used in the makefile, can maybe give the benefit of that constant propagation in this particular case? I mean what happens when the "-flto" is added?The N-Body on this site, with other implementations, has clearly faster C than Rust:
https://benchmarksgame.alioth.debian.org/u64q/performance.ph...
As noted elsewhere in here, the C implementation is using SSE, whereas the Rust implementation isn't. It's just as much of an unfair comparison as the one you're describing. :P
https://benchmarksgame.alioth.debian.org/why-measure-toy-ben...
The OP doesn't even specify the rules for its own benchmark, as far as I understand doesn't verify the results? People here get NaNs?
If the NaNs are produced as results, then the OP code is not measuring the speed of calculations at all but the speed of the failed calculations. Which is especially problematic as the main argument is "attractive for the scientific community" "it guarantees memory safety." Which is presented as good because not having it "can produce random behaviors and affect the scientific interpretation of the results." If the result here is NaN the calculation doesn't even have to be performed after the first NaN that affects the result appears, as anything + NaN is NaN etc.
Edit: kibwen, did I write somewhere that I "refute" something? I gave a link to the rationale behind the "benchmarksgame" site. The rest is about the OP, surely not about your post.
I'm not sure what you're refuting?
Verifying that the results match is a necessary but insufficient quality for ensuring comparability. If the algorithm could be the same in each language but isn't (e.g. quicksort vs. bogosort), then that's not a valid comparison if your objective is to determine the overhead imposed by the language implementation itself (and if you're not trying to determine language implementation overhead, then what are you measuring?). Likewise if the implementation details could be the same in each language but aren't (e.g. if one uses 64-bit integers and the other uses 32-bit integers).
The computer language benchmarks game was initially conceived to determine a ballpark for how slow interpreted and managed languages are compared to C. Quantifying the overhead of interpreters and runtimes is its raison d'etre, and it shows. When it comes to comparing low-level systems languages that have no runtime to speak of, the best it can do is attempt to quantify the quality of each backend's code generator (it's a missed opportunity that it doesn't include Clang for comparison with GCC).
(And yes, I understand that the benchmarks game contains repeated massive disclaimers that people should not take the performance results as a means of serious comparison. Internet commentators remain undeterred.)
If you're just trying to argue that the methodology used in the OP is poor, then obviously we're in agreement (was there ever any doubt?).
Today the benchmarks game is referred to from the Rust FAQ. What is one to do?
https://www.rust-lang.org/en-US/faq.html#performance
Back in the previous millenia -- "[Doug Bagley's] goal was to compare all the major scripting languages. Then [Doug Bagley] started adding in some compiled languages for comparison…"
http://web.archive.org/web/20010125021400/http://www.bagley....
Actually one small disclaimer --
"Non-motivation: We are profoundly uninterested in claims that these measurements, of a few tiny programs, somehow define the relative performance of programming languages."
There's much more about PL benchmarks in general.
If so then an equivalent Rust program would be welcome.
If not then it's hardly unfair to compare what actually is provided in current Rust stable to what actually is provided in an old C compiler (gcc 5.4.0).
Yes, stable Rust provides equivalent SSE support to the language specified as ISO 9899:2011. :P
Apparently this is just Rust stable not providing the functionality.
In table 1
0m13.660s 0m14.640s 2m32.910s 4m26.240sThey did have a note that C could be faster if language specific features could be used.
Is rust really that performant compared to C? Or did they neglect to even bother checking C's performance?
Which would be the equivalent of writing a comparison between a sledgehammer and jackhammer, without turning the latter on.
https://github.com/marblestation/benchmark-leapfrog
It would be great if somebody have a fresh look at the C version, since I completely agree that it is not normal that its execution time is so far away from Fortran or Rust.
rustc 1.15.1 (021bd294c 2017-02-08)
Edit: Looking into this a bit more, I think 'rustc -C opt-level=3' is optimizing out the actual integration. If I put a println!("{}", x[0][0]) at the end, I end up with 1m43s. Not sure what 'gcc -O3 -ffast-math' is doing; I haven't looked at the disassembly.
It's surprising that gcc does such a poor job at -O3 though.
Yes, without that it's basically cheating. There is now a GitHub issue: https://github.com/marblestation/benchmark-leapfrog/issues/6
The submitter writes: "final output [for Rust] is 134 seconds, and also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]] (batman)"
Rust is able to avoid allocations and de-allocations in many areas where C programmers would manually use alloc and free. I must admit though, that's less relevant to scientific programming, where the heaviest work is mainly matrix matrix multiplication and the sizes of everything is known in advance.
Without reading through the paper, I expect work on the Rust/Fortran side is being vectorized and/or being kept in L1/L2 caches, whereas the C and Go compilers don't use SIMD instructions and/or are generating code that triggers cache misses.
http://benchmarksgame.alioth.debian.org/u64q/compare.php?lan... -- in this benchmarks C in most cases faster than Rust, but only in few they have the same performance.
Edit: Looks like using -Ofast instead of -O3 makes a big difference: https://news.ycombinator.com/item?id=13634119
Offering context? ;-)
2. How many years already has SIMD not supported on stable been posted as a response in these discussions ?
2. 1.75, which is coincidentally how long a stable version of Rust has existed. :P
https://github.com/marblestation/benchmark-leapfrog
Clang produces practically the same between the two levels.
On OS X, so can't test if this actually makes a performance difference.
The `time` command is part of base OS X.
Also the output is [[NaN, NaN, NaN], [NaN, NaN, NaN]], which is a bit worrying...
Not sure how much this affects the benchmark, but it'd probably be smart to randomise the starting positions.
In other words, it's measuring how long it takes to count to 3.6525E+8 by adding 0.04 repeatedly. LLVM (which both Rust and clang use as a backend) optimizes away the actual work of the program.
GCC doesn't seem to optimize things out as aggressively, so that probably explains the difference: https://godbolt.org/g/lBRIIW
Never trust a benchmark you didn't falsify yourself.
Benchmarking is hard.
It's not in the general case, but can be for particular code.
Unless the C program was written badly, the fact that Fortran is also faster seems to point to compiler optimizations not easy/possible in C (e.g. involving aliasing assurances).
https://github.com/astrogo/fitsio
(a pure Go version)
* Safer than C
* Almost as fast as C
* Good concurrency support
* High productivity due to simple abstractions and good tooling
Is the downside of a GC language really relevant? Does astrophysics suffer from "stop the world interrupts" or is it just because of the performance? The Go GC is already really fast.
So safety gets you faster development: less debugging, easier parallelisation etc., but it does not give you correct results, nor does it give you those results by spending less computational time.
And you legitimately can't imagine how reducing the likelihood of making bugs by using a language+compiler which can identify/prevent more of them would help?
I'll leave you to it then.
> For instance, a common source of error is the access to invalid memory regions, which produces random execution behaviors and affects the scientific interpretation of the results.
Such errors are astronomically more likely to cause crashes than silent errors in calculations.
Fortran 2008 is already quite modern in language features and doesn't suffer from C safety issues.
EDIT: typo.
Then again, Fortran would cover these scenarios and is better established.
And your point about Fortran is great. I used that and C++ for my research and never once encountered an issue that Rust's guarantees would have helped with.
That's the key point. Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
A grad student (if paid properly) costs maybe 60k €/year. In comparison, we regularly spend more than 250k on new and faster computers, not including maintenance, the electricity bill etc. If you can make software even 50% faster by increasing development time, this is nearly always worth it in an academic setting. Note that the benchmark implementation shown here is a very simple example, normally computing jobs take days to weeks (multiplied by however many cores you can sensibly use) of CPU time.
Decreasing runtime also increases productivity a lot, since the turnaround time becomes shorter; waiting 3 versus 6 weeks for results is a noticeable difference.
Additionally, these tools rarely have safety concerns: In my own code, there is no "untrusted user input". There is correct user input (good) and incorrect user input (mostly it will crash, with common mistakes it will try to notify the user). Safety is handled by the operating system (such that you can’t overwrite someone else’s data, for example).
This means that the only advantage Go could have over C/C++ in academia is the "good concurrency support". However, concurrency in HPC is usually handled via OpenMP (shared memory) or MPI (distributed memory) parallelisation. This takes a while to get used to, but is very different (and in a sense much easier) than e.g. the typical case for a web server, where you wish to serve as many users as possible at the same time using as little CPU time as possible – a busily waiting loop is horrible in the latter case but perfectly fine in the former (under some circumstances).
So overall, languages are not interesting in the academic HPC crowd if they sacrifice speed for anything, which, incidentally, makes Rust also interesting, because that is precisely not the case (apparently, under some circumstances, etc.pp.).
> Judging by the (wrong) benchmark above where both C and Go seem to do some work, Go is about half as fast as C.
I think this is what is most flawed in the paper. For (some) concurrent problems Go is about as fast as C [1].
But then again, I don't know what kind of computing requirements they have. Is there a reason why GRP computing isn't mentioned? They are really efficient and CUDA isn't that hard to learn.
> Rust allows the user to avoid common mistakes such as the access to invalid memory regions and race conditions
They seem to care about safety.
[1] https://benchmarksgame.alioth.debian.org/u64q/compare.php?la...
Certainly, this was really only taken as a ballpark estimate of the performance difference. Looking at your link, it seems to have been slightly overestimating the difference, though the general point still stands: C is (in most cases) faster and there is (nearly) no case in which Go beats C.
> Is there a reason why GRP computing isn't mentioned?
> They are really efficient and CUDA isn't that hard to learn.
Sorry, not sure if you mean GPU instead of GRP in the first sentence. CUDA helps to some degree, but not always, e.g. when you are memory bandwidth bound (common in tensor networks in physics). I have no experience with Monte Carlo methods and can’t comment on whether they substantially benefit from CUDA; I know of at least one (Quantum Monte Carlo) code which runs much faster on a standard Xeon than on the Xeon Phi, though.
> They seem to care about safety.
Yes of course safety is nice to have and I’ll gladly learn Rust when I have some free time to get that safety at hopefully zero cost. But sacrificing performance is simply not competitive, if you can throw someone with gdb at the problem and get essentially the same "safety".
That's an odd tradeoff. That time of debugging could be significantly longer than just writing the software in a safe language. Seems like a bad tradeoff, and in many scenarios, bugs can lead to very bad things that you can't recover from. I've experienced all of these, having to fix them over long periods of time (not all my code, but sadly some was): data loss, concurrency (tough to debug in gdb), major memory leaks (even from std libs), corrupted data because of misused non null terminated c strings, array out of bound issues. Each one of these took weeks to track down, maybe because I'm not smart, which is a valid criticism; with Rust I've never had issues with any of those (2 years and running), and given that I'm not smart, it helps me by telling me where I got something wrong.
Be safe out there people...
It is easier to debug and improve a fast program when you can have some result in 1 day versus same algorithm implemented in a slower language that takes one week.
It depends. If it takes 8 weeks for the program to run using a high-level language, but 1 week for it to run using a faster language, dropping down to C to cut the running time down to 6.5 days doesn't help if it take several weeks to do it, and you are not sure that you won't have to rerun the program due to hard-to-find bugs.
> If you’re using more than one word in your filename, use an underscore: hello_world.rs rather than helloworld.rs.
Is the use of underscores a recommendation or a limitation?
I don't see how Go would be a better option than it.
1. Compile C version with better optimization flags or ask an expert to tweak it for faster performance.
2. Include sample run on some known / meaningful data to test that software runs correctly (not only fast).
IMO rust doesn't have to be faster than C to make a valuable article. If it is as fast or almost as fast it is still a good data point.
Part of the point they are trying to make is that you don't need to be an expert to reap performance benefits in certain languages. They explicitly state that they did not do things that C experts would know how to do. This is pretty reasonable in the context of astrophysics. The whole idea of the thing is that if there is a language that allows you to write faster code than C without having to "ask an expert to tweak it", you will be more productive using that language.
That said, your point 2 is entirely valid; the paper fails to demonstrate pretty much all of its claims, and it never makes a case for needing Rust's safe dynamic memory allocation features in a program that doesn't use dynamic allocation at all.
For whatever it's worth, the astrophysicists I most recently talked to wrote their simulations in Python (with SciPy or similar) or even something called IDL: https://en.wikipedia.org/wiki/IDL_(programming_language) and they also tend to have mad Bash skills for glueing different data-processing programs together.
At this point it is not the language expertise, but knowing about your tools.
Rust's safety features aren't tied at all to dynamic memory allocation. You can write a Rust program with pointers (into the stack, for example) with no dynamic memory allocation at all and still benefit immensely from the safety features.
It appears that it doesn't run correctly (rust optimizes out the actual integration, and values are all set to NaN!)
https://github.com/marblestation/benchmark-leapfrog/issues/6
This are not valid benchmarks at all. What is the point to run N-Body sim with all particles set to 0.0 ?
Where is the result of the sim ?
Why it will be not validated at all ?
To make this look like real benchmark one will need to use the same start condition and then validate that result for all languages is the same.
It would be great to have C++ code using Eigen library for example ? To show the difference .
Also this is naive O(N^2) sim that could be speed up using Fast multipole method (FMM), but of course this would be much more complicated to do this in Rust or Go as in C++.
Not sure about language it self, my knowledge about Rust is limited. But st is at least harder because we already have optimized libraries that can do this in C++.
Do some one know such library for Rust ? It would be great too look at it.
The Rust community is actively hurt by people doing things like this, no-one takes this type of work seriously.
Edit: please come back when you've replicated and validated at least something like a basic, well-understood finite difference solver of single-phase incompressible Navier-Stokes, and shown that it's either equivalently fast and much easier code, or significantly faster (very unlikely). Do note that a readable Fortran version is going to be ~400 lines, so there's not a terribly large room for improvement. Also, we already have a good language that gives us slower and much easier code, namely Python.
(Protip: don't use if's inside tight loops. Just don't.)
That said, while I'm really excited to see rust go more places, astrophysics seems like one of very few fields where practitioners learn a high performance language as a matter of course, and so rust may have a less empowering effect for them. I would wager that after paying the upfront cost to learn it, students and researchers may see improved iteration time from fewer bugs. But it seems like rust may have an incremental improvement to offer which is not as powerful as the transformative nature of its potential in other computationally intensive fields.
a) access to invalid memory regions, b) dangling pointers and attempts to free already freed memory, c) memory leaks and, d) race conditions.
The first three benefits also come with essentially any GC'ed language. d) is interesting. What are the costs that come with such a guarantee? Presumably it prohibits certain kinds of parallelism? Are the ownership/borrowing mechanics interesting programming tools, or are they a hindrance?
People talk a lot about static typing, and I can see the benefits for critical/user-facing applications, but not for numerical code. The nightmare with numerical code is finding out that I forgot a minus sign somewhere, and that it invalidates my last 6 months of published results.
Actually it enables parallelism you couldn't do before, because now you're confident the compiler will prove that what you're doing is safe.
Like any sort of static analysis, some see them as a tool, some see them as a hindrance ;)
> also come with essentially any GC'ed language.
Ownership and borrowing can extend to arbitrary resources, not just memory.
Also, C and D aren't correct; while Rust helps with memory leaks, it absolutely does not prevent them, and Rust prevents _data races_, a specific form of race condition, but cannot prevent race conditions generally.