Yes i know that benchmarks mean fairly little, but Ruby here is WAY off the chart. Wasn't 2.0 suppose to improve on performance? And it is even worst then PHP.
Exactly - the flexibility of Ruby still means it's going to do things that align with its core strengths better. No language is one-size-fits-all anyway.
Maybe is fine if you have a controlled traffic, but with better speed you can manage more traffic with the same hardware. That's why Facebook had to build a PHP compiler at some point.
Most of your time is probably spent in I/O wait.
For most applications, it's not that important to optimise the 5% waiting time as opposed to the 95% waiting time
That's not entirely true, for instance a ranking algorithm would use way more CPU that I/O (in most cases you load your data in memory anyway)
In any case, How you would justify using Ruby over another script language like Python, Perl, etc? All of them are comparable in terms of productivity and seems to be orders of magnitude faster.
The majority of applications don't do a lot of number crunching.
I'd recommend Ruby because it's readable, it has a good tool ecosystem a great community and design decisions that please me. Python and Perl are just not what I'd like to see in a language. I will use them if I have to, but given the chance I'll use Ruby.
For probably almost every website/webapp in the world, the amount of servers you have is 1. Facebook is not a typical application.
Besides, code is written for more than just websites. Chef/puppet/cfengine, one-off scripts you write, etc. There's a whole lot of these types of things that don't matter if your code takes 0.01 seconds or 0.05 seconds.
"For probably almost every website/webapp in the world, the amount of servers you have is 1."
Still if I can do the same thing in a cheaper server with the same level of productivity, why wouldn't I do that?
"There's a whole lot of these types of things that don't matter if your code takes 0.01 seconds or 0.05 seconds."
Yeah but why would you prefer Ruby instead of PHP/Python/Perl for those things then?
Because they prefer Ruby? I personally wouldn't choose PHP or Perl unless I had no other choice. Nothing wrong with other people choosing them, but they aren't for me. And given the choice between Python and Ruby, I'll choose Ruby unless there is a compelling reason not to. And saving a few hundred dollars on a commodity server would never be compelling enough for me.
Edit: And when I actually need better performance from a specific piece of code, I can and do drop down to C.
I haven't taken a look at the actual programs, but given the nature of the benchmarks the result can probably be interpreted as 'numpy is still quite fast' also?
Of course an advantage of python over ruby (speed-wise) is that you can leverage numpy if you need to do fast matrix operations, whereas ruby has no such library (that I'm aware of).
I use Ruby ALL the time, and I've never once thought, "I wish this was executing faster".
Yes, perhaps it could be more efficient, but I don't use Ruby for speed. The way that I see it, Ruby exchanges execution time for programmer time. Alternatively, If I want something to be fast, I'll spend a few days writing it in C/C++/Assembler, but I'd expect to spend at least an order of magnitude more time writing code.
>I use Ruby ALL the time, and I've never once thought, "I wish this was executing faster".
That's because you (subconsciously or because of your specific work needs) constrain yourself with problems that don't need the speed.
But who says that having a X times faster Ruby is not very desirable?
- For one, a cash-strapped startup could deploy RoR instances with fewer boxes, saving a lot of hosting costs.
(ASIDE: People compare server costs to the salary of a developer to trivialize them. What they always assume though is that the developer makes a US sized salary (in parts of the world, $500/month IS a nice salary). Or they assume he someone is even paid in the first place (think a ramen startup, a university student deep in debt wanting to launch something on the cheap etc).
- Second, people might like to write a game in Ruby, a number-crunching app, a drawing app, etc. Why be constrained by the language and need to employ another for the critical parts? This puts a barrier to a programmer (in code complexity, in having 2 codebases, in time needed to manage memory and such). Barriers mean: you can do fewer interesting stuff than what could be possible.
>The way that I see it, Ruby exchanges execution time for programmer time.
Sure, by why it has to be that way? Nothing inherent in dynamic languages make it so. Of course they will be slower than C, but they can be 10 or 20 times faster than what they are NOW.
Pure V8, Smalltalk VMs, JVM languages etc are consistently much faster than pure (no C delegation) Ruby or Python. Ruby has a lot of room to get much faster.
Well, if it ran any faster all those Rails sites would get compromised even quicker, so surely this is a good thing? Brogrammer security through performance woes.
It's very relevant for some types of programs. For web development, unless you're building the next Twitter, not so important...
...now if you're building a chess engine, that's a different story.
(I just take chess as an example as it's my latest hobby. It's the first and only project where I found it necessary to switch to C because of performance considerations (switched from C#).)
For the various tasks involved in a chess engine Python and Ruby are likely to either be performance equivalent (in that performance doesn't really matter) or entirely unsuitable.
Ruby 2.0 was JUST released. It'll get faster, just like it happened for 1.8 and 1.9.
Python was released in what? 2008? 2009?
I don't want to be that person, but if we're going to compare language based on benchmarks that are pretty irrelevant to modern uses already (like the CLBG is), at least let's pick "equivalent" languages and let's try avoiding the obvious biases.
JRuby is growing stronger and so is Rubinius. As those implementations grow and mature, they'll get faster because they offer solutions in terms of runtime that piggyback on technologies that are getting better and faster at optimizing the bytecode produced (JVM, LLVM in that case).
I do think Ruby will get faster eventually. Maybe not to the point that Ruby suddenly becomes "fast" of course, but by a margin that can make a difference. This is only my opinion (backed by some benchmarks in -for now- particular circumstances) of course, but I'm hopeful that the situation will improve.
Except their benchmarks are full of poo poo. The Python pidigits benchmark uses a third party C extension for high performance math, whereas the Ruby 2.0 pidigits benchmark is pure Ruby.
Of course someone will lose a fight when their opponent is wearing a jetpack and rocket powered shoes.
Other languages in those pidigits benchmarks (Perl, Lua, PHP, Racket, Lisp SBCL and probably more) also use the GMP maths library. So there is nothing stopping Ruby benchmarks also using it as well (just need to request relevant Ruby GMP library are installed on the Alioth benchmark machines).
Because someone can't change their opinion in 4 years..
I hadn't looked at the source code back then. Only now when this particular variant of the benchmark shows some rather unrealistic results. Another example (binary trees, I believe) has Ruby using a single process whereas the Python example is using multiple processes to spread the load, so even though Ruby wins on CPU time, Python wins with a slightly lower wall-clock time.
And making the Ruby pidigits benchmark use GMP too? That defeats the point of benchmarking one language against another if you can just call out to C libraries for everything (hey, there's an idea.. Ruby could be parallel with C if we just did that). It should be pure language against pure language otherwise it's totally meaningless (and, well, it is, as it stands).
Generally I agree with you on this however agreeing on a level playing field may not be easy!
I assume that some languages, stdlibs or VM's come with GMP (or similar ilk) which perhaps gave them an unfair advantage?
And what about languages that come with coroutines? In real world I use Coro/AnyEvent with Perl but these are separate modules and not part of Perl core libraries. So languages like Lua/Python have an (unfair?) advantage here?
> It should be pure language against pure language
Ideally then it should be same algorithm against algorithm?
I recently rewrote the fasta.pl implementation on Alioth using an exact port of the fasta.rb code. Before change Ruby fasta was two times faster than the Perl version. Afterwards Ruby was 30% slower than Perl - https://news.ycombinator.com/item?id=5141179
Nothing, it seems. But a benchmark is pointless if you aren't comparing like for like, yet people are clearly taking the results as some sort of serious indicator.
As it is, it's like sending two different cars around two different tracks and comparing how long they took.
>>But a benchmark is pointless if you aren't comparing like for like<<
According to you it defeats the point when we do compare like to like -- "if you can just call out to C libraries".
>>yet people are clearly taking the results as some sort of serious indicator<<
Offer them something better.
>>sending two different cars around two different tracks<<
No. It's like saying here's the track - Ruby guy bring out your best cars, Python guy bring out your best cars - and sending them around the same track.
The possible mismatch between programs written for quad-core and those written for single-core is marked.
You can mention the difference between CPU time and wall-clock time because both are intentionally shown, along with "CPU Load" so you can see how much difference quad-core has made.
And as draegtun informed you, there's another set of measurements that removes that difference by forcing all the programs onto a single core.
The problem with the Ruby binary-trees programs is that they run out of memory.
47 comments
[ 4.0 ms ] story [ 106 ms ] threadMaybe is fine if you have a controlled traffic, but with better speed you can manage more traffic with the same hardware. That's why Facebook had to build a PHP compiler at some point.
In any case, How you would justify using Ruby over another script language like Python, Perl, etc? All of them are comparable in terms of productivity and seems to be orders of magnitude faster.
I'd recommend Ruby because it's readable, it has a good tool ecosystem a great community and design decisions that please me. Python and Perl are just not what I'd like to see in a language. I will use them if I have to, but given the chance I'll use Ruby.
Exactly how do you know what the majority of applications do or don't do?
Besides, code is written for more than just websites. Chef/puppet/cfengine, one-off scripts you write, etc. There's a whole lot of these types of things that don't matter if your code takes 0.01 seconds or 0.05 seconds.
"There's a whole lot of these types of things that don't matter if your code takes 0.01 seconds or 0.05 seconds." Yeah but why would you prefer Ruby instead of PHP/Python/Perl for those things then?
Edit: And when I actually need better performance from a specific piece of code, I can and do drop down to C.
However I think the basis of these benchmarks is really to pit languages against each other and not it's frameworks/libraries.
Yes, perhaps it could be more efficient, but I don't use Ruby for speed. The way that I see it, Ruby exchanges execution time for programmer time. Alternatively, If I want something to be fast, I'll spend a few days writing it in C/C++/Assembler, but I'd expect to spend at least an order of magnitude more time writing code.
That's because you (subconsciously or because of your specific work needs) constrain yourself with problems that don't need the speed.
But who says that having a X times faster Ruby is not very desirable?
- For one, a cash-strapped startup could deploy RoR instances with fewer boxes, saving a lot of hosting costs.
(ASIDE: People compare server costs to the salary of a developer to trivialize them. What they always assume though is that the developer makes a US sized salary (in parts of the world, $500/month IS a nice salary). Or they assume he someone is even paid in the first place (think a ramen startup, a university student deep in debt wanting to launch something on the cheap etc).
- Second, people might like to write a game in Ruby, a number-crunching app, a drawing app, etc. Why be constrained by the language and need to employ another for the critical parts? This puts a barrier to a programmer (in code complexity, in having 2 codebases, in time needed to manage memory and such). Barriers mean: you can do fewer interesting stuff than what could be possible.
>The way that I see it, Ruby exchanges execution time for programmer time.
Sure, by why it has to be that way? Nothing inherent in dynamic languages make it so. Of course they will be slower than C, but they can be 10 or 20 times faster than what they are NOW.
Pure V8, Smalltalk VMs, JVM languages etc are consistently much faster than pure (no C delegation) Ruby or Python. Ruby has a lot of room to get much faster.
Woes of developers who prefer complaining to extracting the CPU bound parts of their app to another language still difficult to worry about.
...now if you're building a chess engine, that's a different story.
(I just take chess as an example as it's my latest hobby. It's the first and only project where I found it necessary to switch to C because of performance considerations (switched from C#).)
Python was released in what? 2008? 2009? I don't want to be that person, but if we're going to compare language based on benchmarks that are pretty irrelevant to modern uses already (like the CLBG is), at least let's pick "equivalent" languages and let's try avoiding the obvious biases.
There's absolutely no reason to believe that's true.
I do think Ruby will get faster eventually. Maybe not to the point that Ruby suddenly becomes "fast" of course, but by a margin that can make a difference. This is only my opinion (backed by some benchmarks in -for now- particular circumstances) of course, but I'm hopeful that the situation will improve.
Of course someone will lose a fight when their opponent is wearing a jetpack and rocket powered shoes.
Anyway you didn't seem too fussed about what the Alioth benchmarks showed in the past - https://news.ycombinator.com/item?id=634738 :)
I hadn't looked at the source code back then. Only now when this particular variant of the benchmark shows some rather unrealistic results. Another example (binary trees, I believe) has Ruby using a single process whereas the Python example is using multiple processes to spread the load, so even though Ruby wins on CPU time, Python wins with a slightly lower wall-clock time.
And making the Ruby pidigits benchmark use GMP too? That defeats the point of benchmarking one language against another if you can just call out to C libraries for everything (hey, there's an idea.. Ruby could be parallel with C if we just did that). It should be pure language against pure language otherwise it's totally meaningless (and, well, it is, as it stands).
The single core benchmarks doesn't change the picture much either - http://benchmarksgame.alioth.debian.org/u32/ruby.php
re: GMP / C libraries
Generally I agree with you on this however agreeing on a level playing field may not be easy!
I assume that some languages, stdlibs or VM's come with GMP (or similar ilk) which perhaps gave them an unfair advantage?
And what about languages that come with coroutines? In real world I use Coro/AnyEvent with Perl but these are separate modules and not part of Perl core libraries. So languages like Lua/Python have an (unfair?) advantage here?
> It should be pure language against pure language
Ideally then it should be same algorithm against algorithm?
I recently rewrote the fasta.pl implementation on Alioth using an exact port of the fasta.rb code. Before change Ruby fasta was two times faster than the Perl version. Afterwards Ruby was 30% slower than Perl - https://news.ycombinator.com/item?id=5141179
>>if you can just call out to C libraries for everything<<
You can't just call out for everything.
There have been language implementations that used GMP under-the-covers, so - to level the playing field - everyone can call out for GMP and PCRE.
As it is, it's like sending two different cars around two different tracks and comparing how long they took.
According to you it defeats the point when we do compare like to like -- "if you can just call out to C libraries".
>>yet people are clearly taking the results as some sort of serious indicator<<
Offer them something better.
>>sending two different cars around two different tracks<<
No. It's like saying here's the track - Ruby guy bring out your best cars, Python guy bring out your best cars - and sending them around the same track.
The possible mismatch between programs written for quad-core and those written for single-core is marked.
You can mention the difference between CPU time and wall-clock time because both are intentionally shown, along with "CPU Load" so you can see how much difference quad-core has made.
And as draegtun informed you, there's another set of measurements that removes that difference by forcing all the programs onto a single core.
The problem with the Ruby binary-trees programs is that they run out of memory.
http://benchmarksgame.alioth.debian.org/u32q/program.php?tes...