If you present lots of tables, some colour coding (green for high performing, red for slow, and some ranges in-between) might make it easier to parse visually.
Very thorough work and some interesting points along the way.
The title is a bit misleading though - This is not so much benchmarking Ruby as it is benchmarking the infrastructure around it. People may be switching to other languages because the actual application code is faster.
Well that could be true. The intent was for it to be a web benchmark, so benchmarking all the basic ruby web variants.
It's certainly not a pure ruby hello world test, but I think there is a lot of value in testing the infrastructure around Ruby. That has a huge impact on performance.
I don't doubt that other languages have faster application code, but figuring out how much faster vs what the best possible option for the existing code is worth exploring I think.
However, the combination of Standard Ruby + Thin/Unicorn + Rails is about the slowest possible combination. That on Ruby 2.1 gives you 1,455 req/sec. The fastest possible combination (JRuby + Torqbox + Rack) gets you 10,159 req/sec.
That is giving up 85.67% performance before you write a single useful line of code. That means at best possible performance, the standard rails app is 6.98 times slower than the fastest Ruby code could be. That also means your standard rails app is at least 7 times slower than the comparable Go or Scala app, simply because of your server, runtime, and framework.
Part 2 : measure development time without frameworks? ;)
awesome work! I enjoyed this as I was making comparisons between puma and unicorn.
A co-worker of mine has been battling a rails 3 -> 4 and ruby 1.9 -> 2.1 transition. I'm hoping this helps drive the needle in the right direction.
I haven't seen anything out there that goes this deep on PHP or Python or Node, but there are also fewer runtime and server choices out there for those languages.
You can certainly test frameworks, but that's where a lot of the fun begins and ends. Also, Techempower benchmarks pretty much did this outstandingly well.
Probably down the line I might look at PHP, Python, Clojure, .NET, and Java....
It's funny how these things spiral out of control.
As a satisfied JRuby user for over 3 years in production environments, I can say this has translated into real life for me, as far as the variants I've used (Ruby 2.1, JRuby / Torquebox, Torqbox, Puma, Trinidad / Rails, Sinatra, Cuba, Straight Rack)
17 comments
[ 2.4 ms ] story [ 53.5 ms ] threadI was working on even more charts, but they took forever to get on each thing, so hopefully for a later version...
The title is a bit misleading though - This is not so much benchmarking Ruby as it is benchmarking the infrastructure around it. People may be switching to other languages because the actual application code is faster.
It's certainly not a pure ruby hello world test, but I think there is a lot of value in testing the infrastructure around Ruby. That has a huge impact on performance.
I don't doubt that other languages have faster application code, but figuring out how much faster vs what the best possible option for the existing code is worth exploring I think.
That is giving up 85.67% performance before you write a single useful line of code. That means at best possible performance, the standard rails app is 6.98 times slower than the fastest Ruby code could be. That also means your standard rails app is at least 7 times slower than the comparable Go or Scala app, simply because of your server, runtime, and framework.
Part 2 : measure development time without frameworks? ;)
awesome work! I enjoyed this as I was making comparisons between puma and unicorn.
A co-worker of mine has been battling a rails 3 -> 4 and ruby 1.9 -> 2.1 transition. I'm hoping this helps drive the needle in the right direction.
http://www.techempower.com/benchmarks/
You can certainly test frameworks, but that's where a lot of the fun begins and ends. Also, Techempower benchmarks pretty much did this outstandingly well.
Probably down the line I might look at PHP, Python, Clojure, .NET, and Java....
It's funny how these things spiral out of control.
I am curious how much JRuby performance advantages impact the rest of the code beyond just routing. There will be more benchmarks.