Another day, another "proof" that Ruby is slower than X, which surprises exactly no one.
The interesting thing here though is that considering the trivial amount of work to be done, Ruby shouldn't be that slow. I'd like to see the source code of the Ruby project to see if some simple improvements could have been made that aren't too invasive.
The spike in latency at around 15,000 requests could be explained by the garbage collector, which normally is put out-of-band. This makes me believe that he used a single-threaded, single-worker approach for Ruby. I don't know about the Go framework he used, but maybe it was multi-threaded (and not green-threads).
Puma (http://puma.io/) is multithreaded, but only JRuby and Rubinius support concurrent execution - standard MRI/YARV have a global VM lock just like Python.
I kinda sorta suspect he was using ruby-json, which is substantially slower at JSON serialization than something like Oj. It's not going to make the Rails implementation start seriously competing with Go, but it would help matters substantially.
require 'benchmark'
require 'json'
require 'oj'
require 'open-uri'
TIMES = 100000
hash = JSON.parse(open("https://gist.github.com/gnip/764239/raw/6c6a2297f3e4e29a626f07db0c57b45af7d7e5d7/Twitter+%28json+format%29.js").read)
Benchmark.bm do |x|
x.report("Oj") { TIMES.times { Oj.dump(hash) } }
x.report("ruby-json") { TIMES.times { JSON.dump(hash) } }
end
# user system total real
# Oj 0.980000 0.000000 0.980000 ( 0.983426)
# ruby-json 8.780000 0.000000 8.780000 ( 8.832761)
Edit: I'm working on a Go implementation of the above benchmark. I haven't touched Go before today, though, so it's a bit of learning. :)
Edit edit: Here's the benchmark in Go. First time I've touched it, tell me if I stuffed something up awfully. Amusingly, it ended up being almost exactly 10x many lines as the Ruby code. :)
I can't speak to Matt's source code, but you can find ours at GitHub [1].
In fact, the ETA for Round 6 is next Thursday, June 20. If you can point out any tuning for us to put in place for the Ruby tests before June 17, we may be able to get pull requests accepted before we kick off the test runs.
A slow dynamic bytecode-interpreted globally VM-locked language running a massive completely-overkill framework is only 10x slower than a statically compiled natively concurrent language running a tiny microframework? I'd say that's a massive positive for Ruby and Rails to be honest.
It can be deemed as a "massive positive" for RoR only when "slow dynamic bytecode-interpreted globally VM-locked language running a massive completely-overkill framework" is a good thing.
I meant: one I can write 3X faster. And yes, "I". Not sure how this generalizes - but it's generally a bad idea to pick a new language so the code will run fast if the time to delivery is important.
I've done my fair share of both Java and Ruby web development, and without a doubt, I was much faster writing Ruby.
For non-trivial Java apps, startup time was > 20 seconds. That kind of REPL while developing kills productivity and developer satisfaction.
The last Java app I wrote had more lines of configuration than my first Rails app had lines of code, at equivalent complexity.
The first Java job I had required a full-time script writer, a DBA, and someone to keep the web server up. In Rails, that entire team would have been shrunk by 75% (mind you it didn't exist back then).
I do work in other languages. For machine learning, Ruby's speed is a killer except for toy data sets, so I have used Octave and Python. When it comes to the web though, Rails really did do many things right, from a sane MVC, default ORM, helpers and migrations. There are times execution speed trumps development speed. For fast dev on web apps, Rails really was cutting edge and for those with an investment it can still be a better trade-off than doing something in Go.
Not do advocate Java, but your knowledge is dated, I'd say at least 5yrs off. Play has on the fly code compilation, Jetty does fast reloads, JRebel hot class reloading etc.
I pay you $1000 (or EUR) if you show me how you can be 3x faster from idea to live with RoR against something else, say Java/Play. If 50% of your time is HTML/CSS fixing,thinking,finding edge cases,testing,debugging,communication with customers etc. you need to be 6x faster. And not in the first 5 days but you need to maintain this speed even as your code size grows and more and more time is spent on understanding existing code.
The only way to reach 6x is with 3rd party libs you have in RoR but you have no equivalent on say the JVM. Massive code reuse is the only major productivity gain today.
[edit] Sorry my mistake. If 50% of your time is taken by other tasks when writing say Java code, the most you can achieve is doubling your productivity by reducing development time to zero. If you tripple your productivity during development, your overall gain is 33% I think (not that good with on the fly calculation obviously).
[edit2] Just realized, this is Amdahl's law for developer productivity with your other tasks being the sequential fraction.
Amdahl's law: yeah, this is pretty much it. Of course I was only discussing dev time.
Of note, many of the best Ruby / Rails folks I know moved on to much better roles, whether CTO at growing companies or started working on cash-generating projects. Once they optimized dev time, they got more time to work on things like design or copywriting, and optimized the hell out of that too.
Completely agreed. These kinds of comparisons are always a joke. Rails has always been about programmer efficiency, not execution efficiency since day one.
We shouldn't be so quick to call someone's learnings that they are sharing with the community a joke. I think knowing the performance difference could be very useful. The Rails app I work on has an API that gets over 300K RPM which adds up to quite an EC2 bill each month. Reading someone's experiments and results allows someone like myself to some super simple estimates on what the savings might be when going with Go vs Rails. It opens a conversation.
Now if you are building a side project or haven't hit the level of traffic where performance improvements are extremely noticeable on your hosting projects, then maybe this isn't the post for you. But I would just advise that we be a little slower to call something that a fellow member of this community spent time learning and then sharing with others a joke.
Yes, if you're a VC backed startup, 10x the site costs (say 10 M instead of 1M) is not a problem. Otherwise the productivity gains of the developers need to make up the 9M difference.
Of course it is a positive. The question becomes, is the performance difference worth it? If you were comparing RoR to a Servlet framework, I'd say no. But I have yet to see any Go code that doesn't look at least as "productive" as any of the Ruby code I have seen or created.
10x is a lot though. Definitely not a positive for Ruby. People were making these comparisions between Java and C++ for ages, and called Java slow when something took 1.4x as long as native C++. Rails doesn't even try to compete.
Yeah, but Ruby is famous for being really, really, slow. Almost everything is faster than Ruby, and many things—including other typeless dynamic bytecode-interpreted languages—are crazy faster.
Being described as "much faster than that really super slow thing" is... well... I suppose it's not exactly a negative, but I wouldn't put it on my resume...
That's a positive for Ruby only when you're not running on a cloud server, or when you are cost-conscious about the amount of electricity you're sucking in a data center somewhere.
That being said, that Ruby is 10x slower than Go isn't that much of a surprise. It's not a string-interpreted language like the BASICs of old. It's interpreting byte-codes, and anyone who's ever written a 6502 or Z-80 emulator (essentially the same thing) can tell you, on average, most byte-codes take 10x as long to run as the corresponding function written in in-line C.
I wrote a simple service in Go that takes <11MB RAM and can handle 7500 requests/sec. That was a fast one-off when playing around. I've never had even a simple Sinatra app run half that fast.
In 2002, our Layer 7 streaming traffic redirector written in C, running on an old generic P3 800 CPU with 512 MB RAM, easily handled 15,000 requests per second of real world (unpredictable latency and error rates) HTTP and RTSP traffic.
The service wasn't simple. It maintained a content map of all CDN files across layers of storage, servers, and cache, health of all server pools, and a geo IP database, and balanced users according to load optimization algorithms taking all these into account.
The redirector and session mapping cache took <80MB RAM, with the rest devoted to async content promotion and demotion service, async geo IP service, async server health service, etc.
Two points: (1) interpreted and heavy scaffolding framework Rails vs. hand coded Go doesn't tell us much we don't already know: that layers of abstraction and scaffolding can simplify development but slow execution, and (2) many of the requests per second benchmarks these days mostly illustrate that frameworks in general are wastefully slow, requiring far too much investment in hardware to scale.
With one or two servers, this doesn't matter. Frameworks are invaluable for rapid iteration towards MVP proof, and fantastic for churning out web apps for mid tier companies with moderate (less than millions) traffic. But if you're going bigger, have a plan to get your bottlenecks closer to the metal, because with hundreds or thousands of servers, the savings becomes significant and possibly even crucial to your survival if your type of offering gets commoditized.
Compiled, concurrency/parallelism-focused language is more performant than a dynamically typed, interpreted language in a micro benchmark. News at 11.
While it'd be great to get to the point where you need to worry a lot about performance, the reality is that most projects won't ever get there. Some of us will fail quite a few times before getting to the point of having to worry about Ruby-specific performance issues.
Don't get me wrong, I have really enjoyed playing with Go. I just think we have a tendency to geek out about "performance", when it's not nearly as important as productivity/maintainability for most of us.
Apologies for the rant, but these benchmarks are just short of useless, and I wish we would spend more time "benchmarking" ease of development, maintainability, and talent availability.
Coding in Go isn't more difficult than coding in Ruby. Any competent dev should be able to pick it up and be productive using it within a couple of weeks. So for the same effort you get considerable performance improvements.
The problem isn't picking up a language, it's language support (and 3rd party libraries) that are crucial.
For example, if a client requests that I add a module to my application to fetch records from AD and cross reference with data pulled from salesforce, am I going to have to write a ton of low level networking code by hand or will it be trivial to find a robust open source module that does all of that for me?
Was going to reply to arkitaip, but you nailed it on the head. I have really enjoyed working with Go, but the much more limited ecosystem does mean rolling up your sleeves and doing it yourself more often.
So I agree, I don't think you can say Go is just as fast for development in every case.
But these are much more interesting discussions than these silly micro benchmarks...
So a statically typed compiled language is quicker than Ruby... Not exactly shocking. Ruby's advantage isn't in its speed, but in how easy and quick it is to get things done.
Statically-typed and speed aren't necessarily correlated. If you want a framework faster than Go, dynamically-typed and rivaling Ruby's development speed, have a look at OpenResty+Lua+LuaJit (And MoonScript if you really want to code fast).
For a little bit of fairness, shouldn't he at least be comparing Sinatra, not Rails? And where's the source code so we can compare the other part of this: developer effort.
He links to a filtered view of our results which also includes Sinatra [1]. I can't provide his source code, but you can review the source code of the Go [2] and Sinatra [3] implementations for our project at GitHub. Note that the Go test implements all of our current tests; the Sinatra code only implements the JSON-serialization and database query tests.
I guarantee it's the bottle neck. I made a service getting between 1.5mil to 2mil hits a day, and the DB is the bottleneck for me, I was shocked to see how little resources Go and nginx were using.
The really important saving between the two will probably be memory usage for most uses - 10x less memory is far more important as memory is a limit far more frequently hit than processing speed. This does depend on the application of course, but for the vast majority of web applications speed is not really an issue, even when they have a lot of users, so then it comes down to tooling and libraries, where Go is a little lacking at the moment (though I'm sure it'll catch up).
It's a great language to work with though and I can see it easily overtaking Ruby in this space given a few more years, partly because it's so easy to pick up when coming from languages like Ruby and Python.
Using Go is a constant temptation and for small APIs it's probably a good choice.
Ruby (esp. Rails), for me, is worth the performance trade-off - even though that stings me coming from a C/assembler background originally.
With Rails you can build complex/deep APIs with very efficient syntax and clear tests - the conventions enforced mean that everything is in the same place/style no matter who coded it - so the finished app is more maintainable than in any other language/framework I've ever known.
A small note on Go and enforced conventions. Gofmt enforces a standardized syntax for all code - there's no debating or arguing over code style. Now for the rest of it (file structure and the like) Rails blows Go away. If Go had a framework like Rails, but designed to play to Go's strengths, it could decimate Rails in every facet except the amount of existing code written.
I don't like ad-hoc benchamarks that do not attempt at all to investigate what was the cause of measured differences.
There are many factors that influence request processing time, language is only one of them. Web APIs are most often IO bound, so large differences between frameworks often indicate the IO is handled differently. From my experience, dynamic languages can easily handle HTTP requests below 10ms without any DB interaction, and out-of-a-box DB interaction that framework provide is often suboptimal. If you investigate, it can for example turn out that ORM runs several SQL queries for operations that could be performed with just one, or that it makes a separate DB connection for each request (looking at you Django).
Ruby 10x faster than Golang tells nothing.
Why is that there there is a big spike around 15,000 requests? If you investigate, you may discover that Puma by default creates 16 threads, and the benchmark simulates 20 clients (https://github.com/puma/puma), so requests from 4 clients need to wait in queues. As a result, the benchmark may be comparing apples to oranges: code with no upper limit on number of concurrent requests, with code that can handle 16 concurrent requests.
Rails is a heavyweight framework. I'd be more interested in a comparison with a lightweight framework like Cuba. Ruby will still lose to Go, but the difference probably won't be 10x.
I know that these tests results are kind of bogus but there's been a series of web framework comparisons http://www.techempower.com/benchmarks/ (there was also a lot of discussion about it on HN) and Sinatra seems to be faster than Rails but not by as much as I would have guessed.
That's consistent with Cuba's own benchmarks - http://files.soveran.com/cuba/#9 - where Sinatra improves over Rails by a little but Cuba improves by a lot.
I really don't think he understands ruby web servers well... Using Puma on MRI is silly. You get none of the threading gains it was designed for. It needs to run in jruby or rubinius for it to be mildly accurate.
Also, Puma ain't the fastest ruby web server. I'd like to see this same test run using Thin 2.0 (though it'll make you cry-- or so they say), Unicorn, or Rainbows.
As well, as others have said, comparing a large framework to one without is silly.
I'm not hating on go or saying rails is fast, I just feel like this is a particularly bad demonstration of the authors point.
This is more of a correlation, than causation. Too many factors not accounted for, or mentioned, which obviously contribute to the results.
Your benchmarks are ridiculously skewed and inaccurate. Note: I'm not a Ruby developer and I already gave up on Go months and months ago.
You're comparing a full stack with a million bells and whistles attached to each request (Rails) vs basically nothing but a basic route mapper.
You should do real world benchmarks IMO. There's no doubt Rails will be slower but you're going to spend so much time reinventing things that every other popular web language has had for years upon years [note: this is the main reason why I stopped using Go, I kept having to solve problems that were solved in every major other framework].
59 comments
[ 3.9 ms ] story [ 107 ms ] threadThe interesting thing here though is that considering the trivial amount of work to be done, Ruby shouldn't be that slow. I'd like to see the source code of the Ruby project to see if some simple improvements could have been made that aren't too invasive.
The spike in latency at around 15,000 requests could be explained by the garbage collector, which normally is put out-of-band. This makes me believe that he used a single-threaded, single-worker approach for Ruby. I don't know about the Go framework he used, but maybe it was multi-threaded (and not green-threads).
Edit edit: Here's the benchmark in Go. First time I've touched it, tell me if I stuffed something up awfully. Amusingly, it ended up being almost exactly 10x many lines as the Ruby code. :)
https://gist.github.com/cheald/5784471
So the final results: I'm not quite sure that feels right, so if someone else could sanity check me, please do.In fact, the ETA for Round 6 is next Thursday, June 20. If you can point out any tuning for us to put in place for the Ruby tests before June 17, we may be able to get pull requests accepted before we kick off the test runs.
[1] https://github.com/TechEmpower/FrameworkBenchmarks
Developer efficiency is notoriously difficult to measure and prone to all sorts of biases. Perhaps the key word in your sentence is "I"?
I've done my fair share of both Java and Ruby web development, and without a doubt, I was much faster writing Ruby.
For non-trivial Java apps, startup time was > 20 seconds. That kind of REPL while developing kills productivity and developer satisfaction.
The last Java app I wrote had more lines of configuration than my first Rails app had lines of code, at equivalent complexity.
The first Java job I had required a full-time script writer, a DBA, and someone to keep the web server up. In Rails, that entire team would have been shrunk by 75% (mind you it didn't exist back then).
I do work in other languages. For machine learning, Ruby's speed is a killer except for toy data sets, so I have used Octave and Python. When it comes to the web though, Rails really did do many things right, from a sane MVC, default ORM, helpers and migrations. There are times execution speed trumps development speed. For fast dev on web apps, Rails really was cutting edge and for those with an investment it can still be a better trade-off than doing something in Go.
It still seems as though these frameworks are all playing catch-up, with e.g. things like migrations.
The only way to reach 6x is with 3rd party libs you have in RoR but you have no equivalent on say the JVM. Massive code reuse is the only major productivity gain today.
[edit] Sorry my mistake. If 50% of your time is taken by other tasks when writing say Java code, the most you can achieve is doubling your productivity by reducing development time to zero. If you tripple your productivity during development, your overall gain is 33% I think (not that good with on the fly calculation obviously).
[edit2] Just realized, this is Amdahl's law for developer productivity with your other tasks being the sequential fraction.
Of note, many of the best Ruby / Rails folks I know moved on to much better roles, whether CTO at growing companies or started working on cash-generating projects. Once they optimized dev time, they got more time to work on things like design or copywriting, and optimized the hell out of that too.
Now if you are building a side project or haven't hit the level of traffic where performance improvements are extremely noticeable on your hosting projects, then maybe this isn't the post for you. But I would just advise that we be a little slower to call something that a fellow member of this community spent time learning and then sharing with others a joke.
Being described as "much faster than that really super slow thing" is... well... I suppose it's not exactly a negative, but I wouldn't put it on my resume...
That being said, that Ruby is 10x slower than Go isn't that much of a surprise. It's not a string-interpreted language like the BASICs of old. It's interpreting byte-codes, and anyone who's ever written a 6502 or Z-80 emulator (essentially the same thing) can tell you, on average, most byte-codes take 10x as long to run as the corresponding function written in in-line C.
Computer science -- it works!
In 2002, our Layer 7 streaming traffic redirector written in C, running on an old generic P3 800 CPU with 512 MB RAM, easily handled 15,000 requests per second of real world (unpredictable latency and error rates) HTTP and RTSP traffic.
The service wasn't simple. It maintained a content map of all CDN files across layers of storage, servers, and cache, health of all server pools, and a geo IP database, and balanced users according to load optimization algorithms taking all these into account.
The redirector and session mapping cache took <80MB RAM, with the rest devoted to async content promotion and demotion service, async geo IP service, async server health service, etc.
Two points: (1) interpreted and heavy scaffolding framework Rails vs. hand coded Go doesn't tell us much we don't already know: that layers of abstraction and scaffolding can simplify development but slow execution, and (2) many of the requests per second benchmarks these days mostly illustrate that frameworks in general are wastefully slow, requiring far too much investment in hardware to scale.
With one or two servers, this doesn't matter. Frameworks are invaluable for rapid iteration towards MVP proof, and fantastic for churning out web apps for mid tier companies with moderate (less than millions) traffic. But if you're going bigger, have a plan to get your bottlenecks closer to the metal, because with hundreds or thousands of servers, the savings becomes significant and possibly even crucial to your survival if your type of offering gets commoditized.
While it'd be great to get to the point where you need to worry a lot about performance, the reality is that most projects won't ever get there. Some of us will fail quite a few times before getting to the point of having to worry about Ruby-specific performance issues.
Don't get me wrong, I have really enjoyed playing with Go. I just think we have a tendency to geek out about "performance", when it's not nearly as important as productivity/maintainability for most of us.
Apologies for the rant, but these benchmarks are just short of useless, and I wish we would spend more time "benchmarking" ease of development, maintainability, and talent availability.
For example, if a client requests that I add a module to my application to fetch records from AD and cross reference with data pulled from salesforce, am I going to have to write a ton of low level networking code by hand or will it be trivial to find a robust open source module that does all of that for me?
Ultimately library availability affects productivity as much as language features do.
So I agree, I don't think you can say Go is just as fast for development in every case.
But these are much more interesting discussions than these silly micro benchmarks...
[1] http://www.techempower.com/benchmarks/#section=data-r5&p=3q
[2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
[3] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
It's a great language to work with though and I can see it easily overtaking Ruby in this space given a few more years, partly because it's so easy to pick up when coming from languages like Ruby and Python.
Ruby (esp. Rails), for me, is worth the performance trade-off - even though that stings me coming from a C/assembler background originally.
With Rails you can build complex/deep APIs with very efficient syntax and clear tests - the conventions enforced mean that everything is in the same place/style no matter who coded it - so the finished app is more maintainable than in any other language/framework I've ever known.
There are many factors that influence request processing time, language is only one of them. Web APIs are most often IO bound, so large differences between frameworks often indicate the IO is handled differently. From my experience, dynamic languages can easily handle HTTP requests below 10ms without any DB interaction, and out-of-a-box DB interaction that framework provide is often suboptimal. If you investigate, it can for example turn out that ORM runs several SQL queries for operations that could be performed with just one, or that it makes a separate DB connection for each request (looking at you Django).
Ruby 10x faster than Golang tells nothing.
Why is that there there is a big spike around 15,000 requests? If you investigate, you may discover that Puma by default creates 16 threads, and the benchmark simulates 20 clients (https://github.com/puma/puma), so requests from 4 clients need to wait in queues. As a result, the benchmark may be comparing apples to oranges: code with no upper limit on number of concurrent requests, with code that can handle 16 concurrent requests.
Incredible!
"My code is simpler, more flexible and easily cross compiles"
Show us the code then, cause I so much doubt that.
In this case, this is good news for Ruby.
Rails is a heavyweight framework. I'd be more interested in a comparison with a lightweight framework like Cuba. Ruby will still lose to Go, but the difference probably won't be 10x.
Also, Puma ain't the fastest ruby web server. I'd like to see this same test run using Thin 2.0 (though it'll make you cry-- or so they say), Unicorn, or Rainbows.
As well, as others have said, comparing a large framework to one without is silly.
I'm not hating on go or saying rails is fast, I just feel like this is a particularly bad demonstration of the authors point.
This is more of a correlation, than causation. Too many factors not accounted for, or mentioned, which obviously contribute to the results.
You're comparing a full stack with a million bells and whistles attached to each request (Rails) vs basically nothing but a basic route mapper.
You should do real world benchmarks IMO. There's no doubt Rails will be slower but you're going to spend so much time reinventing things that every other popular web language has had for years upon years [note: this is the main reason why I stopped using Go, I kept having to solve problems that were solved in every major other framework].