I'd like to see the results with nginx and siege, rather than Apache and ab. Other than that, it's clear that the more magic you bake in, the slower the framework will tend to be.
I would like to see Noir in there also. I have been using Noir a lot in the last 4 or 5 months: good performance and nice to develop with. BTW, off topic but since there was a lot of HN discusion yesterday about DHH's article on Basecamp Next and using pjax: I just blogged about using pjax with Noir: http://blog.markwatson.com/2012/02/using-pjax-with-clojure-a... (with a link to a github repo I just created).
Noir has been really fun to work with. It's really fast, API is predictable and easy to learn. It seems to be inspired heavily by rails but without the constant dependence on "magic" to do simple things.
It's also nice to break away from the Active Record overhead and be able to find new optimal ways to model data with Redis/Mongo.
I'd still recommend rails to the newbie developer, but if you're experienced in developing web apps, I'd highly recommend Noir.
Would be curious to see the other end of the coin: given a complex web app with dozens of controllers and hundreds of templates, what's the amount of developer time required to build it with each of these frameworks.
It would be nice to see a hello-world example with 1, 10, 100 sample routes.
It's pretty easy to get a naive routing implementation perform really well with a handful of routes. The tables can easily turn in the context of a larger application however.
The database tests are meaningless, I think, since they're comparing e.g. ORM in Django and Rails to a single, non-abstracted SQL query in Bottle. It's very apples and oranges.
It serves to show that, if you want SQL performance, maybe you'll have to ignore the ORM. Every point tells you something. Whether this something answers an important question is a completely different matter.
It's still a better framework. Codeigniter has traditionally been the framework of designers rather than programmers.
Just last week I discovered it writes the session every time you set a session variable. This results in a cookie header being added multiple times and sent in the response. Apparently this is not a bug and it does this by design.
Do these results have any real world value?
Probably not. (...)
Kind of sums it up. It's fun to play around with web frameworks and draw some bar charts, but please don't take this seriously. And definitely don't say "X is slower than Y" because of this.
On idiotic benchmark comparisons "Try to carry 500 people
from Los Angeles to Tokyo in an F-15.
No? Try to win a dogfight in a 747. No? But they both
fly, so it must be useful to compare them... especially
on the basis of the most simplistic test case you can
think of. For extra points, use *only one* test case.
Perhaps this paper can be described as "comparing an F-15
to a 747 on the basis of required runway length".
The DB Query test is pretty interesting, actually. It draws results into a much closer range, and shows that your bottleneck for WebApp/DB style apps will probably be your database.
Yeah, especially when you take into account that some of them are using a proper ORM and others are just using straight SQL queries. Sinatra looks twice as fast as Rails at DB based pages, but if they through in ActiveRecord in the Sinatra app it might be even closer.
> Yeah, especially when you take into account that some of them are using a proper ORM and others are just using straight SQL queries. Sinatra looks twice as fast as Rails at DB based pages, but if they through in ActiveRecord in the Sinatra app it might be even closer.
Absolutely. It's also true the other way - you don't have to use ActiveRecord with Rails, or the native ORM with Django, etc.
Of course, we already know that these benchmarks don't really have any parity, but they're interesting at a very high level as long as you understand why one might be faster.
PS: A more complex test would be nice, though - at a cursory glance, Django's ORM doesn't appear to be as slow as it used to be/is often attributed to be.
I agree. High-traffic websites are running on many of these frameworks. As long as several companies have scaled the frameworks in question, it's better to just choose the one that makes you more productive.
When building a company it's also good to choose something your team-members know and you can recruit for.
I mostly agree with you but there's a touch of real world value here... I mean, once you start adding code to 'hello world' it's not going to get faster. So, if you have an app with some stringent perf requirements, this would tell you if you can even really start with any of these frameworks.
For that same reason, I think metrics on how much RAM they settle in at would also be interesting. They could disqualify some frameworks from RAM-constrained requirements right away (ex. if you're building some hardware with an embedded-but-sophisticated web ui).
"Hello World" tests really need to go away and be replaced by a concurrent workload that represents 100s of users accessing, writing, etc.
I remember a test that compared MySQL to SQL Server (Microsoft's SQL Server) a few years back.
MySQL had SQL Server beat hands down... For 1 concurrent user.
Once things scaled passed 10-20, SQL Server started winning.
At 30-50, MySQL could no longer respond and would crash, while SQL Server scaled to 90!
But everyone was wowed by the 1 user-load test! And countless posts were made over the years showing how MySQL is superior to the obviously flawed SQL Server.
I disagree with others that have said that these tests are worthless for a real application. It is worth knowing the baseline speed of a framework as this affects the maximum number of requests you will be able to achieve, without scaling out your application.
I strongly disagree. There's no such thing as baseline speed, there's only baseline speed on a given hardware config. Rails does not have a max speed of X reqs/second. It has a max speed of X reqs/second on a given hardware config. If you're profiling your own app you'll have to retest it on your own hardware first to figure out how close to that baseline you are.
Of course there is. The "hello-world" microbenchmark tells you something about the core stack performance of the frameworks relative to one another. This tends to be surprisingly indicative of the relative real-world throughput per node that you may expect for a more complex application.
I.e. Rails (ruby) scores 4 times lower than Python in that micro-benchmark. This is pretty close to the difference that I've observed between real applications on the respective platforms. If anything you'll see the difference magnified in a complex application, but it's unlikely to turn around.
Whether that matters or not in the big picture is a different question (it usually doesn't).
If anything you'll see the difference magnified in a complex application, but it's unlikely to turn around.
I wouldn't say so. The bottleneck in a hello world benchmark might be a component of the framework, but in complex applications it's likely to be something else. Just look at the "Template Test with DB Query": There, Sqlite is the bottleneck, and the performance difference between Django and Rails fades into background completely.
in complex applications it's likely to be something else
Nope. Ruby/Python app-servers always end up CPU-bound, unless you're doing something very much out of the ordinary (such as blocking on an embedded SQLite...).
With all due resepct, bollocks. The dispatcher is not tested at all here for example. To do a proper test add in some complicated url dispatch rules, session handling, various kinds of nasty hard-to-abstract-out business logic and we might be talking. The trouble is there you're then talking about a week's work to write each app. On the other hand the "benchmarks" are an interesting ultra-simple rosetta stone.
The comparison is reasonable as long as you test against an equivalent feature-set on each platform.
It might actually be an interesting exercise to gradually layer more features on top (such as the dispatcher chain and different templating/ORM layers) to discover inefficiencies in the various frameworks by comparing the relative performance impact.
E.g. if you involve the dispatcher and Django suddenly gets 20% slower but Rails only 10% slower then that might hint at an inefficient dispatcher-impl in Django.
The test is actually not completely meaningless as it illustrates how our modern web frameworks have shifted the bottleneck from the database back to the frontend-nodes again.
That's usually a worthwhile tradeoff (hardware is cheap) but one that a growing number of junior devs don't seem to be making consciously anymore. For them it's just normal to be spending more time per request in the controller code than in the database query. Whereas for many oldtimers it will never cease to feel a little funny...
A side-effect of this trend is that you can't reasonably run your beefy rails-frontend on a mediocre CPU anymore. When latency is dominated by ruby-code crunching on strings then you really want the fastest per-thread performance that money can buy.
It's an interesting, but trivial test. The Rails numbers are incorrect and irrelevant now though because it's using an outdated Ruby version, 1.9.3 is now the gold standard, and outdated Rails version.
It would be more interesting, but a simple test like this doesn't demonstrate much. A benchmark testsuite with a few algorithms or data processing patterns common to a domain would be more helpful.
I'm disappointed that there are no Perl frameworks at all!
I've submitted a pull request[1] to rectify this gross oversight. Even though this benchmark is not particularly scientific, it still provides some meaningful data.
75 comments
[ 4.1 ms ] story [ 113 ms ] threadNoir has been really fun to work with. It's really fast, API is predictable and easy to learn. It seems to be inspired heavily by rails but without the constant dependence on "magic" to do simple things.
It's also nice to break away from the Active Record overhead and be able to find new optimal ways to model data with Redis/Mongo.
I'd still recommend rails to the newbie developer, but if you're experienced in developing web apps, I'd highly recommend Noir.
Rather harder to automate that, though. :)
It's pretty easy to get a naive routing implementation perform really well with a handful of routes. The tables can easily turn in the context of a larger application however.
Given that Symphony had such a huge buzz (aside from CodeIgniter), it is apparently the slowest..
Just last week I discovered it writes the session every time you set a session variable. This results in a cookie header being added multiple times and sent in the response. Apparently this is not a bug and it does this by design.
Things like that always remind me of this great quote by Tom Lane (source: http://en.wikipedia.org/wiki/Tom_Lane_%28computer_scientist%...):
Absolutely. It's also true the other way - you don't have to use ActiveRecord with Rails, or the native ORM with Django, etc.
Of course, we already know that these benchmarks don't really have any parity, but they're interesting at a very high level as long as you understand why one might be faster.
PS: A more complex test would be nice, though - at a cursory glance, Django's ORM doesn't appear to be as slow as it used to be/is often attributed to be.
When building a company it's also good to choose something your team-members know and you can recruit for.
For that same reason, I think metrics on how much RAM they settle in at would also be interesting. They could disqualify some frameworks from RAM-constrained requirements right away (ex. if you're building some hardware with an embedded-but-sophisticated web ui).
I remember a test that compared MySQL to SQL Server (Microsoft's SQL Server) a few years back.
MySQL had SQL Server beat hands down... For 1 concurrent user.
Once things scaled passed 10-20, SQL Server started winning.
At 30-50, MySQL could no longer respond and would crash, while SQL Server scaled to 90!
But everyone was wowed by the 1 user-load test! And countless posts were made over the years showing how MySQL is superior to the obviously flawed SQL Server.
Just curious, how many?
Of course there is. The "hello-world" microbenchmark tells you something about the core stack performance of the frameworks relative to one another. This tends to be surprisingly indicative of the relative real-world throughput per node that you may expect for a more complex application.
I.e. Rails (ruby) scores 4 times lower than Python in that micro-benchmark. This is pretty close to the difference that I've observed between real applications on the respective platforms. If anything you'll see the difference magnified in a complex application, but it's unlikely to turn around.
Whether that matters or not in the big picture is a different question (it usually doesn't).
I wouldn't say so. The bottleneck in a hello world benchmark might be a component of the framework, but in complex applications it's likely to be something else. Just look at the "Template Test with DB Query": There, Sqlite is the bottleneck, and the performance difference between Django and Rails fades into background completely.
Nope. Ruby/Python app-servers always end up CPU-bound, unless you're doing something very much out of the ordinary (such as blocking on an embedded SQLite...).
It might actually be an interesting exercise to gradually layer more features on top (such as the dispatcher chain and different templating/ORM layers) to discover inefficiencies in the various frameworks by comparing the relative performance impact.
E.g. if you involve the dispatcher and Django suddenly gets 20% slower but Rails only 10% slower then that might hint at an inefficient dispatcher-impl in Django.
https://github.com/carbonfive/hellod/blob/master/results.md https://github.com/carbonfive/hellod/blob/61bcf7495470350ea1...
The OP benchmark is missing netty and other frameworks meant to be high performance, for higher performance languages.
Tornado is definitely a web framework.
If there are cases where they may overlap, how would you want to develop a test?
That's usually a worthwhile tradeoff (hardware is cheap) but one that a growing number of junior devs don't seem to be making consciously anymore. For them it's just normal to be spending more time per request in the controller code than in the database query. Whereas for many oldtimers it will never cease to feel a little funny...
A side-effect of this trend is that you can't reasonably run your beefy rails-frontend on a mediocre CPU anymore. When latency is dominated by ruby-code crunching on strings then you really want the fastest per-thread performance that money can buy.
I've submitted a pull request[1] to rectify this gross oversight. Even though this benchmark is not particularly scientific, it still provides some meaningful data.
[1] https://github.com/seedifferently/the-great-web-framework-sh...
This may ruffle some feathers of people who love their particular framework and derive satisfaction from using it over others....
I think some other questions it's bringing up are even more interesting:
- If no problem is the same, does any framework matter, except the fact that you have one that decently keeps your organized?
- Is there really a one framework fits all approach? I think not. Some benefit from ORM, some might not.
- We all know premature optimization is not productive. Does it matter what really anyone uses?
- If we don't like trivializing a particular framework to these metrics, why do we trivialize any other framework in general?
- How do we build a better comparison?