I worked with Pat (pfalls) on this effort. He pulled the benchmarks together and built the script to automate the tests. We aimed to deploy each framework/platform according to best-practices for a production environment and then stress test common operations: JSON serialization of objects and database connectivity. We were surprised by the wide spectrum of performance we observed and hope that this interesting to you as well. Four orders of magnitude in one of our tests!
If you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.
I think you may misunderstand Vert.x's polyglot features:
While vert.x supports many programming languages, all of these are run on the JVM runtime. This means when you use the ruby vert.x API, you're using JRuby; likewise with Javascript run through Rhino, Python through Jython, and Groovy/Scala run through their own interpreter/compilers.
That said, it would definitely be interesting to see the performance implications of using one of those languages and vert.x on the JVM.
I'm no expert, but I think certain languages/frameworks are better suited to be behind certain servers when high concurrency is tested. E.g. from http://nichol.as/benchmark-of-python-web-servers it seems django would be better served behind gevent.
There are a lot of variables and tweaking that can be done, and it would be nearly impossible to optimize each.
Similarly, I was wondering what sort of an effect connection pooling would have, as the out of the box django distribution doesn't do that. It really didn't perform too well in their tests.
At LEAST gevent with session write-through caching, psycopg2pool, Postgres SQL (hello, excellent South support?), no unnecessary middlewares or applications that rely on them (if it's a speed oriented use of Django, we're hosting on a specific API sub-domain, right?). At most, THEN you tune the settings to have an optimum number of Postgres threads staying alive and tweak some gunicorn/nginx max connections parameters for your site. If running all locally, use UNIX sockets. This article is trash when it comes to providing any useful data other than Django that's barely been configured beyond not using SQL-Lite, and who the hell uses that in production, so I don't buy their argument about "oh well we just wanted to see what It'd do out of the box" rhetoric. Might as well benchmark ./manage.py runserver. I wish they'd it right or don't publish, let alone publish to shill their company that doesn't provide what they advertise.
Thanks for your pull requests, knappador. We will try to get a revised post out in roughly a week or so with as many of the tweaks we've received (as pulls and tips) as we can muster.
Since these tests seem to be all about JSON serialization, it would've been interesting to see the tests with rails-api instead of the standard Rails stack:
There is a huge difference between raw PHP and CakePHP. I'd be curious to see other PHP frameworks (such as Zend, or Slim) in there-- is Cake just particularly slow, or is that simply what happens when you have a PHP framework?
Per my experience, CakePHP is probably one of the slowest PHP frameworks. It has a large overhead and lots of legacy code that slows down the whole system. Generally, frameworks targeted at PHP 5.3+ are faster. PHP 5.4 has further performance boost in production and PHP 5.5 has Zend Optimizer built-in, which suppose to further speed up but I have never try that in production.
CakePHP is always at the bottom end of these "framework performance" tests even when comparing PHP only. It's the everything but the kitchen sink of PHP frameworks with bells and whistles never used by these "hello world" performance tests. Just the wrong tool for the job if you want to spill out some js.
I'm curious if they configured php with apc or zend optimizer. The huge difference is easy to explain as parse overhead for the framework's code, which happens on each request if you're not using a bytecode cache.
The relation of PHP to everything else in this benchmark is so unusual among benchmarks, that I strongly suspect a failure of parity with other configurations. Hopefully that is unintentional.
Yep, this is exactly what I was wondering. Opcode caching is a key part of production PHP environments - in this case, since the code isn't changing, they should even disable the "change check" (apc.stat=0) as one would do in a production environment.
And if this is the case with their configuration of PHP, it makes me wonder what other platforms are not configured for production in this benchmark :)
If we're going that route, the fastest php performance you would probably get from facebook's HHVM JIT compiling php engine: https://github.com/facebook/hiphop-php
It's no accident that zend optimizer (bytecode cache) is being bundled into php 5.5 as open source, as mere bytecode caching is now not fast enough to charge money for when hhvm is open source. I expect the next version of zend's commercial php server product to contain a JIT engine to match what hhvm can do.
What was the parallelization like across these tests? Were they all running single thread/process mode or did you try to take advantage of threading/multiple-processes/etc. to optimize a production-like performance across all the different frameworks? If the latter, that's a really impressive amount of work!
We attempted to take advantage of threading/multiple-processes as best we could (see the nodejs code for an example of using the cluster module). But we suspect there are additional areas of improvement here.
After having a look at the code on github it looks like they did set up multi processes/threads to really test the scalability on these platforms. That's really impressive, way beyond the single thread experiment I would have anticipated something like this doing! Wow!
The short answer is that we attempted to use the CPU as fully as possible across all frameworks. So for those that had tunable parallelism, we used the settings that seemed best given the number of cores available on each platform (2 for EC2 Large, 8 for i7).
We posted the deployment approach for each framework to the Github page.
Why would you? If you write clean maintainable code and are productive using it there's no problem, there are tons of ways of tuning Cake (as most frameworks). The default configuration is meant for 'bootstrapping'.
However, if you are interested, you can always check out other PHP frameworks (Yii, slim), or if you like all the bell and whistles from Cake and are willing to dive into another language, you can always experiment with ruby (ror, sinatra), node (express) and python (flask).
Not really. Take this benchmark with a grain of salt unless they benchmark it again with APC at minimum. Nobody serious about a web app would run a PHP app without APC.
Similarly, relatively modern deployment standards (not really cutting-edge) like nginx and php-fpm OR apache 2.4 with worker mpm + php-fpm should be added to the mix.
apc.enable=1 turns on Opcode caching (when php-apc is installed), and apc.stat=0 turns off "stat" checks - this means that once a file is opcode cached, PHP won't even have to touch the file on disk to execute it. The I/O gains from this, as well as the execution gains from not having to parse the file, should help quite a bit.
All data is returned as an associative array, so there is no way to call something like:
$cows = $this->Cows->findByStatus(COW_STATUS_NOT_MOOING);
foreach($cows as $cow)
{
$cow->moo();
}
But that's pretty minor.
The big issue is: all data is returned as an associate array! There is no lazy-loading of the data as you e.g. seek through it, which means it's very easy for people to pass around huge arrays with thousands upon thousands of rows and associations without even thinking about it.
I don't use Cake, so don't read this is a defence of Cake.
Lazy loading seems to me a kludge trying to fix poor SQL querying. If you can filter at the app layer, why not pass the information down and filter at the DB layer? You'll be transferring less data and may even read less data from disk (with proper indexes)
Lazy loading seems to me a kludge trying to fix poor SQL querying.
I very much agree with you, lazy loading should not be used as substitute for only querying and/or filtering the data you actually need.
However, there's always going to be a minimum; eventually, the data needs to be processed or displayed! Why load all of it immediately into memory (and potentially pass that around), rather than when it's actually needed?
I think of it a bit like using pointers; passing around the location of the data, rather than the data itself, makes life good for all concerned.
I don't use Cake, so don't read this is a defence of Cake.
Thanks for the disclaimer, as I'm guessing you've experienced before, it's all too easy to get into accidental flamewars about this stuff :)
If you're serving only json snippets from memory and might have hundreds of simultaneous users, yes!
If you have existing sites built with it which work fine, then this benchmark doesn't tell you a great deal other than that php is relatively slow, and frameworks built upon it slower. For many websites which have a caching strategy sitting behind a server like apache or nginx and less than a few thousand users a day, this really doesn't matter, and other things like features are more important. This hlds even for bigger site too - Facebook for example still runs PHP (compiled to one big binary). Personally I wouldn't use PHP because of the language/std lib but this sort of performance shoot-out should not put you off it.
Obviously, the data should not steer you towards rewriting applications using a different framework. Really, it should just show as another metric to help deciding which framework is best suited for your needs. If you plan on writing up a blog application for yourself and you might get 100 concurrent users, then you could use CakePHP for the sake of familiarity. Then again, if you are trying to write the next Twitter and have the requirement of needing to be able to service 20,000 concurrent requests, then I would say "yes, this means you should stop using CakePHP."
I'd like to see how .Net MVC would compare. I realize you'd have to spin it up on a Windows EC2 instance and there would definitely be some variance in the performance of that box vs. the nix EC2 instances but I'd still be interested in seeing how it fares in comparison.
We agree. We aim to provide a .NET MVC test soon. We did briefly test that on our i7 hardware and if memory serves me correctly, it clocked in at around the position as Spring.
Interesting - C# should have also been added. Java still rules the roost. If you should absolutely have a scripting language - old php with warts is better. More magic in the framework - more abstraction and indirection and code inefficiency - that can be the spoil port when it comes to performance.
pfalls - amazingly, I spent the last 2 days of my holiday doing the same thing for a future open source project. I was just stumped when I saw you guys did the same (could have saved me a couple of days!)
I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it.
Scala took the lead with amazing results. More over, a good metric was latency which Scala was the only one to take micro-second resolution.
I'm not a fan of Scala because of its surrounding tools, which is why I'm still considering going for either Clojure or Node.js.
I think the most surprising positively was Clojure, being that it is a dynamic language. And most surprising negatively was Go - by itself is impressive, but when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to see this correlates with your findings too; I'm assuming this is a symptom of library maturity..?
I'd be happy to see how Scala fares on your tests.
This started out as a small exercise, that quickly ballooned because we were curious about every framework and platform. Obviously we had to stop somewhere, but we're very interested in adding more tests in the future. In fact, we're hoping the community will help us out as well!
What started as a couple of hours of exercise for myself ended up as 2 days of hacking and barely sleeping, as surprises in my assumptions kept unfolding, and as I wrote and rewrote POCs just to validate that Clojure is as fast as the number say, and that Scala is faster than Java, etc.
Dropwizard looks awesome, just the kind of project I've been looking for, also it links to JDBI which is very similar to a sql lib I maintained for myself all these years and looks awesome. Thanks for posting!
Would you consider releasing some of the benchmark code/setup? (better yet integrated into OP's project) I am very interested in seeing clojure's performance. What kind of framework did you use for clojure?
Symfony and Zend might perform even worse. At least Zend's bootstrapping is even heavier. The sheer number of files that have to be loaded from disk on each hit is appalling. Hence I mentioned in another comment the opcode cache (ex. APC) requirement and how every serious PHP app utilizes it.
Yes, I'd definitely like to see Slim tested as a light framework. People keep mentioning Silex as a "light" framework, but I've found it to be almost as heavy as Symfony2 (which isn't much of a surprise considering it's just Symfony2 components strapped together).
One of the most interesting things this comparison brings out to me is not so much the differences between the various frameworks (although the differences between options on the same platform is definitely very useful information), but also the issue that few of us seem to think about these days: the cost of any of the frameworks above the bare standard library of the platform its hosted on.
Theres a consistant, considerable gap between their "raw" benchmarks (things like netty, node, plain php, etc.) and frameworks hosted on those same platforms. I think this is something we should keep in mind when we're tuning performance-sensitive portions of APIs and the like. We may actually need to revisit our framework choice and implement selected portions outside of it (just like ruby developers sometimes write performance-critical pieces of gems in C etc.) or optimize the framework further.
I'd like to crunch these numbers further to get a "framework optimization index" which would be the percentage slowdown or ratio of performance between the host platform and the performance of the framework on top of it. I might do this later if I get a chance.
I think Go is almost the ideal example here, you're right. Go provides a pretty rich "standard library" for writing web serving stuff so it's a good place where you could really imagine writing your performance critical stuff just on the base platform even if you use something like Webgo for the rest of your app.
Some of the other platforms are much less amenable to that since the standard primitives the platform exposes are very primitive indeed (servlets api, rack api, etc.). Perhaps there's some value in looking at how your favorite framework stacks up against its raw platform and trying to contribute some optimizations to close the gap a bit.
Webgo... I'll stick with "net/http" and Gorilla thanks.
Also, They used Go 1.0.3... I hope they update to 1.1 next month. Most everyone using Go for intensive production uses is using Go tip (which is the branch due to become 1.1 RC next month)
This is great to know, we were hesitant to use non-stable versions (although we were forced to in certain cases), but knowing that it's what is common practice for production environments would change our minds.
We switched to using tip after several Go core devs recommended that move to us, the folks on go-nuts IRC agreed and we tested it and found it to be more stable than 1.0.3
A good tip build tends to be more stable then 1.0.3 and has hugely improved performance (most importantly for large application in garbage collection and generation).
To select a suitable tip build we use http://build.golang.org/ and https://groups.google.com/forum/?fromgroups#!forum/golang-de... . My recommendation would be to find a one or two week old build that passed all checks, do a quick skim of the mailing list to make sure there weren't any other issues and use that revision. Also, you will see some the the builders are broken-
Of course if your application has automated unit tests and load tests, run those too before a full deployment.
Thanks, this comment really helped me in my evaluation of Go today. I had been playing around with 1.0.3 for a couple days, but tip is definitely where it's at.
Or it could speak poorly of their release process, which is more accurate. The stable release is simply so bad compared to tip that everyone uses tip. There should have been multiple releases since the last stable release so that people could get those improvements without having to run tip.
Why not both? Insisting on a very stable API can result in long times between releases, which can mean more people using tip. That's distinct from how stable tip is.
Given the frequent complaints that the previous stable release isn't very stable, I think trying to interpret it as "tip is super stable" is wishful thinking. Tip is currently less bad than stable. The fact that stable releases are not stable is a bad thing, not a good thing.
What does stable mean? If stable means there are not unexpected crashes, then Go 1.0.3 is extremely stable.
If stable means suitable for production, Go tips vastly improved performance, especially in regards to garbage collection, make it more suitable than 1.0.3 for large/high-scale applications in production.
I'm curious about that - because there's so little to webgo I suspect the answer is something really trivial. I haven't really looked at it before, but the framework itself is just 500 lines or so unless I'm looking at the wrong one...
Given that the json marshalling and server components would be exactly the same between go and webgo, I'm curious as to whether changing the url recognised to be just /json in the goweb tests would make any difference, any reason it was different?
Shouldn't all these examples at least be trying to do the same sort of work? For such a trivial test differences like that could make a huge difference to the outcome.
It's great to see replicable tests like this which show their working, but they do need to be testing the same thing. I also think they should have something more substantial to test as well as json marshalling on all the platforms, like serving an HTML page made with a template and with the message, as that'd give a far better indication of speed for tasks web servers typically perform.
Still, it's a great idea and really interesting to see this sort of comparison, even if it could be improved.
One of the next steps we'd like to take is to have a test that does cover a more typical web request, and is less database heavy than our 20 query test, just like you describe. Ultimately, we felt that these tests were a sufficient starting point.
I was a little confused by the different urls used in the tests, as for this sort of light test, particularly in Go, where all the serving stuff is common between frameworks, you're mostly going to be testing routing. Any reason you chose a different route here? (/json versus /(.★) )?
I can't think of much else that this little web.go framework does (assuming the fcgi bits etc are unused now and it has moved over to net/http). I don't think many people use web.go, gorilla and its mux router seems to be more popular as a bare bones option on Go, so it'd possibly be interesting to use that instead. It'd be great to see a follow up post with a few changes to the tests to take in the criticisms or answer questions.
While you may come in for a lot of criticism and nitpicking here for flaws (real or imagined) in the methodology, I do think this is a valuable exercise if you try to make it as consistent and repeatable as possible - if nothing else it'd be a good reference for other framework authors to test against.
I'm not sure many people would use webgo in real life. I don't know... maybe some people... certainly not pros.
Also, the 1.0.3 thing is probably dragging on the numbers a bit. 1.1 would boost it a little. Not enough to get it into the top tier... but a little.
Also, for Vert.x, they seem to be only running one verticle. Which would never happen in real life.
Play could be optimized a bit... but not much. What they have is, to my mind, a fair ranking for it.
Small issues with a few of the others but nothing major. I think Go and Vert.x are the ones that would get the biggest jumps if experts looked at them. And let's be frank... does Vert.x really need a jump?
So what they have here is pretty accurate... I mean... just based on looking through the code. But Go might fare better if it used tip. And Vert.x would DEFINITELY fair better with proper worker-nonworker verticles running.
OK... I ran the Vert.x test... runs a bit faster here with 4 workers instead of 8. I suspect what is happening there is that at times all 8 cores can be pinned by workers, while responses wait to be sent back on the 8 non workers. But not that big a change in speed actually. One thing more, when you swap in a couchbase persister for the mongo persister it's faster yet. The difference is actually much larger than the difference you get balancing the number of worker vs non worker verticles. Also thinking that swapping gson in for jackson would improve things... but I don't think that those are fair changes. (well... the couchbase may be a fair change)
Also tested Cake just because it had been a while since I have used it... and I couldn't believe it was that much worse than PHP. Your numbers there seem valid though given my test results. That's pretty sad.
Finally, tried to get in a test of your Go stuff. I'm making what I think are some fair changes ... but it did not speed up as much as I thought. In fact, initially it was slower than your test with 1.1.
The Play example was totally unfair since it blocks on the database query which will block the underlying event loop and really lower the overall throughput.
Could you point me at an example of an idiomatic, non-trivial Go REST/JSON API server? I've been trying for a while to find something to read to get a better handle on good patterns and idiomatic Go, but I haven't really come up with anything. I've found some very good examples of much lower-level type stuff, but I think I have a decent handle on that type of Go already. What I really would like is a good example of how people are using the higher level parts of the standard library, particularly net/http etc.
We have a pull request that changes the Play benchmark (thank you!) so we will be including that in a follow-up soon.
We tested node.js with both Mongo and MySQL. Mongo strikes us the more canonical data-store for node, but wanted to include the MySQL test out of curiosity.
I used Go's benchmarking tool to compare raw routing performance of various frameworks. The handlers all return a simple "Hello World" string. Here are the results:
In the first test, Pat is almost twice as a fast as the Gorilla framework. In the second test, when we added a bit more logic to the handler (marshaling a struct to JSON), Pat was only about 18% faster than Gorilla. In fact, it turns out it takes longer to serialize to JSON (8000ns) than it does for Pat to route and serve the request (6000ns).
Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
I would personally choose my framework not just based on performance, but also based on productivity. One that can help me write code that is easier to test and easier to maintain in the long run.
> Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
If you're performing a DB query on every request, you're doing something wrong. In the real world your app will check Memcached, and if there's a cached response, it will return it. Thus making the framework performance quite important.
rorr, you appear to be hellbanned. Here's your comment, since it seemed like a reasonable one:
> Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
If you're performing a DB query on every request, you're doing something wrong. In the real world your app will check Memcached, and if there's a cached response, it will return it. Thus making the framework performance quite important.
Notice the top 3 frameworks (pat, routes and Gorialla) have almost identical performance results. The point being is that routing and string manipulation are relatively inexpensive when compared to even the most lightweight TCP request, in this case to the memcache server.
I'd love to see your framework optimization index. Honestly, all of this would be a wonderful thing to automate and put in a web app - a readily-accessible, up-to-date measure of the current performance of the state of the art in languages and frameworks. I bet it would really change some of the technology choices made.
Here's a quick version of the framework optimization index. Higher is better (ratio of framework performance to raw platform performance, multiplied by 100 for scale):
These could probably be further broken down into micro-frameworks (like Express, Sinatra, Vert.x etc.) and large MVC frameworks (like Play and Rails).
Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected.
Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young.
Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
The play test was written totally incorrectly since it used blocking database calls. Since play is really just a layer on top of Netty it should perform nearly as well if correctly written.
That's because they inexplicably use the Jackson library for the simple test, rather than Play's built in JSON support (they use the built-in JSON for the other benchmarks).
Both Netty and Play use Jackson though one the Netty version uses a single ObjectMapper and the Play version uses a new ObjectNode per request (created through Play's Json library).
I'd imagine that being relatively young is an advantage in a test like this. You're not utilizing any features, and features are what slow down requests. The less features something has, the faster it should perform in these trivial tests.
It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily interpreted as too much editorializing. Nevertheless, as you point out, it's interesting to know how using a framework impacts your performance versus the underlying platform.
I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
For Play, you'll want to either 1) handle the (blocking) database queries using Futures/Actors that use a separate thread pool (this might be easier to do in Scala) or 2) bump the default thread pool sizes considerably. The default configuration is optimized purely for non-blocking I/O.
I'm pretty shocked that Play scored so low. One would think that being built on netty would put Play in a higher rank. Database access for sure needs to be in async block
Good thing you opted against sensational journalism.
About the cost per additional line of code for PHP, it mainly comes from not having an opcode cache and having to load and interpret files on every visit. mod_php was and will always be trash. I commented earlier about it too.
In case of Ruby, and talking about Rails, even when using Passenger, the rails app is cached via a spawn server. That's not the case with PHP.
Similarly, Python has opcode cache built-in (.pyc files). Also, I am not sure about gunicorn but others do keep app files in-memory.
Servlet does the same about keeps apps in memory. You get the idea.
Frameworks definitely have an impact but it's very hard for one person to know the right configuration for every language. You had done some good work there, but it will take a lot of contributions before the benchmark suite becomes fair for every language/framework.
We've received a lot of great feedback already and even several pull requests. Speaking of, we want to express a emphatic "Thank you" to everyone who has submitted pull requests!
We're hoping to post a follow up in about a week with revised numbers based on the pull requests and any other suggested tweaks we have time to factor in. We're eager to see the outcome of the various changes.
I am completely agree with you, this is not proper bench marking as opcode caching is missing in php, benchmarking should be re calculated by configuring APC.
In the same vein, I was curious to compare the max responses/second on dedicated hardware vs ec2 on a per framework basis. The following is percentage throughput of ec2 vs dedicated (in res/s):
cake 18.9% (312 vs 59)
compojure 12.1% (108588 vs 13135)
django 16.8% (6879 vs 1156)
express 16.9% (42867 vs 7258)
gemini 12.5% (202727 vs 25264)
go 13.3% (100948 vs 13472)
grails 7.1% (28995 vs 2045)
netty 18% (203970 vs 36717)
nodejs 15.6% (67491 vs 10541)
php 11.6% (43397 vs 5054)
play 20.6% (25164 vs 5181)
rack-jruby 15.6% (27874 vs 4336)
rack-ruby 22.7% (9513 vs 2164)
rails-jruby 22.7% (3841 vs 871)
rails-ruby 20.7% (3324 vs 687)
servlet 13.4% (213322 vs 28745)
sinatra-jruby 21.2% (3261 vs 692)
sinatra-ruby 22.2% (6619 vs 1469)
spring 7.1% (54679 vs 3874)
tapestry 5.2% (75002 vs 3901)
vertx 22.3% (125711 vs 28012)
webgo 13.5% (51091 vs 6881)
wicket 12.7% (66441 vs 8431)
wsgi 14.8% (21139 vs 3138)
I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown.
Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2.
Disclosure: I am a colleague of the author of the article.
You're saying that running a query across the internet to ec2 is 5 times faster than running it on dedicated hardware in the lab? I find that hard to believe.
Sorry, maybe my original post was not entirely clear. Let's take tapestry, for example. On dedicated hardware, the peak throughput in responses per second was 75,002. On ec2, it was 3,901 responses per second.
So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.
If you have developers who for one reason or another prefer a given platform, then the most important performance comparisons are about how close various frameworks on that platform get to the performance of the platform itself.
Knowing how much I'm giving up in performance in order to get the features a given framework gives me is an important consideration. Also understanding when it's worthwhile to work outside the framework on the bare platform given the speedup I'll get versus the cost I'll incur by doing so is a very important optimization decision-making tool.
There are performance numbers for a framework (Cake PHP, for example) and for the raw primitives of the platform it runs on (PHP, in that case). By finding the ratio between the two one can arrive at the performance loss attributable primarily to the framework you've chosen.
See my "framework optimization index" in comments below for a rundown on all these ratios which I was able to back out from this set of benchmarks.
I'm torn about this. On the one hand, while I've known my framework of choice (Rails) is slow, I didn't know how much slower it could be in the grand scheme of things. But on the other hand, I'm more shocked by the difference between EC2 and dedicated hardware (10x improvement with rails), and even 89 requests per second (20 query benchmark on EC2) is still a decent amount of traffic. (Plus this doesn't count any optimizations I would make anyway, like caching).
Either way good architectures usually optimize the high traffic or high CPU areas anyway away from a scripted language.
Thanks for the really informative post! Go seems to be a good balance as a high performance language without having to go back to my traumatic Java days.
Thanks for the feedback, atonse! We had a great amount of fun putting this together, as you can imagine.
I agree, a remarkable take-away for us was how dramatically our i7s excelled over the EC2 instances. Admittedly, those were EC2 Large and not Extra Large instances.
A previous draft of this blog entry had a long-winded analysis of hosting costs--discussing the balance between ease and peace-of-mind provided by something like AWS versus the raw performance of owned hardware--but we elected to remove that since it wasn't really the point of the exercise.
As a rule I like to divide this world into "Featureless" and "Featurefull" products.
When you use Rails, you're aiming to pile up features. You want to react to Product managers, to users, you want to work fast and satisfy the needs of customers - or else you won't have anyone to build to.
In this reality, the fact that you're doing 20req/s is OK. In fact, I'm betting that even when you take Go or Node.js - pile up all of the infrastructure and features that exist in Rails, and pile up a ton of your code - buggy and not buggy - you'll get around the same kind of satisfaction index from users.
This is because your product can be perceived as slow even though your servers are blazingly fast.
On the other side of the spectrum there are "Featureless" products. These are infrastructural products. A logging service. An analytics service. A full-text search. A classification and recommendation engine.
These you don't want to build in Rails. I'm sure you haven't even considered it. These you want to build with one of the top-notch libraries that this survey indicate.
Also, there are certain features about rails like thin or unicorn that can drastically increase your overall performance. So in that sense, I think it's a lot more complicated to determine.
EC2 was a hard platform to test on, only because our i7 hardware would give us results fairly instantaneously, but we became impatient when we had to wait upwards of 10x as long for the data on the EC2 large instances.
We're actually very interested in how the large/newer instances perform.
I Will love to see dotNet (C#) incluid in this test.
Asp.Net WebAPI (Synchronous and Asycnhrounous)
Asp.Net MVC (Synchronous and Asycnhrounous)
Asp.Net HTTP Handlers (Synchronous and Asycnhrounous)
Also ServiceStack vs ASP.NET MVC vs NancyFX vs Fubu, Mono vs IIS/windows, default serializer vs JSON.NET vs ServiceStack's, etc. There are tons of variations that could be done.
Interesting thought. Would you be able to put together a test for one of those? pfalls can give you the details for how we expect the test for a new framework to work.
No, they're not. They can run via CGI (so, yes, compiled for each request) but that's slow as ass. They run via Plack/PSGI - they can be deployed in a multitude of ways, including via FastCGI or running standalone using built-in webservers, or via Starman - the latter in particular is very fast indeed. With a simple "hello world" type app, > 6000 requests/sec can be easily handled. Obviously an app of more realistic complexity won't be quite that fast, but you'll still handle many requests every second with no trouble. http://stackoverflow.com/a/4770406/4040 contains some basic benchmarks.
EDIT: also, mod_perl is generally best avoided these days; it's old, not very pleasant to work with, and ties you to Apache. Writing an app with Dancer / Mojolicious etc means you can deploy in various different ways with ease.
"Oh, they're just frameworks for CGI/Perl and thus severely crippled by CGI's mode" - This isn't true.
Dancer use PSGI (not CGI) and most modern Perl webapp frameworks (Mojo & Catalyst) use PSGI/Plack (http://plackperl.org/). The PSGI model is very similar to wsgi which allows for different types of persistent servers.
I'd be interested to know this as well. I can understand time constraints in compiling these kind of comparisons, but as I'm about to embark on a web project in Go, I'd be interested to know if there was any technical constraints that prevented benchmarking Go.
We recognize that deficiency and we do aim to add database tests for Go. If you could quickly draft that up and submit a pull request, we'd love to add it.
Sorry, but as much as I'd love to contribute I'm still learning how to write database apps in Go myself (that's actually part of the reason I wanted to see your tests). So I'd be the wrong person to to write code for that particular test.
How about Lift? Btw, the play framework you tested is Java or Scala based?
Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!!
Maybe Node.js critics should start appreciating it after all..
It is probably worth noting that while we strive to make the tests as fair as possible, we followed the official tutorials for each framework when building out the tests but we fully expect there to be small instances where minor tweaks improve a given test. Given that I am no Play expert, it would be of great value to have one who is (and it sounds like you could lend a hand there) to check out the code on the github page. If we did anything wrong with the setup or in general, we would rerun the tests. Again, we followed all the official 'getting started' posts for each framework, so we believe we have best practices used.
Disclaimer: I am a colleague of the author of the linked article.
The problem is that the database queries were done in a blocking fashion. The test essentially blocks the main event loop which is of course going to kill performance.
There are a few problems with your Play code that are causing it to be unnecessarily slow.
First-- what you're really testing here is the Jackson library. A majority of the cycles used in your application are being burned in that toJson call of an array of objects. This isn't a fair test compared to the servlet implementation because you're calling Jackson against a map in the Play example, versus against a simple String in the servlet example.
Second-- you are running database calls serially, and those are blocking. Considering that you're using the more-or-less default Play/Akka configuration, there are only enough threads as you have available processor cores. I would start by increasing the parallelism-factor and parallelism-max to be higher, so you'll have more available threads. More importantly though, the database access should be wrapped in a Future, and you should be returning asynchronously. This should speed up the application by a huge amount.
Do you think such a configuration could outrun the Vert.x configuration they've posted? I'm not challenging you, I'm just genuinely curious! Because if Play+Akka can outrun Vert.x, then it would be an interesting game altogether...
I think Play, with well-written asynchronous code, could approach the Netty/Vert.x speed. In other words, I'd be willing to trade the ease-of-use of Play for the slight speed impairment vs. writing directly to Netty/Vert.x/etc.
Not sure if your theory can really account for the 7X difference between Play and Netty in the JSON test. They both create a trivial name/value pair and pass it to the JSON serializer. In Play's case, it gets passed to their Jerkson wrapper for Jackson. What would you even change about that code?
(Note: I'm not necessarily saying the Jerkson wrapper is the culprit. Could be Play's routing framework, or something else.)
Would love to see how these results compare to some of the web frameworks for concurrent functional languages like Erlang/Haskell: Nitrogen, Chicago Boss, Snap, Yesod, etc.
Please don't encourage them. Even if Haskell comes out on top, I would still be unsatisfied because the rest of the benchmarks are unfair. Lies by confusion are still lies.
Please don't encourage them. Even if Haskell comes out on top, I would still be unsatisfied because the rest of the benchmarks are unfair. Lies by confusion are still lies.
If I wanted to return a bunch of _ids from a mongo database i would do a _id:$in[array of ObjectIDs] and then stream the resulting cursor to res.
Streams are core to node. You should use them.
This seems to be a nice benchmark. For the Python group, I would suggest two things: (1) include a lightweight framework like Bottle, and (2) Try pypy.
Thanks. Agreed, we'd love to get a Python micro-framework in the test. If you have some free time and feel like putting together a test for Bottle as a Github pull request, we'd really appreciate that.
I'd be curious about a gevent-worker-driven server, too. Presumably they're running Django on either a thread- or processor-driven concurrency server, and gevent can show some pretty major gains. You could also do permutations of these: gevent on pypy (using pypycore), etc.
We'd love to have ASP.Net MVC included. One minor gotcha is that to do it justice, we'd need to spin up a Windows EC2 instance and figure out how to script that. It's on our to-do list!
We did briefly test ASP.Net on Mono (see another comment in this thread) but didn't include it since we didn't believe that qualifies as a "production" grade ASP.Net MVC deployment.
Since these benchmarks are so wide-ranging, I agree. But that means setting up an entire new testbed on Windows, and then trying to make it comparable to the other platform testbed; possibly tuning. You need a Windows expert to do this. My question is, why aren't Windows experts setting these up?
If you can cover ASP.Net MVC, then I'd recommend including ServiceStack. My own tests of their JSON implementation have shown it to be 5-10x faster than ASP.net MVC.
If you seriously thought that the difference between a netty and django would be 4 times then you simply don't understand what these frameworks do in the first place.
I would have guessed something like less than 100 times slower and that would have been still fine since the cost of all kinds of latencies in the system usually far outweigh the speed of the framework itself.
What's the difference between "php" and "php-raw" in some of the data? Maybe I'm still in my morning fog, but having trouble thinking of what "php-raw" might mean. Sigh.
The code says php-raw refers to using PDO (which all project should be using) vs using an ORM or ActiveRecord.
I am completely stunned by the performance cost of using ORM/AR, and will be using this to shame our team lead into giving it up and going for raw queries.
Important to note that OP doesn't mention using APC with this, something that any production code would be using in a real world case. This would have an impact on the numbers he uses and on the diff we are seeing.
Don't be so hasty. As others pointed out, caching often plays a factor on larger projects. A degree of portability provided by ORMs is nice (test mode hits SQLite, prod is pg or mysql, etc). Also, the readability of ORM-oriented code shouldn't be overlooked, especially as new people come in to the team. Two or three expressive lines in ORM vs 15-20 of nested SQL hitting weird tables names and aliased column names and such isn't easy to decipher the intent of (especially when there's a problem).
I typically use ORM for about 95% of a project, falling back to a few explicitly native SQL calls when performance can be shown to be a bottleneck in those locations.
Hi Chasing. We put a note about that suffix in the "Environment Details" section. The "raw" suffix means there is no ORM used. If there is no "raw" suffix, you can assume some type of ORM is used.
414 comments
[ 2.9 ms ] story [ 183 ms ] threadIf you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.
While vert.x supports many programming languages, all of these are run on the JVM runtime. This means when you use the ruby vert.x API, you're using JRuby; likewise with Javascript run through Rhino, Python through Jython, and Groovy/Scala run through their own interpreter/compilers.
That said, it would definitely be interesting to see the performance implications of using one of those languages and vert.x on the JVM.
Similarly, I was wondering what sort of an effect connection pooling would have, as the out of the box django distribution doesn't do that. It really didn't perform too well in their tests.
https://github.com/rails-api/rails-api
What webserver were you using on JRuby? Was it Trinidad? Did you try Jetpack?
And if this is the case with their configuration of PHP, it makes me wonder what other platforms are not configured for production in this benchmark :)
It's no accident that zend optimizer (bytecode cache) is being bundled into php 5.5 as open source, as mere bytecode caching is now not fast enough to charge money for when hhvm is open source. I expect the next version of zend's commercial php server product to contain a JIT engine to match what hhvm can do.
I really hope the author picks up on this and compares it. It could provide some tremendous insights into pushing PHP further into this space.
We posted the deployment approach for each framework to the Github page.
However, if you are interested, you can always check out other PHP frameworks (Yii, slim), or if you like all the bell and whistles from Cake and are willing to dive into another language, you can always experiment with ruby (ror, sinatra), node (express) and python (flask).
Similarly, relatively modern deployment standards (not really cutting-edge) like nginx and php-fpm OR apache 2.4 with worker mpm + php-fpm should be added to the mix.
apc.enable=1 turns on Opcode caching (when php-apc is installed), and apc.stat=0 turns off "stat" checks - this means that once a file is opcode cached, PHP won't even have to touch the file on disk to execute it. The I/O gains from this, as well as the execution gains from not having to parse the file, should help quite a bit.
I always strongly advise not using CakePHP, purely because its model system is so broken, it makes my teeth itch.
All data is returned as an associative array, so there is no way to call something like:
But that's pretty minor.The big issue is: all data is returned as an associate array! There is no lazy-loading of the data as you e.g. seek through it, which means it's very easy for people to pass around huge arrays with thousands upon thousands of rows and associations without even thinking about it.
/rant over
Lazy loading seems to me a kludge trying to fix poor SQL querying. If you can filter at the app layer, why not pass the information down and filter at the DB layer? You'll be transferring less data and may even read less data from disk (with proper indexes)
I very much agree with you, lazy loading should not be used as substitute for only querying and/or filtering the data you actually need.
However, there's always going to be a minimum; eventually, the data needs to be processed or displayed! Why load all of it immediately into memory (and potentially pass that around), rather than when it's actually needed?
I think of it a bit like using pointers; passing around the location of the data, rather than the data itself, makes life good for all concerned.
I don't use Cake, so don't read this is a defence of Cake.
Thanks for the disclaimer, as I'm guessing you've experienced before, it's all too easy to get into accidental flamewars about this stuff :)
If you have existing sites built with it which work fine, then this benchmark doesn't tell you a great deal other than that php is relatively slow, and frameworks built upon it slower. For many websites which have a caching strategy sitting behind a server like apache or nginx and less than a few thousand users a day, this really doesn't matter, and other things like features are more important. This hlds even for bigger site too - Facebook for example still runs PHP (compiled to one big binary). Personally I wouldn't use PHP because of the language/std lib but this sort of performance shoot-out should not put you off it.
But don't quote me on that! :)
(win | mono) + (httphandler | asp.net mvc | webapi | servicestack | nancyfx)
it would hopefully compare with java stacks!
I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it.
I tested: Go, Java (servlet, dropwizard), Scala (scalatra), Ruby, Node.js (connect).
For me it was:
* Scala
* Java
* Clojure (equal to Java - big surprise here)
* Node.js
* Go (almost equal to Node.js)
* Ruby (far far down)
Scala took the lead with amazing results. More over, a good metric was latency which Scala was the only one to take micro-second resolution.
I'm not a fan of Scala because of its surrounding tools, which is why I'm still considering going for either Clojure or Node.js.
I think the most surprising positively was Clojure, being that it is a dynamic language. And most surprising negatively was Go - by itself is impressive, but when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to see this correlates with your findings too; I'm assuming this is a symptom of library maturity..?
I'd be happy to see how Scala fares on your tests.
You've done an awesome job!
This started out as a small exercise, that quickly ballooned because we were curious about every framework and platform. Obviously we had to stop somewhere, but we're very interested in adding more tests in the future. In fact, we're hoping the community will help us out as well!
What started as a couple of hours of exercise for myself ended up as 2 days of hacking and barely sleeping, as surprises in my assumptions kept unfolding, and as I wrote and rewrote POCs just to validate that Clojure is as fast as the number say, and that Scala is faster than Java, etc.
I'm working with a setup like this and just love it!
We'd also love to have anyone who has production experience with those to contribute a test for them. The test should be fairly quick to write.
I'd also like to see a "light" framework meant for building APIs, like Slim, for instance.
Theres a consistant, considerable gap between their "raw" benchmarks (things like netty, node, plain php, etc.) and frameworks hosted on those same platforms. I think this is something we should keep in mind when we're tuning performance-sensitive portions of APIs and the like. We may actually need to revisit our framework choice and implement selected portions outside of it (just like ruby developers sometimes write performance-critical pieces of gems in C etc.) or optimize the framework further.
I'd like to crunch these numbers further to get a "framework optimization index" which would be the percentage slowdown or ratio of performance between the host platform and the performance of the framework on top of it. I might do this later if I get a chance.
Some of the other platforms are much less amenable to that since the standard primitives the platform exposes are very primitive indeed (servlets api, rack api, etc.). Perhaps there's some value in looking at how your favorite framework stacks up against its raw platform and trying to contribute some optimizations to close the gap a bit.
Also, They used Go 1.0.3... I hope they update to 1.1 next month. Most everyone using Go for intensive production uses is using Go tip (which is the branch due to become 1.1 RC next month)
To select a suitable tip build we use http://build.golang.org/ and https://groups.google.com/forum/?fromgroups#!forum/golang-de... . My recommendation would be to find a one or two week old build that passed all checks, do a quick skim of the mailing list to make sure there weren't any other issues and use that revision. Also, you will see some the the builders are broken-
Of course if your application has automated unit tests and load tests, run those too before a full deployment.
If stable means suitable for production, Go tips vastly improved performance, especially in regards to garbage collection, make it more suitable than 1.0.3 for large/high-scale applications in production.
Given that the json marshalling and server components would be exactly the same between go and webgo, I'm curious as to whether changing the url recognised to be just /json in the goweb tests would make any difference, any reason it was different?
http://localhost:8080/json
http://localhost:8080/(.*)
Shouldn't all these examples at least be trying to do the same sort of work? For such a trivial test differences like that could make a huge difference to the outcome.
It's great to see replicable tests like this which show their working, but they do need to be testing the same thing. I also think they should have something more substantial to test as well as json marshalling on all the platforms, like serving an HTML page made with a template and with the message, as that'd give a far better indication of speed for tasks web servers typically perform.
Still, it's a great idea and really interesting to see this sort of comparison, even if it could be improved.
I can't think of much else that this little web.go framework does (assuming the fcgi bits etc are unused now and it has moved over to net/http). I don't think many people use web.go, gorilla and its mux router seems to be more popular as a bare bones option on Go, so it'd possibly be interesting to use that instead. It'd be great to see a follow up post with a few changes to the tests to take in the criticisms or answer questions.
While you may come in for a lot of criticism and nitpicking here for flaws (real or imagined) in the methodology, I do think this is a valuable exercise if you try to make it as consistent and repeatable as possible - if nothing else it'd be a good reference for other framework authors to test against.
Also, the 1.0.3 thing is probably dragging on the numbers a bit. 1.1 would boost it a little. Not enough to get it into the top tier... but a little.
Also, for Vert.x, they seem to be only running one verticle. Which would never happen in real life.
Play could be optimized a bit... but not much. What they have is, to my mind, a fair ranking for it.
Small issues with a few of the others but nothing major. I think Go and Vert.x are the ones that would get the biggest jumps if experts looked at them. And let's be frank... does Vert.x really need a jump?
So what they have here is pretty accurate... I mean... just based on looking through the code. But Go might fare better if it used tip. And Vert.x would DEFINITELY fair better with proper worker-nonworker verticles running.
Also tested Cake just because it had been a while since I have used it... and I couldn't believe it was that much worse than PHP. Your numbers there seem valid though given my test results. That's pretty sad.
Finally, tried to get in a test of your Go stuff. I'm making what I think are some fair changes ... but it did not speed up as much as I thought. In fact, initially it was slower than your test with 1.1.
So after further review... well done sir.
the Vert.x example, as configured, blocks massively as well waiting for mongodb queries.
We tested node.js with both Mongo and MySQL. Mongo strikes us the more canonical data-store for node, but wanted to include the MySQL test out of curiosity.
>> I'm not sure many people would use [xxx] in real life. I don't know... maybe some people... certainly not pros.
Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.
I would personally choose my framework not just based on performance, but also based on productivity. One that can help me write code that is easier to test and easier to maintain in the long run.
If you're performing a DB query on every request, you're doing something wrong. In the real world your app will check Memcached, and if there's a cached response, it will return it. Thus making the framework performance quite important.
> Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck. If you're performing a DB query on every request, you're doing something wrong. In the real world your app will check Memcached, and if there's a cached response, it will return it. Thus making the framework performance quite important.
https://plus.google.com/u/0/115863474911002159675/posts/L3o9...
More SPEEEEEEEEEEEEED coming down the pipe in 1.1 for Go's net/http. :)
Framework Framework Index
Gemini 87.88
Vert.x 76.29
Express 68.85
Sinatra-Ruby 67.88
WebGo 51.08
Compojure 45.69
Rails-Ruby 31.75
Wicket 29.33
Rails-Jruby 20.09
Play 18.02
Sinatra-Jruby 15.96
Tapestry 13.57
Spring 13.48
Grails 7.11
Cake 1.17
Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected.
Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young.
Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
Here's the source: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
So the question stands: If Play & Netty are using the same JSON serialization code, why is Play seven times slower?
I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
See e.g. https://github.com/playframework/Play20/blob/master/document... and https://gist.github.com/guillaumebort/2973705 for more info.
About the cost per additional line of code for PHP, it mainly comes from not having an opcode cache and having to load and interpret files on every visit. mod_php was and will always be trash. I commented earlier about it too.
In case of Ruby, and talking about Rails, even when using Passenger, the rails app is cached via a spawn server. That's not the case with PHP.
Similarly, Python has opcode cache built-in (.pyc files). Also, I am not sure about gunicorn but others do keep app files in-memory.
Servlet does the same about keeps apps in memory. You get the idea.
Frameworks definitely have an impact but it's very hard for one person to know the right configuration for every language. You had done some good work there, but it will take a lot of contributions before the benchmark suite becomes fair for every language/framework.
We're hoping to post a follow up in about a week with revised numbers based on the pull requests and any other suggested tweaks we have time to factor in. We're eager to see the outcome of the various changes.
cake 18.9% (312 vs 59)
compojure 12.1% (108588 vs 13135)
django 16.8% (6879 vs 1156)
express 16.9% (42867 vs 7258)
gemini 12.5% (202727 vs 25264)
go 13.3% (100948 vs 13472)
grails 7.1% (28995 vs 2045)
netty 18% (203970 vs 36717)
nodejs 15.6% (67491 vs 10541)
php 11.6% (43397 vs 5054)
play 20.6% (25164 vs 5181)
rack-jruby 15.6% (27874 vs 4336)
rack-ruby 22.7% (9513 vs 2164)
rails-jruby 22.7% (3841 vs 871)
rails-ruby 20.7% (3324 vs 687)
servlet 13.4% (213322 vs 28745)
sinatra-jruby 21.2% (3261 vs 692)
sinatra-ruby 22.2% (6619 vs 1469)
spring 7.1% (54679 vs 3874)
tapestry 5.2% (75002 vs 3901)
vertx 22.3% (125711 vs 28012)
webgo 13.5% (51091 vs 6881)
wicket 12.7% (66441 vs 8431)
wsgi 14.8% (21139 vs 3138)
I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown.
Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2.
Disclosure: I am a colleague of the author of the article.
So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.
Do you mind breaking it down for me a bit please?
Cost in dollars or cost in hardware utilization or some other cost?
Knowing how much I'm giving up in performance in order to get the features a given framework gives me is an important consideration. Also understanding when it's worthwhile to work outside the framework on the bare platform given the speedup I'll get versus the cost I'll incur by doing so is a very important optimization decision-making tool.
See my "framework optimization index" in comments below for a rundown on all these ratios which I was able to back out from this set of benchmarks.
Either way good architectures usually optimize the high traffic or high CPU areas anyway away from a scripted language.
Thanks for the really informative post! Go seems to be a good balance as a high performance language without having to go back to my traumatic Java days.
I agree, a remarkable take-away for us was how dramatically our i7s excelled over the EC2 instances. Admittedly, those were EC2 Large and not Extra Large instances.
A previous draft of this blog entry had a long-winded analysis of hosting costs--discussing the balance between ease and peace-of-mind provided by something like AWS versus the raw performance of owned hardware--but we elected to remove that since it wasn't really the point of the exercise.
As a rule I like to divide this world into "Featureless" and "Featurefull" products.
When you use Rails, you're aiming to pile up features. You want to react to Product managers, to users, you want to work fast and satisfy the needs of customers - or else you won't have anyone to build to.
In this reality, the fact that you're doing 20req/s is OK. In fact, I'm betting that even when you take Go or Node.js - pile up all of the infrastructure and features that exist in Rails, and pile up a ton of your code - buggy and not buggy - you'll get around the same kind of satisfaction index from users.
This is because your product can be perceived as slow even though your servers are blazingly fast.
On the other side of the spectrum there are "Featureless" products. These are infrastructural products. A logging service. An analytics service. A full-text search. A classification and recommendation engine.
These you don't want to build in Rails. I'm sure you haven't even considered it. These you want to build with one of the top-notch libraries that this survey indicate.
We're actually very interested in how the large/newer instances perform.
EDIT: also, mod_perl is generally best avoided these days; it's old, not very pleasant to work with, and ties you to Apache. Writing an app with Dancer / Mojolicious etc means you can deploy in various different ways with ease.
Thanks for the correction though, but that still doesn't answer my original question: how does Plack compare with mod_perl?
These are all abstraction layers (for each language) which then can be run seamlessly on top of CGI, SCGI, mod_(perl|python|ruby), etc.
- http://plackperl.org
- http://en.wikipedia.org/wiki/Plack_(software)
- http://en.wikipedia.org/wiki/PSGI
- http://en.wikipedia.org/wiki/Wsgi
- http://en.wikipedia.org/wiki/Rack_(web_server_interface)
Dancer use PSGI (not CGI) and most modern Perl webapp frameworks (Mojo & Catalyst) use PSGI/Plack (http://plackperl.org/). The PSGI model is very similar to wsgi which allows for different types of persistent servers.
Dancer's deployment POD is good reference on options: https://metacpan.org/module/Dancer::Deployment
Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!!
Maybe Node.js critics should start appreciating it after all..
Disclaimer: I am a colleague of the author of the linked article.
First-- what you're really testing here is the Jackson library. A majority of the cycles used in your application are being burned in that toJson call of an array of objects. This isn't a fair test compared to the servlet implementation because you're calling Jackson against a map in the Play example, versus against a simple String in the servlet example.
Second-- you are running database calls serially, and those are blocking. Considering that you're using the more-or-less default Play/Akka configuration, there are only enough threads as you have available processor cores. I would start by increasing the parallelism-factor and parallelism-max to be higher, so you'll have more available threads. More importantly though, the database access should be wrapped in a Future, and you should be returning asynchronously. This should speed up the application by a huge amount.
(Note: I'm not necessarily saying the Jerkson wrapper is the culprit. Could be Play's routing framework, or something else.)
Silex/Symfony2
Yii
Zend
Kohana
Fuel
Laravel
seriously, in my rough helloworld and sqlite value increment by 1 benchmark, Bjoern+wsgi app runs 2x as fast than nodejs.
(The big four in my book are: ASP.Net MVC, Rails, Django and CakePHP)
We did briefly test ASP.Net on Mono (see another comment in this thread) but didn't include it since we didn't believe that qualifies as a "production" grade ASP.Net MVC deployment.
It pains me to see charts and reporting done like this while leaving out my favorite framework.
Drop me a line if you need a hand with either :)
If you seriously thought that the difference between a netty and django would be 4 times then you simply don't understand what these frameworks do in the first place.
I would have guessed something like less than 100 times slower and that would have been still fine since the cost of all kinds of latencies in the system usually far outweigh the speed of the framework itself.
I am completely stunned by the performance cost of using ORM/AR, and will be using this to shame our team lead into giving it up and going for raw queries.
We plan a subsequent test, time permitting, that enables caching.
I typically use ORM for about 95% of a project, falling back to a few explicitly native SQL calls when performance can be shown to be a bottleneck in those locations.
By default, PDO does not do true prepare()s, it just does string interpolation. You need to pass this parameter with the options:
And then you'll actually be using MySQL prepared statements. You'll see a noticeable performance improvement for large amounts of queries.php - https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
php-raw - https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
For "php", they used an ORM. For php-raw they used the stdlib (pdo).