Benchmarks aside, I'm really impressed with the whole One ASP.NET thing. Seeing the whole MVC with web api and SignalR all now coming together now is making me finally feel excited about developing on the MS stack again.
I implemented the threadpool tuning originally first for the ASP.NET tests (https://github.com/TechEmpower/FrameworkBenchmarks/issues/39...) because it improved performance 5-10%. If you've seen data showing that it actually hurts performance, please open an issue or send a pull request (especially for the additional ~60% performance gain). Thanks.
I’d like to see ASP.net MVC natively on Windows, as well as ServiceStack. I think they’d compete. Mono is just not a thing for those of us in the .Net world, frankly.
Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++.
Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.
"As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD
Two Amazon EC2 m1.large instances for the EC2 tests
Switched gigabit Ethernet"
So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
There are still practical limits, and sometimes you have to work within those. I've got a project on a 16G server - the data center doesn't have anything bigger. Moving everything to a custom server or different data center to get to 24G or 32G is... a lot of work. Some time spent finding memory reductions is still worth it.
And what if we went to 32G or 64G, but still needed more? "RAM is cheap" doesn't scale. The hardware needed to host a 128G system (or multiple 16/32G systems) isn't cheap.
Either you're wrong about the cost or we have a different definition of "cheap". The sweet spot for RAM pricing right now seems to be 16GB DIMMs, which cost $150 to $250. I can go to dell.com and configure a R320 with a Xeon CPU and 96GB of RAM for $2,500. That's less than a high-end Macbook Pro! A R420 with two Xeon CPUs and 192GB of RAM costs $4,300.
I recently upgraded two Dell Poweredge 2950s from 16 to 32GB of RAM. The cost per server was $300. If I'd needed to, I could have upgraded them to 64GB. That's on a five year old server that sells refurbished for $800.
I'm a fan of Go and an even bigger fan of Lua, and I'm not sure I agree with your summary. Go and OpenResty can push 300,000 "Hello World" plaintext responses a second, and some Java stuff can do x2 that, so JVM frameworks are strictly better? And something called cpoll_cppsp trumps a sane language choice? Don't be cute. :)
It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde.
The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for.
A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.
I think the idea of a "meaningless benchmark" has become a dogma. Can you possibly know all the complexities that an app will contain when benchmarking a framework? No, but that doesn't make baseline benchmarks useless. Part of the reason for including the most basic (CRUD-ish) operations as separate benchmarks in their tests is so that you can make comparisons between basic "Hello, World" operations and more complex operations, which gives you a bit more insight into the overhead of each framework.
I'll take that any day, over guessing which framework is fastest.
Actually, it's other way around. In 'JSON encoding' tornado and flask-pypy are 4x slower than compojure, for example, but in 'Multiple queries' they are 14x slower.
Just for the record, my comment didn't intend to imply that JVM frameworks are strictly better in all cases, and I don't believe that at all. In fact, I'd argue that for the majority of applications, choosing any modern framework purely for benchmark performance is very foolish since all of them perform more than adequately.
Rails, for example, despite always having been "slow," performs far more than adequately for the vast majority of web applications, scaling patterns are well established, and it's obviously hugely productive for many, many developers and companies.
Well, it really depends on the application type. We are a Java and Ruby on Rails software house (http://codedose.com), we also have experience working for investment banks, and for some cases Java with Spring or Play is the best way to go, especially if you think about apps with complex back-end.
We have recently released an online market for physical gold trading capable of handling 10k+ concurrent users with horizontally scalable architecture and complex trading engine/accounting logic, and it's all Java. It has to be fast and there is not enough static or almost-static content to make caching effective.
Actually I was quite surprised that multiple queries benchmark showed more than 1 entry for PHP in top 10. take-away: if you don't do much sophisticated data processing then PHP is ok choice. gets ready to run
That's because php is a really thin wrapper around the underlying C libraries. The thing that makes it ugly is what makes it fast. Php is still interpreted and not jit'ed, even with the opcode cache used in this benchmark, so it loses out in anything that exercises the php code side of the equation, which is why the big php frameworks (symfony, cake and laravel) are 50 times slower than raw php. I wonder how facebook's hhvm engine would stack up, as it is a jit'ing php engine and supposedly an order of magnitude faster.
As a related aside, we have an intention to eventually capture some additional statistics about the implementations such as source lines of code, total number of commits, and possibly lines of code of libraries (where available).
These are just additional data points to use as the reader sees fit, but they may provide some insights. For example: (a) developer efficiency, assuming you are comfortable using sloc as a proxy for developer efficiency, which is admittedly a hotly debated matter; (b) commits are a proxy for the level of attention the particular test implementation has received in our project, since a test with only 1 commit may be unrefined and in need of tuning and a test with dozens of commits may be considered highly volatile; (c) where applicable, the performance impact of additional code, perhaps even as granular as a calculated average per-line cost in rps. The last point might be particularly illuminating for languages such as PHP.
Doesn't look to be "an order of magnitude faster" - 200% faster in some cases - certainly nice numbers, but not a massive game changer for many (yet?).
This. In my last project I tried to use scalatra with slick, and while I really liked scalatra, slick made me go nuts. I had to jump over so many hoops that it was just a pain.
I'm currently working on a new project (still research phase) where I was pondering going with Spray and a lightweight Postgresql wrapper because I primarily need to read data from a database, do some transformations on it, and write it out as JSON as fast as possible. I had it working in Spray, but I had issues with server crashes and the speed wasn't as I'd have expected. I fiddled with it for one day, and then, out of frustration, decided to give openresty a try. I've never written much Lua in my life, but after only a couple of hours, I had it working, and it was far, far faster than the Spray implementation. I did some research there, and it seems that the database stuff took a whole lot longer in Scala/Spray than in Lua. Now, of course, I loose type safety, so there may be hidden issues in there, but since I'm really just doing simple data transformations, I think I'm fine with lua / openresty.
As a first verdict, I really, really like what I've seen of Openresty so far.
With all the nice alternative frameworks available for the JVM these days, including some pretty micro/bare-bones alternatives, I wonder if Java will find new favour with startups or if they'll stick with Rails. The old "the Java ecosystem is too complex/over-engineered" excuse falls down with stuff like Play or Grizzly.
I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.
Which is more expensive, servers or developers? Which is more expensive, server or opportunity cost?
In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation.
In addition, the relationships between those variables likely changes over time.
I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance.
Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
Not to beat a dead horse, but ask Twitter. I expect their answer would be something like "Migrating off a non-performant solution to Java was more expensive than just doing it right in the first place."
Also, my point was that the newer jvm frameworks aren't much more complex or time-consuming than Rails so developer cost isn't such an issue. That said, not much compares to Rails for getting stuff up and running quickly.
Cost is relative to the maturity of the business. They may not have had the funds or time to do things the right way the first time, and if they had tried, they might not have ever made it into near-unlimited-budget-land.
Fair enough, back then the newer, sleeker jvm frameworks may not have existed. But now, perhaps there's no such excuse.
Maybe these options will become really compelling once Kotlin is released. It compiles to both the jvm and JS so it will hit the performance, convenience and "fun to write" sweet spots all at once...or so I hope.
I said that in jest not because it's ridiculous, but because I wish it were true. I use Mojolicious and love it.
My only complaint is that I started with Mojolicious::Lite way back in the beginning of the project, and I really needed to port to Mojolicious proper with a better source file layout quite a while back, but Mojolicious makes it too easy to throw in another lite-style route. At least the templates were easy enough to separate that I did it long ago.
This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.
Go's numbers are quite impressive considering how young the language is. That combined with the concurrency primitives and ease of deployment makes Go very appealing.
Updating the versions in the Environment details became a game a whack-a-mole, so I removed that information from the Environment section for now. Maybe some day I'll come up with a nice way to automatically pull it from the GitHub repo or meta-data about each test. For now, you can find some details in the Rack script [1]: ruby-2.0.0-p0.
We'd be happy to receive a PR to update these versions if anyone is interested in crafting one.
The errors column is a sum of non-200 HTTP responses and socket connect, read, and write errors. In practice, it tends to be predominantly 500-series HTTP responses, meaning the web server acknowledged the request but was too busy to assign it to a worker process or thread.
This can cause the latency of some frameworks to appear artificially low because the 500 responses tend to come back very quickly once they start.
Incidentally, if you hover over the error values, you'll get a success rate percentage.
That is correct. We have received many contributions from the community (I would call them subject matter experts, but they seem very humble to me, so I'm not sure what they'd call themselves) to improve the C# tests. It looks as if they have helped quite a bit, as I pointed out in the blog entry that accompanies Round 7.
I noticed the same, and I suspect this has more to do with the implementation of the Mongo driver it's using than with Vert.x itself. Vert.x is a very high-performance server.
I thought so too, so I looked at the source. It's communicating with a MongoDB persistor over the event bus rather than querying the database directly. So the poor performance is because there's an extra layer of communication happening, which is kind of misleading for a benchmark like this.
Vert.x was in our original batch of tests (round 1). At the time, its documentation showed that using the persistor and event bus was the standard intended mode of communicating with a database. If someone can point us to documentation showing that is no longer how you're supposed to use a database with Vert.x, we'd be happy to accept a pull request that changes the test. Otherwise, I think it's valid as is, and not misleading.
There is a category of web frameworks that imply specific web application architectures which make them somewhat difficult to shoehorn to our test types. Meteor is one of those, as are JSF and GWT. We'd welcome an implementation that attempts to match Meteor to these test types, but acknowledge that might be awkward.
See my reply elsewhere in this thread [1] concerning a new less-forgiving approach we've adopted to glitches/errors. It's not the most polite way to run the project, and I apologize for any oversights and errors, but it's just a necessity since we don't have expertise in everything.
With Grails specifically, a very late PR arrived that unfortunately broke the test implementation for us, so we removed it from the Round 7 test. We aim to push new rounds on a monthly cycle, so I hope that anything that is missing in one round won't be missing very long.
I have the dubious honor of the slowest framework on the test on page #1! For some reason the i7 tests perform really badly for Phreeze, whereas the EC2 tests it performs near the top of all the PHP frameworks.
I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!
Although this is Round 7, we have no delusions that there is not a lot of room for improvement (particularly in the toolset, although that's a separate matter). If there are tuning tweaks, we'd love to receive a pull request.
Now, let's look at the best case comparison: 109,882 rps vs 3.4 million iterations per second. Is cutting iteration time down to 1/3 significant (298ns vs 103ns)? Yes. Is it significant in the overall context? No.
With a 31x difference between 109,882 and 3.4 million* (and, since this is comparing to an i7, that 31x is probably closer to the ballpark of 100x), this simply isn't likely to be the place where optimization will help much. Put simply, the {"message": "Hello, world"} vs dict(message="Hello, world") cost difference is likely insignificant when compared to the cost of the rest of the request.
With that said, this sounds like a change that should be committed as part of a pull request! Why? A couple of reasons:
- It may still help the performance in a minor way (at the very least, it shouldn't hurt it)
- We prefer idiomatic code and, assuming the literal {} notation - vs dict() - is idiomatic for such a case, this would be a good change.
Your point about this is also interesting:
What you have is the slower way to contruct a dictionary
and then passing to the Python native JSON vs. a C
optimized JSON for output.
Please do consider a pull request for both. We love community contributions.
* Yes, I know these aren't directly comparable numbers, but they compare well enough in this case.
Edit: Fixed the literal {} vs dict() mixup that e12e pointed out in my comment.
Note, I'm not the one that noticed this, I just tried the difference between literal construction "{}" and using dict().
Good points on the overall (likely) impact on the benchmark(s).
> We prefer idiomatic code and, assuming dict() is idiomatic for such a case, this would be a good change.
I think you mix up two things here: dict() is slower (presumably method look up and maybe class instantiation? Just guessing here) -- and I'd say using the literal notation is in general more idiomatic:
Oops, yes. I did mix up dict vs the literal version in the comparison. Thank you for the correction.
And yes, I noticed that you weren't the one that noticed the original possible performance issue. What I do appreciate is that you actually went and tested the performance difference.
> For each request, an object mapping the key message to Hello, World! must be instantiated.
Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible.
Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.
Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.
Right. Related to this, I feel like a couple of frameworks are cheating on the plaintext test because they aren't actually testing string... serialization. They encode the "Hello, World!" string into bytes once, retain those bytes in memory, and send those bytes on each request just to avoid encoding the string again.
It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.
I want the test to require the work of creating an object or allocating the memory for the response and then use of a JSON serializer.
These tests are intended to be stand-ins for realistic application behaviors. They are not realistic applications, but they should make use of the same functionality that a real application would. I want the tests to establish realistic high-water marks for the types of operations they exercise.
Running a single query per request? The high-water mark you might expect is what you see as our single-query test. It would be very difficult to make a single-query operation in a real application more trivial than what we have.
If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back.
Bottom line: nothing beats benchmarking your own application. But the tests in this project are intended to give a first-pass filter of sorts, assuming performance is on your list of requirements.
> If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back.
A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request.
Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID.
If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run.
What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.
That wouldn't work sufficiently well; unless the benchmark is quite complex it would almost always be better to implement an in-memory custom datastore rather than use a traditional backend. Even if you require persistance, since the intentionally small dataset means you can get away quite well with raw memory dumps or even an mmap variant. You wouldn't use a full-fledged JSON serializer but a limited string concatenation.
In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app.
Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.
Custom datastores (custom at the framework level, not necessarily at the application level) have been very useful for me in the past for getting high performance.
For example, I wrote a little in-memory DB for use in a .NET framework to avoid having to deserialize an object graph per request. Instead, it navigated a byte array and plucked out just the data needed.
Another time, I wrote a compact in-memory representation of a trie, using bit-packing and a few other tricks, to get an order of magnitude reduction in memory usage, making it possible to cache a lot more of a data set.
Why go to the database and instantiate a classic object if you can avoid it?
You're going to stream a Hello, World value? Why use JSON, at all, when we can just print the resulting JSON text! Benchmarks should be defined, that's all.
> Why use JSON, at all, when we can just print the resulting JSON text!
Indeed. But if you want to actually force the server to encode some JSON, then make the reply include portions of the request. Then you don't have to have artificial rules about how it is implemented to get interesting/meaningful results: https://news.ycombinator.com/item?id=6650499
No, you still need to, because you can otherwise do a trivial string concat in most cases, even though in reality that would be most unwise for security, robustness, and maintainability reasons.
Since you don't know the inputs ahead of time, the benchmark would be incorrect if it could not handle the corner cases like special characters that need to be escaped in JSON strings.
If you wanted you could also make the request specify the key at which a value must be written.
Be creative. :) But don't think that apps always have to be written the way they are now to be "legitimate."
Otherwise you won't be able to compare the results.
For example, the Benchmarks Game [1] specifies how the algorithms in those benchmarks should work. For some of the benchmarks, there are faster algorithms. However, it's not about comparing algorithms, it's about comparing language implementations.
This has puzzled/surprised me over multiple benchmarks. I had assumed that node.js would be faster at pretty much everything than PHP (especially for concurrency).
Time and time again though, these benchmarks show PHP outperforming node.js. I'm sure node.js will have lower RAM usage than PHP but as I'm more interested in reqs/s, I'm going to reconsider using node.
PHP is just a thin layer on top of C. Of course it will perform decently. PHP frameworks are usually designed in a dumb way and thus perform badly. Usually they load all or most of their files into memory with every request, even though only couple files would be enough.
That's node vs bare PHP, which has limited utility. Performance drops like a rock if you use any kind of framework - see cake, CI, symfony all at the very bottom.
PHP benchmarks are extremelly outdated and not even using PHP 5.5 with OPCache (that is now native). Python and Ruby implementation choices are also really weird.
Not to be an ass, I don't know much about their approaches and Php (APC in particular)... But wouldn't enabling caching sort of defeat a portion of the test?
Wouldn't it just be better to compare caches/caching mechanisms?
While other sensible language interpreters have a feature like byte-code caching built-in in them, PHP for quite a while did not, because someone was monetizing a proprietary cache add-on named ZendOptimizerPlus!
They released it as part of the language in 5.5 because alternative APC cache got just good enough and everyone was installing it by default.
So, to answer your question: yes, APC should be used on any PHP benchmark if performance is in question.
I was surprised not to see Revel included in the round 7 results. You can filter results by it but nothing is shown.
This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks.
Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?
I'm curious as to why Django was paired with gUnicorn and only gUnicorn for its tests. I would have loved to see it benchmarked with uWSGI, gEvent or something with a little more gusto.
Probably because the testers aren't experts in all frameworks and languages and deployment options, and don't have infinite time and resources to try all combinations of options to test stuff.
Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
Looking through all your comments, I see one vein of similarity between a majority of your words: you're especially, unnecessarily, and consistently impolite. Please, a little internet civility. My question was merely curiosity, and I'm (gasp) well aware that the testers at techempower don't possess infinite time or resources. Your reply was useless and in no way appropriate. Have a bit of respect for yourself.
You wanted to know why they didn't test whatever you thought was the optimal setup for your use case, and the answer should've been obvious - there's dozens of frameworks across multiple languages and OS platforms. If you look around on this thread and those for previous iterations of the benchmarks, a consistent response from the benchmarkers is that they didn't test some specific scenario because they lack the expertise to do so.
The tests are on github. Go write one that suits you and send a pull request, or, god forbid, do it yourself and make a post about it on HN and get yourself some karma. But as it stands, your post was lazy and trivial. I've said similarly dumb things in the past, gotten downvoted or called out, and we all moved on.
I'm sorry your fee-fees got hurt because you said something dumb, and I pointed it out in terms that failed to indicate respect for you. I upvoted your response as a gesture of love and understanding.
Looking through all your comments, I see one vein of similarity between a majority of your words: you're especially, unnecessarily, and consistently impolite. Please, a little internet civility. My question was merely curiosity, and I'm (gasp) well aware that the testers at techempower don't possess infinite time or resources. Your reply was useless and in no way appropriate. Have a bit of respect for yourself.
In the tests that use a database, this distinguishes using an ORM vs. direct SQL queries. From the guide on the filters panel:
We classify object-relational mappers as follows:
Full, meaning an ORM that provides wide functionality, possibly including a query language.
Micro, meaning a less comprehensive abstraction of the relational model.
Raw, meaning no ORM is used at all; the platform's raw database connectivity is used.
187 comments
[ 3.0 ms ] story [ 218 ms ] threadSlightly sad to see the Mono performance is still abysmal.
Edit: link to the blog. It was submitted earlier, but apparently got killed:
http://www.techempower.com/blog/2013/10/31/framework-benchma...
I'm impressed by the Java options. Definitely something to consider if absolute performance is the requirement.
[0] https://github.com/TechEmpower/FrameworkBenchmarks/pull/272#... [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/31... [2] https://github.com/TechEmpower/FrameworkBenchmarks/pull/339#...
In my own testing there was another ~60% performance to gain with better tweaks.
[1] https://github.com/damianh/FrameworkBenchmarks/blob/nowin/no...
Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.
"As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet"
So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
Raw performance is nothing without memory usage information.
And what if we went to 32G or 64G, but still needed more? "RAM is cheap" doesn't scale. The hardware needed to host a 128G system (or multiple 16/32G systems) isn't cheap.
I recently upgraded two Dell Poweredge 2950s from 16 to 32GB of RAM. The cost per server was $300. If I'd needed to, I could have upgraded them to 64GB. That's on a five year old server that sells refurbished for $800.
It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde.
The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for.
A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.
I'll take that any day, over guessing which framework is fastest.
Rails, for example, despite always having been "slow," performs far more than adequately for the vast majority of web applications, scaling patterns are well established, and it's obviously hugely productive for many, many developers and companies.
Facebook and Twitter seem to be doing alright. Care to elaborate?
We have recently released an online market for physical gold trading capable of handling 10k+ concurrent users with horizontally scalable architecture and complex trading engine/accounting logic, and it's all Java. It has to be fast and there is not enough static or almost-static content to make caching effective.
As a related aside, we have an intention to eventually capture some additional statistics about the implementations such as source lines of code, total number of commits, and possibly lines of code of libraries (where available).
These are just additional data points to use as the reader sees fit, but they may provide some insights. For example: (a) developer efficiency, assuming you are comfortable using sloc as a proxy for developer efficiency, which is admittedly a hotly debated matter; (b) commits are a proxy for the level of attention the particular test implementation has received in our project, since a test with only 1 commit may be unrefined and in need of tuning and a test with dozens of commits may be considered highly volatile; (c) where applicable, the performance impact of additional code, perhaps even as granular as a calculated average per-line cost in rps. The last point might be particularly illuminating for languages such as PHP.
Doesn't look to be "an order of magnitude faster" - 200% faster in some cases - certainly nice numbers, but not a massive game changer for many (yet?).
And Java/Scala are disqualified pretty quickly.
I'm currently working on a new project (still research phase) where I was pondering going with Spray and a lightweight Postgresql wrapper because I primarily need to read data from a database, do some transformations on it, and write it out as JSON as fast as possible. I had it working in Spray, but I had issues with server crashes and the speed wasn't as I'd have expected. I fiddled with it for one day, and then, out of frustration, decided to give openresty a try. I've never written much Lua in my life, but after only a couple of hours, I had it working, and it was far, far faster than the Spray implementation. I did some research there, and it seems that the database stuff took a whole lot longer in Scala/Spray than in Lua. Now, of course, I loose type safety, so there may be hidden issues in there, but since I'm really just doing simple data transformations, I think I'm fine with lua / openresty.
As a first verdict, I really, really like what I've seen of Openresty so far.
Sorry, round 7 isn't helping much. Wait for round 8.
I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.
https://grizzly.java.net/quickstart.html
In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation.
In addition, the relationships between those variables likely changes over time.
I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance.
Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
Also, my point was that the newer jvm frameworks aren't much more complex or time-consuming than Rails so developer cost isn't such an issue. That said, not much compares to Rails for getting stuff up and running quickly.
Maybe these options will become really compelling once Kotlin is released. It compiles to both the jvm and JS so it will hit the performance, convenience and "fun to write" sweet spots all at once...or so I hope.
My only complaint is that I started with Mojolicious::Lite way back in the beginning of the project, and I really needed to port to Mojolicious proper with a better source file layout quite a while back, but Mojolicious makes it too easy to throw in another lite-style route. At least the templates were easy enough to separate that I did it long ago.
See https://github.com/TechEmpower/FrameworkBenchmarks/issues/49...
This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.
I'm curious to see what happens to performance when v.12 is released.
Round 6: https://news.ycombinator.com/item?id=5979766
Round 5: https://news.ycombinator.com/item?id=5727012
Round 4: https://news.ycombinator.com/item?id=5644880
Round 3: https://news.ycombinator.com/item?id=5573532
Round 2: https://news.ycombinator.com/item?id=5498869
Round 1: https://news.ycombinator.com/item?id=5454775
Lots of solid discussion.
We'd be happy to receive a PR to update these versions if anyone is interested in crafting one.
The errors column is a sum of non-200 HTTP responses and socket connect, read, and write errors. In practice, it tends to be predominantly 500-series HTTP responses, meaning the web server acknowledged the request but was too busy to assign it to a worker process or thread.
This can cause the latency of some frameworks to appear artificially low because the 500 responses tend to come back very quickly once they start.
Incidentally, if you hover over the error values, you'll get a success rate percentage.
[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
Rails is disappointing. Slow to be expected for interpreted language framework but that is really sh slow.
I wonder of they setup PHP with APC cache (opcode cache) which is basic and easy setup to speed up PHP.
"Have you enabled APC for the PHP tests?" Yes, the PHP tests run with APC and PHP-FPM on nginx.
https://github.com/TechEmpower/FrameworkBenchmarks
With Grails specifically, a very late PR arrived that unfortunately broke the test implementation for us, so we removed it from the Round 7 test. We aim to push new rounds on a monthly cycle, so I hope that anything that is missing in one round won't be missing very long.
[1] https://news.ycombinator.com/item?id=6650136
Because people just aren't going to read the notes. They'll skip to the results.
I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!
Yes, if you have time, please do help us resolve the issue (and I am guessing it is some configuration issue) for Round 8.
wsgi:
tornado: What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output.At 103ns per iteration, you can do 9.7 million iterations per second. At 298ns per iteration, you can do 3.4 million iterations per second.
Let's look at the wsgi numbers for json serialization:
Now, let's look at the best case comparison: 109,882 rps vs 3.4 million iterations per second. Is cutting iteration time down to 1/3 significant (298ns vs 103ns)? Yes. Is it significant in the overall context? No.With a 31x difference between 109,882 and 3.4 million* (and, since this is comparing to an i7, that 31x is probably closer to the ballpark of 100x), this simply isn't likely to be the place where optimization will help much. Put simply, the {"message": "Hello, world"} vs dict(message="Hello, world") cost difference is likely insignificant when compared to the cost of the rest of the request.
With that said, this sounds like a change that should be committed as part of a pull request! Why? A couple of reasons:
- It may still help the performance in a minor way (at the very least, it shouldn't hurt it)
- We prefer idiomatic code and, assuming the literal {} notation - vs dict() - is idiomatic for such a case, this would be a good change.
Your point about this is also interesting:
Please do consider a pull request for both. We love community contributions.* Yes, I know these aren't directly comparable numbers, but they compare well enough in this case.
Edit: Fixed the literal {} vs dict() mixup that e12e pointed out in my comment.
Good points on the overall (likely) impact on the benchmark(s).
> We prefer idiomatic code and, assuming dict() is idiomatic for such a case, this would be a good change.
I think you mix up two things here: dict() is slower (presumably method look up and maybe class instantiation? Just guessing here) -- and I'd say using the literal notation is in general more idiomatic:
http://docs.python.org/2/tutorial/datastructures.html#dictio...
And yes, I noticed that you weren't the one that noticed the original possible performance issue. What I do appreciate is that you actually went and tested the performance difference.
> For each request, an object mapping the key message to Hello, World! must be instantiated.
Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible.
Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.
I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
> I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
It's such a small, silly optimization. It doesn't bother me because it's not going to affect the overall results much, but I do feel it's against the spirit of the test. That kind of thing would be prevented if we made the test echo a request parameter, as was suggested elsewhere in the comments.
These tests are intended to be stand-ins for realistic application behaviors. They are not realistic applications, but they should make use of the same functionality that a real application would. I want the tests to establish realistic high-water marks for the types of operations they exercise.
Running a single query per request? The high-water mark you might expect is what you see as our single-query test. It would be very difficult to make a single-query operation in a real application more trivial than what we have.
If I treated all tests as a black box, none of the database tests would actually query the database; the fortunes test would not add an ephemeral object and re-sort on every request; and so on. Everything would just send static responses back.
Bottom line: nothing beats benchmarking your own application. But the tests in this project are intended to give a first-pass filter of sorts, assuming performance is on your list of requirements.
A better way to avoid this would be to make the output depend on the input. Require that the reply include some JSON that contains portions of the request.
Likewise for databases, require that the reply contain contents of a previous request. If you want to, make the request contain an ID and make the reply return the data keyed under that ID.
If you want to require that the data is persisted durably, require that the benchmark continue to function even if all processes are killed and restarted in the middle of a run.
What really matters about a system are its inputs and outputs. If an innovative system comes along that works differently inside, it shouldn't be disqualified from a benchmark just for being innovative.
In any case, the way I read it, the requirement doesn't mean you have to make anything particularly heavy, just that you need to use a representation that's general enough to be usable in other scenarios without rewriting the entire app.
Of course, a little bit of extra encouragement in the form of a benchmark that's slightly harder to game wouldn't be bad.
For example, I wrote a little in-memory DB for use in a .NET framework to avoid having to deserialize an object graph per request. Instead, it navigated a byte array and plucked out just the data needed.
Another time, I wrote a compact in-memory representation of a trie, using bit-packing and a few other tricks, to get an order of magnitude reduction in memory usage, making it possible to cache a lot more of a data set.
Why go to the database and instantiate a classic object if you can avoid it?
Indeed. But if you want to actually force the server to encode some JSON, then make the reply include portions of the request. Then you don't have to have artificial rules about how it is implemented to get interesting/meaningful results: https://news.ycombinator.com/item?id=6650499
If you wanted you could also make the request specify the key at which a value must be written.
Be creative. :) But don't think that apps always have to be written the way they are now to be "legitimate."
Otherwise you won't be able to compare the results.
For example, the Benchmarks Game [1] specifies how the algorithms in those benchmarks should work. For some of the benchmarks, there are faster algorithms. However, it's not about comparing algorithms, it's about comparing language implementations.
[1] http://benchmarksgame.alioth.debian.org/
I'd really like to see the chart showing RAM usage for these tests.
Time and time again though, these benchmarks show PHP outperforming node.js. I'm sure node.js will have lower RAM usage than PHP but as I'm more interested in reqs/s, I'm going to reconsider using node.
- Laravel Version 3.2.14
- PHP Version 5.4.13 with FPM and APC
Don't waste your time with this crap.
Wouldn't it just be better to compare caches/caching mechanisms?
They released it as part of the language in 5.5 because alternative APC cache got just good enough and everyone was installing it by default.
So, to answer your question: yes, APC should be used on any PHP benchmark if performance is in question.
This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks.
Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?
Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
You wanted to know why they didn't test whatever you thought was the optimal setup for your use case, and the answer should've been obvious - there's dozens of frameworks across multiple languages and OS platforms. If you look around on this thread and those for previous iterations of the benchmarks, a consistent response from the benchmarkers is that they didn't test some specific scenario because they lack the expertise to do so.
The tests are on github. Go write one that suits you and send a pull request, or, god forbid, do it yourself and make a post about it on HN and get yourself some karma. But as it stands, your post was lazy and trivial. I've said similarly dumb things in the past, gotten downvoted or called out, and we all moved on.
I'm sorry your fee-fees got hurt because you said something dumb, and I pointed it out in terms that failed to indicate respect for you. I upvoted your response as a gesture of love and understanding.
We classify object-relational mappers as follows: Full, meaning an ORM that provides wide functionality, possibly including a query language. Micro, meaning a less comprehensive abstraction of the relational model. Raw, meaning no ORM is used at all; the platform's raw database connectivity is used.