70 comments

[ 1.9 ms ] story [ 172 ms ] thread
Interesting that drogon is relatively new (first release a year ago; new to Round 19) and is utterly destroying the benchmarks. https://github.com/an-tao/drogon

Show HN: https://news.ycombinator.com/item?id=19427332

Well, it's hard for high level scripting languages to compete with C++ performance-wise...

I would have liked to see some other C++ ones on there, like Crow[0]

[0] https://github.com/ipkn/crow

I heard some old guard person at Mozilla talk about these benchmarks last year when Rust's ACTIX was on top, something like "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest", at the time I took it to mean ACTIX being the fastest has more to do with it being new then it being written in rust.

Given a newcomer is at the top again that seems to ring true? Does seem odd older frameworks wouldn't keep up with that game, but perhaps once they get the publicity and user base they can worry more about features than having the highest benchmark?

People ask for more features over time. I'd expect older frameworks to be more featureful, but slower.
It's worth noting that actix-web is at the top by doing some insane tricks, which you wouldn't normally do in writing a web application.

e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well.

Nobody is going to bother doing that.

I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".

But most other projects on there do that as well. I regard tech empower as the maximum performance I can get with a given framework if need be. With actin I can add Askama, with Vapor, there's no way to make it faster.

As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns

Regarding your first sentence... I think we have the same conclusion. :)

Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.

I don’t think recompiling for template changes is as unreasonable as you depict. It’s a standard technique in various languages, though I acknowledge that they’re mostly not languages that compile to native code, but rather an IL (e.g. ASP.NET with Razor) where the compilation time is somewhat lower.

For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor).

There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.

I didn't say that recompiling template changes is bad - e.g, you can accomplish the same result with much faster turnaround time using Tera, and for most projects the speed is in no way a bottleneck.
While I agree with your criticism of “cheating” and unnatural tricks that you are not likely to use in production (but it's more a fundamental criticism about this kind of benchmark games than a specific criticism of Actix), I don't really share you view on Askama: unless your templates change a lot (i.e. during initial development) or if you need to be able to change then dynamically in production without downtime, compiled templates like Askama are IMHO the way to go when going to production. You can and should use a interpreted one during initial development, and then switch when you're ready to go live.

An actix-web + Askama user.

Using one during development and one during production is indeed a valid use case, but most people simply don't get anywhere close to that level of need.

Let me be very clear: I like Askama and have submitted pull requests to the project in the past. For a new project where I need templates, I value the speed of reflected changes over compiling more stuff into the binary.

There are also certain classes of problems with Askama that you can avoid with something like Tera - e.g, with Askama, if your global layout relies on a value passed in to the rendering context, you'll need to shove that value on every Template struct.

Furthermore, most people will hit bottlenecks in other areas of their application long before templating becomes the issue (i.e, this was the same in Django-land too - templating via cached compiled Django or Jinja2 templates is almost never an issue). Askama is overkill until you need it.

My recommendation, having built projects with both, is thus: build with Tera initially, and swap out hot paths to Askama down the road as they both use the same syntax (roughly) so it's not too difficult to convert one to the other.

In a perfect world the projects would be merged, and you can use the Tera aspect in dev and get Askama in production for free... but that's easier said than done.

> "whatever is the latest framework can take advantage of all the latest methods and libraries to become the fastest"

I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results).

I fail to understand how such improvements have something to do with “all the latest methods and libraries”?

What's impressing to me is that in the most "IO-bottlenecked" benchmarks: 'Multiple queries' and 'Data updates' (on Cloud) Drogon is smoking the next best frameworks.

What's up with that?

Postgres libs for C++ are probably very good at async.
One of Drogon’s project goals is great performance. That’s also one of the reasons why it’s written in C++.
I was surprised to see the performance of php frameworks. Has there been something special happening with this language ? It looks completely on top of all the other dynamic languages on most benchmarks, which i really don’t understand.
PHP 7.0 (which was released in Dec 2015) featured a new core with dramatic improvement in performance over PHP 5.X.

PHP 8 is due for release in Dec 2020 and will feature a JIT compiler which is expected to improve performance even further.

Regardless of what people think of PHP, it's nice to see the PHP community tackle performace and resource usage as a major priority. A contrast to parts of the development community who seem to care very little for performance and resource usage.

It's important to note that all the PHP results that are high up on the lists are using Workerman[1] with the event module, Swoole[2] or ngx_php[3]. They don't use the traditional PHP runtime model, which boots up and tears down the world for every request. Global variables, which are no problem in "normal" PHP code, are going to cause serious trouble in these environments. They are essentially gluing a very fast async IO framework to a fairly minimal amount of PHP application code.

The vast, vast majority of PHP code you will find in the wild is running a more traditional stack, and those results are much lower down the list.

The results are still impressive though. PHP is very good at gluing fast C code together without slowing it down much. Especially Workerman is impressive, since it appears to still run a significant amount of PHP code.

[1]: https://github.com/walkor/Workerman [2]: https://www.swoole.co.uk/ [3]: https://github.com/rryqszq4/ngx_php7

Part of that's also that php 7.4 is much faster, so of course that alone would improve it more, I wonder what JIT will do when 8.0 launches in December.

I use swoole w/ laravel, still trying to figure out best ways to milk it for more performance, and also grok the way it handles async and passes variables around using swoole_Table.

For the moment (well, february) the improvement with PHP 8.0 JIT is minimal: https://www.phoronix.com/scan.php?page=news_item&px=PHP-8.0-...

But usually a JIT is made in a way that you can then move forward with your optimizations in small tested steps. So it makes sense for JITv1 to have a very similar performance as the non-JITed version. It probably generates highly familiar code. First make it do the same, then optimize.

Because those framework are not written in PHP but C++/C library.
Workerman is made with pure PHP on top of the libevent, without using a web server. It's like nodejs but in PHP.
PHP is fast. But the stack isn't: fast-cgi, no db pool connection, ... It's really good for a shared virtual hosting.

If you change the stack with event driven, a pool and persistent application, you get that fast results.

Rocket (Rust) is the framework on my new project, and the benchmarks clearly show that speed improvements would be very welcome. If anyone here wants to work on this area, please contact me and I'll be glad to boost the Rocket numbers together.
How does Rocket compare with actix on other dimensions? (i.e., what made you decide to pick it for your project?)
Broadly, Rocket and Actix are both great in my opinion.

I use actix-web on another project. IMHO Actix is conceptually sophisticated in a good way, has great async capabilties, great speed, and runs on Rust stable which is necessary for that project. However, actix-web had a continuity problem when the leader stepped away - see HN if you want to know more about this.

So we decided to try Rocket for this new project. This project involves more junior developers, has no special speed needs, and is fine on Rust nightly for now. The Rocket guides are excellent, especially for onboarding. The concepts feel simpler, which is fine for this project. We're eager for Rocket to launch its async version and its Rust stable version.

Ultimately the bulk of the code is similar, because both projects are web apps that use Diesel, Serde, reqwest, and other typical crates. We're looking to the future to extract some of the code into framework-independent crates, such as our API structs, business logic, and the like.

Thanks! I'm very curious about the Rust web space even though it seems to still be in a formative stage. It sounds like you guys are really on the cutting edge
I'm just gonna say here that Rocket is never going to run in stable.

I write this specifically because I'm never right and I hope that my writing this publicly will lead to Rocket's immediate release on Rust stable just to prove me wrong.

And here's a cheer for those doing the work to make it async.

I actually think something was merged on Rust recently that allows Rocket to be on stable now. I don't have a link but I read it. I think 0.5 is supposed to be async, still on nightly, and after that they want to switch to stable.
I actually benchmarked it yesterday and I would like to point out that the async branch of Rocket will get you improved performance, as the current stable implementation v0.4.4 appears to be synchronous. I think Rocket v0.5 will change that.
What was the result of your benchmark – just a ballpark number would be great on what to expect from the async version.
Absolute numbers vary with each machine, to the point they’re irrelevant in isolation.

However, in terms of relative performance the async version was 10x to 20x better. Bear in mind it was a simple hello world benchmark. I also disabled keep-alive. Reusing a connection gets you even better performance.

Rocket is a well designed framework, so I’d expect it to perform similarly to other rust frameworks once the async version hits master.

The addition of the "realistic" vs "synthetic" benchmarks is very welcome, with "realistic" being the default. Hopefully this could help get things to a better state.

Still questionable as to what is allowed in "realistic" though. Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic".

Perhaps there needs to be a "marketed" category, for use as the docs describe. For aspnet core, this would include using entity framework.

Not much point in splitting the categories if it isn't a uniformly split thing.

It'd be nice to see benchmarks where: • Everything was implemented as documented in the framework's tutorials. • Realistic things were included end-to-end (payload validation, csrf (if applicable), etc).

Still seems like a whole basket of fruit comparisons here.

The hardware physical/cloud toggle is also interesting.
> Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic".

Why not? I don't think using raw SQL instead of an ORM is particularly uncommon.

The "platform" benchmarks (aspcore-rhtx-pg, aspcore-ado-pg) that appear to skip the ASP.NET Core framework entirely and write raw HTTP responses are definitely not realistic though. I'm going to assume that these are mislabeled because this filter is new and not all the benchmarks have been updated yet.

I'll agree that raw database queries aren't unheard of, but EF and Dapper are much more suggested (and IMO "realistic") database paths for "asp.net core"

additionally, I'm not quite sure that this is the recommended way to return database results: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

What I believe is more "realistic" is to:

• Use the framework as described: https://docs.microsoft.com/en-us/aspnet/core/tutorials/first...

To be fair, I'm picking on aspnet core because I like it so much (via F#). The framework is great, but this gaming of public benchmarks (techempower, benchmarksgame) by "breaking the same rules as everyone else" rather than investing time into making these results more useful is disappointing.

I'd say I do about 40% raw ado and 60% ORM overall. I think raw ado is completely valid in a performance benchmark because you usually step down to raw ado for that very reason: optimizing performance.
If you take the time to dive into the source on may of these, you'll end up seeing so many strange things.
If you're thinking of evaluating language performance by looking at these benchmarks then think again. A lot of the benchmarks here (h2o.cr and pico.v being prime examples) are thin wrappers over libraries written in C/C++.
That's what's actually fairly impressive about C# being #1 on plaintext. Both ASP.NET and Kestrel (the frontend server) are both fully in managed code.
Various teams at Microsoft should be congratulated for their efforts. A few years back, ASP.NET was last (or near last) on this benchmark for simple, plaintext requests. They've come a long way and put some serious engineering into their system.
You can't even imagine. A goal has been set by the management to get specific results on some benchmarks, by improving the framework. We have weekly meetings to follow up on improvements and next tasks to keep track on these goals. And these meetings involve super qualified engineers and also brilliant open source contributors. I can t describe the level of technicity at each of these meetings. The impact on the framework is tremendous, you can follow it as our charts are public. And the goals are already reached, which is not yet reflected on TechEmpower as we only submit release bits. This is good for PR, but as much for also for any developper using dotnet.
I wrote this article 6 years ago and it got quite a bit of traction here on HN and Reddit: https://blog.jonathanoliver.com/why-i-left-dot-net/

Virtually every single item on my list is now completely irrelevant.

Although I'm not a daily developer using .NET anymore, I am astounded that such a turnaround was even possible. My hat is off to you and the teams you work with. Also, for the record C# is still a phenomenal language in my book.

I wish there was an additional set of data that uses this set, and then computes the average cost of using each framework on various cloud providers.

Its amazing to see python based stuff so low in the benchmarks, yet so high in general popularity.

In cloud based servers, where requests per second directly translates to inverse USD, its imperative costs are incorporated into the selection process for the software stack.

There are some writings by people from Stackoverflow on how much they saved going with native COMPILING NET stack.
People like bashing java, but java based frameworks (and other JVM based ones) are always at the top. Solid foundations, great virtual machine, epic tooling, great and reliable libraries (which do not change every 3 months like in node), and in the recent time much faster evolution of the language. And if you do not like java as a language (which has its downfalls and problems) you can use Kotlin or Scala, or even Haskell on the JVM if you fancy so.

Funny to see rails, django, laravel and other most popular frameworks from dynamic languages at the bottom of the list with such bad performance. Even the Java heavy weight Spring framework is much closer to the top.

Not to mention that JVM can support all kinds of concurrent, parallelism and scaling models.

I'd say performance in this benchmark is inversely related with how nice it is to express business logic in the respective language/framework, and that's a pretty big deal.

For example, https://github.com/an-tao/drogon (benchmark winner) or https://oci-pronghorn.gitbook.io/greenlightning/ just aren't pleasant to work with. For example, compare Green Lightning with http://sparkjava.com/ and look at how much slower Spark is while clearly being nicer to use.

I worked with Netty at work for two years and it was a dream to use something like Express or Flask on the side, not having to worry sharing/disposing ByteBufs, not dealing with the weird in-channel/out-channel abstraction (vs simple middleware), not investigating how async-/thread-safe a given library was, etc.

It just all goes back to trade-offs. You're going to have different considerations if you're building a Caddy competitor vs a complex game server. Another consideration is how horizontally scalable you can be which is going to be different for a forum vs a websocket server.

Vert.X is a relatively light layer on top of Netty and it is quite nice to work with.

Spring WebFlux is also built on top of Netty, and missing some benchmarks, but I have no reason to think it would perform poorly. Its very easy to use for anyone with Spring experience.

The fastest frameworks are all nonblocking, like Express. They use streaming and reactive patterns, if it wasn't for Java's types everywhere some people wouldn't be able to tell the difference between Vert.X using RxJava and Express. The main design difference is that JS does non-blocking on a single core vs Java which does thread-per-core. Basically, in Java you can scale out non-blocking across all cores and still share memory when you want to.

If you don't want to deal with thread safety and non-blocking DropWizard performs quite well using traditional threads.

Just saying, performance and easy of use are not always exclusive. Maybe Vert.X didn't exist back when you were using Netty but these days it would be unthinkable to use Netty itself, the abstraction Vert.X provides is far easier to use and you don't really lose any performance.

Java's bad performance reputation is 90% due to old versions of Spring. Spring Boot has come a huge way, with performance maybe 100X better than Spring 4. Spring WebFlux takes this even farther with a Vert.X-like api, probably reaching near-Netty performance

Netty is network IO framework and is almost never used as is to develop web applications. You usually plug it in your framework or http library which abstract over it (or Jetty, or Apache, or Undertow, or ...).

Look at Composite Scores in Techempower benchmarks. Jooby, Micronaut, Ktor, Dropwizard, Http4k.. are all nice to work with and are all pluggable with different network IO frameworks like Netty for example. And they are not barebones http layers only optimised for speed, they come with a bit of batteries included but not that much that you battle the framework. They are less magic and more composable with other libraries where you can more easily know what is happening. Similar to Express and Flask. But even a heavyweight Spring framework is at the top. Flask is stuck at the bottom. Even express has bad results.

And, this is a personal preference, but having built production apps in both dynamically typed and statically typed languages, I prefer the latter for various reasons. Using Kotlin to write backend apps is really enjoyable for me.

(comment deleted)
Glanced through the Drogon documentation, and it seems about as simple as a C++ web framework can be.

Greenlighting on the other hand would definitely be confusing for your average java developer. You have to write code very differently, if your goal is to not create any garbage. The style will be familiar to the HFT folk, but it's interesting to see some of those same approaches used for http services.

Slightly concerning that a few of their Netty benchmarks show high error counts. Usually meaning it could not handle the load, and behaved odd.
Damn, I just settled on a high performance framework for my next reddit killer meets slack clone + SaaS, and now I need to rethink that and add more months to actually launching, on the off-chance my app takes off and needs to quickly support millions of connections at once.
Stop reading my mind /s
haha, I do this every few months on a side project, then usually fall back to laravel+vue just cause it's what I know. Though, at least now swoole gives me a little more 'oomph' w/ php.
Modern c++ is surprisingly easy to pickup especially for developers coming from other languages, very easy to focus on the good parts and totally disregard the messy bits, working with older libraries is still a pain though.
What does the new "<T>" icon represent on the graphs (only present on some of the frameworks)?
Answering my own question:

> TPR-3 is a composite hardware environment score for a three-machine configuration, derived from all test types for TPR-tagged <T> frameworks. (More about TPR frameworks and TPR-1 versus TPR-3)[1]

[1] https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Tech...

Indeed, this is related to our (experimental) addition of a performance rating for the hardware environment we're testing on. This is rendered as the big number on the composite scores page. Theoretically, our benchmarks covering a diverse spectrum of technologies could be useful for measuring hardware and network performance. But a full run of all test implementations takes 111 hours, and I suspect few people who want to measure hardware performance would be willing to wait that long. So we selected a subset to use for hardware measurement, hence the icon you see.
I wrote a new C++ framework (lithium) last december ranked #4 in this benchmark and while this benchmark does a great job comparing raw performances of many frameworks on a set of test, here are my thoughts: 1/ Best Frameworks are all using Non blocking network IO (based on linux epoll or equivalents) 2/ Very few companies need such performances. Don't look for those number if you have less than 10M visitor/months 3/ Tuning the number of sql connections can do x5 on performances and all framework did not tune this parameter. 4/ Even if you a low ranked framework, it may not be the bottleneck of your stack so changing to a top performer may not change your thoughput overall.
A fun diversion, but largely meaningless.
Just came to say that it is an amazing and extremely thorough job they have done in all these runs. And also still mind-blowing how much work the fastest frameworks can do.