Go is extremely performant. But then again, isn't the author comparing apples to oranges so to speak? A Go program runs as a native binary, whereas Erlang and Scala/Akka run via their respective VMs. Wouldn't these results be expected?
You mean, he should implement a VM for Go or in Go? So the comparison is fair,
but totally useless by ignoring the way programs in those languages are
typically run? It wasn't meant to be fair, it was meant to be relevant.
Of course the results are more or less expected. JVM is by far the worst,
because it uses OS-level threads, while Go and Erlang use userland threads,
which are way cheaper to spawn and manage.
[edit] According to some other comment around here, Scala is supposed to use userland threads as well.
The author also measures .net core, which has the best performance of them all. I believe that is also running non native.
Moreover, the core assumption that native == faster is not true. One advantage of the jvm is that as a program runs, the runtime can internally profile and adjust the byte code on the fly in order to increase performance.
Hey! Yeah, I just noticed the author added .Net core (this wasn't apart of the benchmarks when it was first posted to HN). I agree that the assumption is violated. On the topic of native code though, check out the numbers for .Net core on Win 8.1 vs. Mac OS X (almost 3x faster on Windows, which is expected).
If anything is unfair it's that they're using long ints. Java (and I assume other JVM languages) have the overhead of allocation for most of the longs.
Testing with allocation in all languages will give a better idea of performance (and I won't be surprised if .Net core fares very well).
Is the .NET version actually spawning any threads at all? It uses Tasks to collect the results, but all the work is done in the main thread by the looks of it.
The async version also doesn't spawn as much Actors/Threads/whatever than all other implementations. It only runs the Task.WhenAll continuation in the ThreadPool - and probably even that not because it can recognize that all child tasks are ready and execute the continuation synchronously. The leaf tasks, whose amount is the biggest, are only synchronous function calls that return a fulfilled task instead of spawning anything. A fair comparison should use at least use something like `Task.Run()`, which would make it a lot slower. And then you can make it a lot faster again by using a custom TaskScheduler.
I think there are by far too many knobs to turn to make this comparison meaningful.
That was a quick update! Looks better now. I would have expected the difference to be much bigger. I made some similar comparison about a year ago (with normal .NET), and it performed slower than Go. However with a custom Actor implementation (similar to the Scalaz implementation) I was a lot faster. F# with the Hopac library also performed very good.
1. That the time (including start-up time) for something on the JVM would be longer because the JVM needs to start up.
2. That Go producing a native binary is automatically faster at things.
In reference to the first, the benchmark timings are started when the software actually starts running.
In terms of the second, being a native binary doesn't mean faster. Go still has a runtime. It's just included in the binary.
Go's performance advantage is probably a few things. Go was made to spawn millions of goroutines and Go uses primitive types. I don't know all that much about Scala, but I believe it doesn't do primitive types which means that every one of those Int objects requires following a pointer to get to the value. That's overhead. EDIT: this looks like it's wrong as Scala has a bunch of pre-defined classes that extend AnyVal and correspond to primitive types in Java.
It also looks like the Scala code is passing back a Long even when the value is still small enough to fit in an Int. That's doubling the amount of memory when that memory only needs 64-bits near the end stages. By contrast, the Go code does `sum := 0` which I'm not actually sure how that's working. If I initialize a 32 bit int without specifying it, I see that I wrap around (http://play.golang.org/p/PiiMZmEzKp). So, I'm not sure how initialising it to 0 and then adding to something over an int32 works. Maybe Go uses a different default int depending on platform?
Go's implementation is also simpler. In the Scala version, there's a receive method and the sub-actors just pass back when they're done potentially causing a bunch of switching. Go, on the other hand, creates all the goroutines and only when that's done starts receiving from them. In the Scala version, the program might think that there's value in switching from creating the sub-actors to receiving from them - and they should pass back very quickly. The implementation here in Go doesn't allow for that switching.
When dealing with something as synthetic as this, all of these little things matter.
It would still be fair as long as they are equally idiomatic in their respective languages. If one solution avoids slow startups,requires less of programmer's 'think cycles', the need to wait twiddling thumbs while something compiles or warms up; then there is still a legit point to be made. Not a fan of Go by any means but have to give it credit for making concurrency more mainstream
actually comparing jitted vm's without a warmup is extremly dump.
Second: using the default thread dispatcher for this kind of scenario is pretty dumb on scala (also I think erlang could be made faster, too. but since I'm not an erlang guy...) Fixed Thread Pool (GOMAXPROCS vs ForkJoin)
There's an update for Akka version, which measures 12/8/4.4 seconds on subsequent attempts, so yes, warmup is important... but somehow that seems kinda unfair for other languages too.
I agree that using the default thread dispatcher is not how this should be used. The application's actors should create and use its own ExecutionContext.
The point to understand here is that Go uses lightweight concurrency and has a layer of indirection between goroutines and hardware threads. Goroutines allocate a tiny stack and Go uses a software scheduler to jump between routines. Then that Go scheduler starts hardware threads as workers to start executing the routines.
The net result is that Go is relatively efficient for naive programs that start hundreds or thousands of threads and let them all run.
I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread in the software and so it is using the kernel scheduler to switch between them with all the overhead that entails.
But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.
I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.
Unclear wrt Scala, jakozaur asserts the bench does use pthreads.
Erlang does use lightweight processes, but the configuration isn't tuned for high creation/destruction throughput: on my system, the smallest possible process is 333 words, or 2.5Kb (control structures, default stack and default private heap)
Scala does not have lightweight threading. It uses the same thread pools from Java. All your millions of Actors are scheduled onto one of these thread pools.
Which means you need to program with futures in order to not do blocking IO on your limited thread pool that the actors are multiplexed across. So at that point you need to write node.js style code.
Coming from an Erlang world to akka, this blew my mind.
> I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread
No. Erlang is only ~five times slower than Go, according to the benchmark
run, which is a marvellous result given the Erlang's VM is slow as hell, even
with HiPE. Scala/Akka is slower by thirty times under this workload.
> But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.
Oh, I bet it wouldn't be faster, but it surely would add tons of complexity.
Erlang was specifically designed for programmer to spawn a thread for each
activity. New client connected? Spawn. Need a background computation? Spawn.
Need a connection to external entity? Spawn. Need to do cleanup (reliably)
after this computation terminates? Spawn. Go shares this approach to some
extent.
If you do file system related things in a goroutine it sometimes spins up a new OS thread, and if you are not careful you'll have thousands of them in no time and they never seem to die.
> I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread in the software and so it is using the kernel scheduler to switch between them with all the overhead that entails.
Incorrect.
> I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
object FutureGen {
def caller(num: Int, size: Int, div: Int): Future[Long] = {
if (size == 1) {
Future.successful(num.toLong)
} else {
val futures = for (i <- 0 until div) yield caller(num + i * (size / div), size / div, div)
Future.sequence(futures).map(_.sum)
}
}
}
object Root extends App {
// Warmup
Await.result(FutureGen.caller(0, 1000000, 10), Duration.Inf)
val startTime = System.currentTimeMillis()
val x: Long = Await.result(FutureGen.caller(0, 1000000, 10), Duration.Inf)
val diffMs = System.currentTimeMillis() - startTime
println(s"Result: $x in $diffMs ms.")
}
This would give Scala: 499ms and Go: 436ms which is pretty neat, since go actually has no type system. (And I'm using the global execution context, which could also be changed) Currently your akka version is slow, since it's creating a new class on every call, so you would have like a dozen more methods to dispatch than my version.
That's not the same. You are just creating a bunch of completed futures. Your code basically is just summing numbers up in a recursion here. Which is not what OP is trying to benchmark.
actually Future.sequence() will run dispatched. it would just make no sense to run the single number concurrently, as the go version won't do this, too.
Future.sequence uses the global Executer.
sum := 0
for i := 0; i < div; i++ {
sub_num := num + i * (size / div)
go skynet(rc, sub_num, size / div, div)
}
This is the go version and only the inside will run on a thread so I made a fair comparsion P.S: The go version will actually run single threaded when you are having GOMAXPROCS=1.
Go example seems off too from what the author intentions were. As for the dispatcher, I don't believe it will actually give any resource to a future that is already complete? OP wanted to have messages going through 1M actors, so there would be a moment when each actor gets a thread from a thread pool.
However it makes a difference if I create 1M actors and dispatch them via reflection or if I create some of them and passing results back and forth.
actually goroutines would use static dispatching and don't have a lot of overhead.
however a scala actor is totally different than any goroutines or erlang actors, they are keeping a lot of state. a promise/future is actually a little bit better in this kind of sense, still the jvm version will actually (even when optimized) use way more memory.
This is a silly argument. If I read you right, you are saying that a goroutine is closer to a promise/future than a Scala actor in terms of resources used, and therefore it's OK to use futures instead of actors in the Scala code and compare them that way.
That betrays the whole benchmark. The basic point of the benchmark is to answer the question, "How expensive is it to create and run 1,111,111 actors in various languages?" You can't make a manual optimization that causes only 111,111 actors to be created and then say that it's a better comparison.
goroutine is not an actor. its a concurrency mechanism.
so a goroutine is not an actor, however a future (http://doc.akka.io/docs/akka/2.4.1/scala/futures.html#Use_Di...) is definitly closer to an actor than a goroutine. so you can't compare golang and scala/erlang. too bad, huh?
And when you are clever you would be comparing monads, which actually a goroutine could be. and a future/promise, too.
Btw. Even a Future does more than a goroutine do. it won't crash your whole program when it fails, which a goroutine does.
Yes, goroutines are a concurrency mechanism. Actors are also a concurrency mechanism, so I fail to see why the comparison is unfair. As you say, Actors are safer in that uncaught exceptions in an actor will not bring down a program. They have a few more dials and knobs, you can store references to them, and they are more heavyweight. They have more overhead as can be expected from these facts and this is exhibited in benchmarks like this one.
Futures are not a concurrency mechanism precisely. They are a monadic data structure around possibly-concurrent execution. You can wrap a Future around concurrent execution by an Actor, or you can do what you have done and wrap a Future around a sequential recursive computation. As with the synchronous .NET core implementation of the benchmark, the performance is better due to not having the overhead of Actors or other concurrency. Like the synchronous .NET core implementation writing the computation this way is not a fair comparison to the other languages in the benchmark and the high performance does not indicate anything about the efficiency of the concurrency mechanisms available in Scala.
The goroutine is actually in a separate concurrently executing computation. The "go skynet()" call from the Go example is a real concurrent call that will block on the "c <- xyz" lines and resume execution when a reader consumes the value. Calling Future.successful is not the same thing.
"c <- num" is a blocking statement. Since c is an unbuffered channel, execution cannot proceed past the statement until a reader consumes a value. The reader in this case creates 10 goroutines executing "c <- num" and only then starts consuming values. If the statements you wrote were inlined execution would immediately halt in a deadlock.
Just gonna put this out there: Scala is full of these scenarios where very small changes to the code have disproportionately large changes to the outcome. It's one of the reasons I'm really not a fan of the language. There are so many features that are mentioned in the literature and lore that you have to learn to not use, for fear of pervasive performance penalties.
This is a great example. I've done a lot of Scala and o found the original version very obvious to read, but only experience tells me why the compiler can't make it efficient. Your version is not really worse in any way, but I think it's fair to say that it shouldn't be obvious to everyone why one is suboptimal at first glance.
even if your downvoted. I think what you wrote is somehow true. it's really hard to optimize. but I don't think that Scala is the only language which has this downside.
Also you could write really really awful code with scala.
The parent comment is precisely saying that they don't all implement the exact same concept. Also, different runtimes will give different results, even if the concept is exactly the same.
If a compiler could proof that creating the channels/go routines/what-have-you does not influence the arithmetics (e.g. because only associative operations are used and the order doesn't matter), it could constant-fold the loops and just output 499999500000 directly.
I don't know any compiler (and language implementation) that can do that, though.
Any of the VM languages could probably do this heuristically as well: the "if size == 1" base case depends only on one of the parameters, you could hoist that out of the concurrent thread/coroutine and into the caller. Then the code path that hits only ever assigns to a future/promise/array/whatever and immediately returns so it won't block and has no other side effects so it can run synchronously for less than the cost of a context switch.
In general programmers are pretty good about not kicking off expensive computations they don't need to, so the easy pickings here are probably pretty small. And as soon as there are any side effects at all (which includes any writes to shared memory) it becomes very hard to reason about moving computation from one thread to another. So it's easy to see why compiler writers are not super excited to start optimizing across concurrent threads.
Probably not entirely, Erlang's lightweight processes have their own private heap (~1.8KB by default on a 64b system), a process dictionary and likely more control structures, and the language is mainly interpreted. A 4x different in performance may be on the high side (idk), but I wouldn't expect erlang processes to be quite as fast to spawn as goroutines.
edit: and the erlang bench allocates a bunch of lists on startup: https://github.com/atemerev/skynet/blob/master/erlang/skynet... `lists:seq()` will allocate a 10-elements linked list, considering this is called recursively and concurrently it might contribute to the issue (or not)
The major problem, without any measurement and just guesswork, is that you have a larger allocation overhead in Erlang. Not only do you need the process control block for the process, you also need to go to the memory subsystem and grab a heap for the process. In the Go variant, you are allocating out of the same heap and you can usually avoid some allocation there because everything you allocate is used. Not so in the Erlang system.
Were these processes/goroutines to do anything more complex, then the difference goes toward zero. But the constant overhead at the lowest level is probably higher.
Yeah, I was wondering that. There could always be the general overhead in the VM, but the HIPE version (which is compiled IIRC) should be closer. A commenter mentioned that the erlang version would be creating intermediate lists that would have to be garbage collected (see all the lists:seq(0, Div-1) ), but I wouldn't know how to avoid that without writing unidiomatic code. I'm no erlang expert though.
I don't think the lists would have to be collected, erlang processes have a relatively hefty private heap (233 words) by default, so chances are the process just dies without needing to run a GC[0]. The lists do need to be allocated and initialised though, which may or may not contribute to the runtime.
[0] the way I understand it, each process would generate a 10-elements list of small integers, which would be 21 words (1 word + 1 word/element + the sum of the element sizes — 1 word each for small integers)
there is a major flaw in the erlang version which causes unnecessary list allocation and garbage collection multiplied by 1 million times. I would try to replace lists:seq(0,9) with a tail recursion.
Even that it's a small list of 10 integers, it is still created 1 million times.
EDIT: it seems that list:seq overhead is insignificant - so it doesn't matter, maybe erlang compiler already optimizing it. I even inlined the calls to spawn - it still the same result.
But I found another problem: it seems that some processes are stuck - so if you run benchmark the 3rd time - yo get out of processes.
EDIT2:
Note, that unlike other solutions. Erlang has process isolation, i.e. if there is a panic in one of your goroutines - your entire server will crash, but if there is an exception in your erlang process - only that process will crash, not affecting other processes in your server (unless they linked with it).
> there is a major flaw in the erlang version which causes unnecessary list allocation and garbage collection multiplied by 1 million times.
There's probably an alloc cost, but there should be no GC cost: erlang processes get a 233 words heap by default, a list of 10 integers is 21 words, so the process should die without needing to run GC.
yes, when process ends/dies all the heap is reclaimed without regular GC.
I also tried spawn_opt/5 with reduced min_heap_size option - it doesn't help.
> I also tried spawn_opt/5 with reduced min_heap_size option - it doesn't help.
Yeah whether using spawn_opt or VM options (-hms and -hpds) it doesn't look possible to reduce the process heap (or process dict) below the initial size, only to increase it.
Another microbenchmark flamebait on top of HN? Seriously?
Those things never describe real workloads - in such a view a Porsche is always better than a truck because it accelerates faster. Tough luck if the workload is shipping 20 tons of manure. Put that in your Porsche.
I'm pretty sure this doesn't measure what it claims. It's a mostly sequential flow with very little meaningful concurrent work going on. I would imagine a for loop would put all of those to shame. I'm not sure why anyone would optimize their runtime for what's basically yet another process ring benchmark but I know it's gotten slower over time on a couple of the measured cases because it's just not a real example and making this fast trades off other things like good scheduler balancing.
Also check out Pony if you want something were spawn and send become basically free. It'll run pretty close to for-loop speed for the number adding which should be a baseline calculated in all implementations.
He has a valid point though. Goroutines are more like for-loops than processes. You can't kill them, can't manipulate them, can't store references to them, etc. Comparing that to actors is incorrect.
Someone should hack together C coroutines using Duff's device and throw it up there. It'd win handily, probably around 1ms. You've got around 100 million instructions to use in 1ms on that i7-4770 and no cache to hit, go nuts.
I think the .NET core implementation is flawed... Task.FromResult doesn't actually happen on another thread, it returns an already completed Task synchronously. You might want to use Task.Run instead.
Doesn't matter. Unless you do TaskFactory.StartNew or Task.Run, it's not going to try to schedule work on another thread. Everything is happening in the main thread in your code.
See the updated version. Runs about 15-20% slower, which is far less than the difference between single-threaded version and task-based one. Also, it's pretty easy to see in Task manager how all the cores get loaded.
That looks better. However, I am really not sure what you are trying to achieve here. You didn't specify any scheduler, it means your tasks will use a default scheduler and that scheduler uses threadpool [1] and it will never create 1M threads. And as there is no actual work performed in the Task, things will run reasonably fast.
I am having hard time understanding what you are trying to test here. I am worried that .NET code is not really testing what you think it is.
You can't create 1M threads on Windows (or OS X, for that matter) because there won't be enough RAM for stack space. (also, running more threads than physical cores on CPU-bound task is pretty pointless).
Maybe you mean we should schedule something which has a complete event loop? That would be a lot more fair wrt the Akka version, but the Go version does not create any threads either.
In go version you're using unbuffered channel, which will cause a lot of extra goroutine swithces
please try to change line 10 to:
rc := make(chan int, 10)
for example.
After this change my results changed from:
Result: 1783293664 in 61586 ms.
to:
Result: 1783293664 in 28401 ms.
I think a lot of people are overlooking something really important here. There are numerous comments about optimizing the code to perform better on this task. I might even go as far to say people are taking offense because their language of choice didn't perform as well they'd hoped. Many of these comments add complexity to the solution and make it harder to maintain. I like the fact that the author kept things simple. It's kind of like using the default settings for each language and then comparing the results. Both the simple version and optimized version are important data points, and they give you a better view of each languages limitations.
sorry but the scala version is actually pretty pretty dumb.
also go is not better than all of these languages.
what we learned here, is actually that go is easier since you only have one common way to do this task. go has goroutines, but go doesn't have runnables, futures, actors, completionstages, executioncontexts. GO has GOMAXPROCS and not dispatchers, thread pools, etc..
Also the JVM eve Java only code would actually beat golang in many ways, but thats nothing you should be scared of. Golang is 1 1/2 year old, the JVM lives like 20+ years and they spent numerous times about the GC implementations (they could be even changed so that you could use a better GC for conurrency or for parallelism or for synchronous execution).
Go is good for the task it is created. Other languages maybe tend to be more general so that you could do many things and due to their years of knowledge they are actually more optimized than go could be / they could be more optimized than go could be.
> go doesn't have runnables, futures, actors, completionstages, executioncontexts
Some of us (most of us, I would contend) who have been successfully programming highly concurrent software without them for 20+ years are totally OK with that.
> Some of us (most of us, I would contend) who have been successfully programming highly concurrent software without them for 20+ years are totally OK with that.
I work on parallel software every day and I would not be OK with having threads and channels as my only tools. I would agree that they're usually the best thing to reach for initially, but it's essential to be able to have low-level control over the scheduling if you want to get good parallel performance out of the hardware. I have workloads that today result in 3x speedups on 4 cores but would become slower than the sequential algorithm on 1 core if I were to rip out the highly-tuned parallel scheduler and replace it with threads and channels.
To be fair, though, I'm working on CPU-bound software and if my workload were I/O bound I'd likely have a different outlook on this.
yeah sorry however I would actually use go version 1 as a reference.
everything before was changing just too often, there was go fix however still many changes. also everything before 1.0 wasn't targeted too much about performance.
also I can remember how much the template spec changed. thats what I used these days quite heavily. I have a book about go that dates 2011, and I think that was go 0.7 as far as I remember.
I am scared that Java takes an order of magnitude more memory than Go with its 20+ years of optimization and all. Java's GC does not appear to be significantly better than Go's one. Also Go's runtime and GC both written in Go, something Java has not been able to do so yet. So Java appears less general purpose to me. Yes Java has humongous number of libraries and frameworks but that is expected of a popular language that is 20+ years old.
Considering Go is showing under ~10ms GC pauses. Java's advantage seems theoretical rather than real world. I work in Java at my job and I have not seen such low GC pauses in Java ever.
edit:
Also considering huge garbage idiomatic Java libraries and user code generates, Generational GC in Java is a basic requirement not a powerful advantage over Go.
Bump allocation in the nursery is a huge advantage. There's a reason why fast Go code goes out of its way to avoid allocations (notice that the benchmark tool contains an allocation profiler), and it's because allocations are expensive when you don't have a generational GC. If you do have a generational GC, then allocations are incredibly cheap: just a pointer bump and check. It's like 4 or 5 instructions compared to the dozens in a typical malloc. With that kind of collector, you don't have to spend nearly as much time tuning your program to avoid allocations. This is the reason why Java has been slow to add value types: when you have a bump allocator, they really aren't needed as much (which is not to say they aren't needed at all).
In the HotSpot VM, you can tune the GC to get any type of max garbage collection pause you want (-XX:MaxGCPauseMillis). This is basic functionality that any concurrent garbage collector supports. The downside, of course, is that pauses will happen more frequently, reducing throughput. That is why just saying "it's under 10 ms, so it's really fast" is very misleading: when your GC is interruptible, you can of course get any max pause time you want. But the GC may not be able to keep up with your allocation rate, and it may start being invoked too often. That's why you need to not just look at latency, but throughput--something that sound bites won't capture.
Pause time is only one metric that's relevant when looking at GC, another very important one is throughput.
In fact you can get very low (10-50 msec) pause times with the latest JVMs even with very large heaps, you can just request it at the command line, but you'll pay for it with slower execution. There is always a tradeoff. Go doesn't have anything fundamentally new in this field.
There is a report here on an adapted HotSpot (not integrated into the mainline JVM though) that gives <10msec pause times at the 99th percentile and an 8 gigabyte heap:
http://welf.se/files/OL15.pdf
But you pay for it with a 15% CPU overhead.
Go's GC seems to actually do a full concurrent mark/sweep of the entire heap every single time. That suggests to me that throughput could be rather poor.
> Go's GC seems to actually do a full concurrent mark/sweep of the entire heap every single time. That suggests to me that throughput could be rather poor.
Not to mention that you lose bump allocation in the nursery. Nongenerational GC is a lose-lose all around.
Right, scanning large heaps means someone has to pay along the way. The Go GC pays with lowered throughput much like the adapted HotSpot VM is showing. The 10ms pause times for go were reported for a 250 gigabyte heap, however, and that is pretty amazing.
But the lowered throughput has to be present as you are scanning the heap and you have enabled the write barrier to make the concurrent scanning work its invariants. In fact, the newest versions will make aggresively allocating goroutines run slower by using some of their timeslot when they request data do run the mark phase.
This has no external data so workers may avoid a cache fetch let alone having enough data for regular cache misses. Even a simple strlen call in each worker could begin to add the kind of issues that affect meaningful performance.
How a language handles misses/blocking/threading is more important than what it can do in a perfect cpu only world.
A true coder wanting to implement things near correct with algorithm complexity (big O), will try to find how to implement the right code by taking time to research understood and time tested patterns they've learned (it is an applied science after all) and implement them.
Saying the average joe doesn't care to understand what his/her Lang is doing under the hood which will doom them to repeat complexity that many here will respond with "duh" is not justification that a Lang is better - because it supports that behavior "better".
I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion.
If one doesn't care for efficiency at all then who cares about benchmarks?
You missed the point. People aren't posting better algorithms, they're posting domain specific knowledge to improve efficiency. The kind of stuff that takes you away from building new features that add value. Benchmarking and efficiency is an important datapoint when picking a language. It's important to understand how un-optimized code performs too.
In my comment I purposely mentioned erlang because to implement algorithmic efficiency you must have domain expertise.
"I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion."
Tail recursion is an efficiency, doing it right in erlang requires domain expertise.
Building new features and adding value is business. Someone with domain knowledge in any of the langs will be able to implement those business values in the best possible way.
If your business is built with shitty coders doing shitty things you will eventually find yourself in hot shit. Even if it takes 5 years. What comes around goes around.
I would take the domain knowledge developer over any monkey anytime. That's your 10x developer. That's who built all that awesome open source stuff you use all day. Not the business feature value guy.
He builds businesses that fail 99% of the time. That other guy building that obscure Lang which turns out useful for that niche topic (I.e concurrency) lives on even in concept to build or inspire other langs/ers.
You see, that domain expert knows to pick the right algorithm for the right job.
He knows the right Lang for the right job.
Even if that job is only business value.
Long live domain knowledge experts and those who wish to attain it.
From the description, almost 90% of created tasks in this benchmark are tiny "leaf" tasks, which are quite likely to be optimized away completely. So it looks like the benchmark compares how well these compilers optimize one particular case, and not general performance of goroutines/tasks/whatever...
162 comments
[ 3.7 ms ] story [ 115 ms ] threadOf course the results are more or less expected. JVM is by far the worst, because it uses OS-level threads, while Go and Erlang use userland threads, which are way cheaper to spawn and manage.
[edit] According to some other comment around here, Scala is supposed to use userland threads as well.
Yup.
However, Actors are not threads, so in a way it's like user threads. One million actors will be scheduled to run on a pool of OS threads.
Moreover, the core assumption that native == faster is not true. One advantage of the jvm is that as a program runs, the runtime can internally profile and adjust the byte code on the fly in order to increase performance.
Please note that Windows and OS X is benchmarked on different CPUs!
You're welcome to re-test at dual-boot MacBook together so we can meaningfully compare OS X/Windows
Testing with allocation in all languages will give a better idea of performance (and I won't be surprised if .Net core fares very well).
I think there are by far too many knobs to turn to make this comparison meaningful.
1. That the time (including start-up time) for something on the JVM would be longer because the JVM needs to start up.
2. That Go producing a native binary is automatically faster at things.
In reference to the first, the benchmark timings are started when the software actually starts running.
In terms of the second, being a native binary doesn't mean faster. Go still has a runtime. It's just included in the binary.
Go's performance advantage is probably a few things. Go was made to spawn millions of goroutines and Go uses primitive types. I don't know all that much about Scala, but I believe it doesn't do primitive types which means that every one of those Int objects requires following a pointer to get to the value. That's overhead. EDIT: this looks like it's wrong as Scala has a bunch of pre-defined classes that extend AnyVal and correspond to primitive types in Java.
It also looks like the Scala code is passing back a Long even when the value is still small enough to fit in an Int. That's doubling the amount of memory when that memory only needs 64-bits near the end stages. By contrast, the Go code does `sum := 0` which I'm not actually sure how that's working. If I initialize a 32 bit int without specifying it, I see that I wrap around (http://play.golang.org/p/PiiMZmEzKp). So, I'm not sure how initialising it to 0 and then adding to something over an int32 works. Maybe Go uses a different default int depending on platform?
Go's implementation is also simpler. In the Scala version, there's a receive method and the sub-actors just pass back when they're done potentially causing a bunch of switching. Go, on the other hand, creates all the goroutines and only when that's done starts receiving from them. In the Scala version, the program might think that there's value in switching from creating the sub-actors to receiving from them - and they should pass back very quickly. The implementation here in Go doesn't allow for that switching.
When dealing with something as synthetic as this, all of these little things matter.
No? Erlang and Scala both have JITs. The JVM does quite a bit more optimization than Go's compilers do.
Second: using the default thread dispatcher for this kind of scenario is pretty dumb on scala (also I think erlang could be made faster, too. but since I'm not an erlang guy...) Fixed Thread Pool (GOMAXPROCS vs ForkJoin)
Warmup's a tricky issue, see this brand new paper: http://arxiv.org/abs/1602.00602
The net result is that Go is relatively efficient for naive programs that start hundreds or thousands of threads and let them all run.
I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread in the software and so it is using the kernel scheduler to switch between them with all the overhead that entails.
But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.
I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.
Erlang does use lightweight processes, but the configuration isn't tuned for high creation/destruction throughput: on my system, the smallest possible process is 333 words, or 2.5Kb (control structures, default stack and default private heap)
http://doc.akka.io/docs/akka/2.4.1/scala/dispatchers.html
Coming from an Erlang world to akka, this blew my mind.
No. Erlang is only ~five times slower than Go, according to the benchmark run, which is a marvellous result given the Erlang's VM is slow as hell, even with HiPE. Scala/Akka is slower by thirty times under this workload.
> But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.
Oh, I bet it wouldn't be faster, but it surely would add tons of complexity. Erlang was specifically designed for programmer to spawn a thread for each activity. New client connected? Spawn. Need a background computation? Spawn. Need a connection to external entity? Spawn. Need to do cleanup (reliably) after this computation terminates? Spawn. Go shares this approach to some extent.
Incorrect.
> I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.
There is https://github.com/dpc/mioco for I/O-based coroutines. For CPU bound tasks you want something like https://github.com/nikomatsakis/rayon.
He says, with absolutely no profiling whatsoever... I would like someone to offer some evidence though :)
If we want to compare apple to apples, we should use thread pool shared across Scala actors: http://stackoverflow.com/a/1597942/341181
Just add 7 lines of code and Scala should get at least 10x faster.
That betrays the whole benchmark. The basic point of the benchmark is to answer the question, "How expensive is it to create and run 1,111,111 actors in various languages?" You can't make a manual optimization that causes only 111,111 actors to be created and then say that it's a better comparison.
And when you are clever you would be comparing monads, which actually a goroutine could be. and a future/promise, too.
Btw. Even a Future does more than a goroutine do. it won't crash your whole program when it fails, which a goroutine does.
Futures are not a concurrency mechanism precisely. They are a monadic data structure around possibly-concurrent execution. You can wrap a Future around concurrent execution by an Actor, or you can do what you have done and wrap a Future around a sequential recursive computation. As with the synchronous .NET core implementation of the benchmark, the performance is better due to not having the overhead of Actors or other concurrency. Like the synchronous .NET core implementation writing the computation this way is not a fair comparison to the other languages in the benchmark and the high performance does not indicate anything about the efficiency of the concurrency mechanisms available in Scala.
"c <- num" is a blocking statement. Since c is an unbuffered channel, execution cannot proceed past the statement until a reader consumes a value. The reader in this case creates 10 goroutines executing "c <- num" and only then starts consuming values. If the statements you wrote were inlined execution would immediately halt in a deadlock.
This is a great example. I've done a lot of Scala and o found the original version very obvious to read, but only experience tells me why the compiler can't make it efficient. Your version is not really worse in any way, but I think it's fair to say that it shouldn't be obvious to everyone why one is suboptimal at first glance.
The point is exactly to compare different implementations of the same concept. That's always the point in performance comparisons!
I don't know any compiler (and language implementation) that can do that, though.
In general programmers are pretty good about not kicking off expensive computations they don't need to, so the easy pickings here are probably pretty small. And as soon as there are any side effects at all (which includes any writes to shared memory) it becomes very hard to reason about moving computation from one thread to another. So it's easy to see why compiler writers are not super excited to start optimizing across concurrent threads.
edit: and the erlang bench allocates a bunch of lists on startup: https://github.com/atemerev/skynet/blob/master/erlang/skynet... `lists:seq()` will allocate a 10-elements linked list, considering this is called recursively and concurrently it might contribute to the issue (or not)
And Python is compiled to bytecode and executed on the CPython virtual machine. That's still interpreted.
Were these processes/goroutines to do anything more complex, then the difference goes toward zero. But the constant overhead at the lowest level is probably higher.
[0] the way I understand it, each process would generate a 10-elements list of small integers, which would be 21 words (1 word + 1 word/element + the sum of the element sizes — 1 word each for small integers)
https://github.com/atemerev/skynet/blob/master/erlang/skynet...
EDIT: it seems that list:seq overhead is insignificant - so it doesn't matter, maybe erlang compiler already optimizing it. I even inlined the calls to spawn - it still the same result. But I found another problem: it seems that some processes are stuck - so if you run benchmark the 3rd time - yo get out of processes.
EDIT2: Note, that unlike other solutions. Erlang has process isolation, i.e. if there is a panic in one of your goroutines - your entire server will crash, but if there is an exception in your erlang process - only that process will crash, not affecting other processes in your server (unless they linked with it).
There's probably an alloc cost, but there should be no GC cost: erlang processes get a 233 words heap by default, a list of 10 integers is 21 words, so the process should die without needing to run GC.
Yeah whether using spawn_opt or VM options (-hms and -hpds) it doesn't look possible to reduce the process heap (or process dict) below the initial size, only to increase it.
and speed doubles :)
Those things never describe real workloads - in such a view a Porsche is always better than a truck because it accelerates faster. Tough luck if the workload is shipping 20 tons of manure. Put that in your Porsche.
Also check out Pony if you want something were spawn and send become basically free. It'll run pretty close to for-loop speed for the number adding which should be a baseline calculated in all implementations.
Indeed, see ".net sync" version. The point is to measure process spawn/message passing performance.
For those who don't know what you're referring to, an example:
http://benchmarksgame.alioth.debian.org/u64q/performance.php...
I am having hard time understanding what you are trying to test here. I am worried that .NET code is not really testing what you think it is.
[1] https://msdn.microsoft.com/en-us/library/dd997402(v=vs.110)....
Maybe you mean we should schedule something which has a complete event loop? That would be a lot more fair wrt the Akka version, but the Go version does not create any threads either.
After this change my results changed from: Result: 1783293664 in 61586 ms. to: Result: 1783293664 in 28401 ms.
also go is not better than all of these languages. what we learned here, is actually that go is easier since you only have one common way to do this task. go has goroutines, but go doesn't have runnables, futures, actors, completionstages, executioncontexts. GO has GOMAXPROCS and not dispatchers, thread pools, etc..
Also the JVM eve Java only code would actually beat golang in many ways, but thats nothing you should be scared of. Golang is 1 1/2 year old, the JVM lives like 20+ years and they spent numerous times about the GC implementations (they could be even changed so that you could use a better GC for conurrency or for parallelism or for synchronous execution). Go is good for the task it is created. Other languages maybe tend to be more general so that you could do many things and due to their years of knowledge they are actually more optimized than go could be / they could be more optimized than go could be.
Some of us (most of us, I would contend) who have been successfully programming highly concurrent software without them for 20+ years are totally OK with that.
I work on parallel software every day and I would not be OK with having threads and channels as my only tools. I would agree that they're usually the best thing to reach for initially, but it's essential to be able to have low-level control over the scheduling if you want to get good parallel performance out of the hardware. I have workloads that today result in 3x speedups on 4 cores but would become slower than the sequential algorithm on 1 core if I were to rip out the highly-tuned parallel scheduler and replace it with threads and channels.
To be fair, though, I'm working on CPU-bound software and if my workload were I/O bound I'd likely have a different outlook on this.
https://youtu.be/rKnDgT73v8s
https://blog.golang.org/go-version-1-is-released
Java's GC is generational, allowing bump allocation in the nursery. That is a huge advantage.
In the HotSpot VM, you can tune the GC to get any type of max garbage collection pause you want (-XX:MaxGCPauseMillis). This is basic functionality that any concurrent garbage collector supports. The downside, of course, is that pauses will happen more frequently, reducing throughput. That is why just saying "it's under 10 ms, so it's really fast" is very misleading: when your GC is interruptible, you can of course get any max pause time you want. But the GC may not be able to keep up with your allocation rate, and it may start being invoked too often. That's why you need to not just look at latency, but throughput--something that sound bites won't capture.
Pause time is only one metric that's relevant when looking at GC, another very important one is throughput.
In fact you can get very low (10-50 msec) pause times with the latest JVMs even with very large heaps, you can just request it at the command line, but you'll pay for it with slower execution. There is always a tradeoff. Go doesn't have anything fundamentally new in this field.
There is a report here on an adapted HotSpot (not integrated into the mainline JVM though) that gives <10msec pause times at the 99th percentile and an 8 gigabyte heap:
But you pay for it with a 15% CPU overhead.Go's GC seems to actually do a full concurrent mark/sweep of the entire heap every single time. That suggests to me that throughput could be rather poor.
Not to mention that you lose bump allocation in the nursery. Nongenerational GC is a lose-lose all around.
But the lowered throughput has to be present as you are scanning the heap and you have enabled the write barrier to make the concurrent scanning work its invariants. In fact, the newest versions will make aggresively allocating goroutines run slower by using some of their timeslot when they request data do run the mark phase.
See the -XX:MaxGCPauseMillis=NNN argument, you might be able to set it to 10ms on highly performant hardware.
Summing a bunch of numbers is a silly task anyways
How a language handles misses/blocking/threading is more important than what it can do in a perfect cpu only world.
Saying the average joe doesn't care to understand what his/her Lang is doing under the hood which will doom them to repeat complexity that many here will respond with "duh" is not justification that a Lang is better - because it supports that behavior "better".
I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion.
If one doesn't care for efficiency at all then who cares about benchmarks?
In my comment I purposely mentioned erlang because to implement algorithmic efficiency you must have domain expertise.
"I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion."
Tail recursion is an efficiency, doing it right in erlang requires domain expertise.
Building new features and adding value is business. Someone with domain knowledge in any of the langs will be able to implement those business values in the best possible way.
If your business is built with shitty coders doing shitty things you will eventually find yourself in hot shit. Even if it takes 5 years. What comes around goes around.
I would take the domain knowledge developer over any monkey anytime. That's your 10x developer. That's who built all that awesome open source stuff you use all day. Not the business feature value guy.
He builds businesses that fail 99% of the time. That other guy building that obscure Lang which turns out useful for that niche topic (I.e concurrency) lives on even in concept to build or inspire other langs/ers.
You see, that domain expert knows to pick the right algorithm for the right job.
He knows the right Lang for the right job.
Even if that job is only business value.
Long live domain knowledge experts and those who wish to attain it.
My laptop runs the original in about 3.07s and my improvements bring it down to 1.20s though I think there's room for improvement.
[EDIT] My improvements were actually not the same algorithm as the original's, with algorithm corrections my version is actually slower!