HN Explorer

Is Go fast enough for large computations?

9 points by 999900000999 ↗ HN
Let's say I basically want to generate a series of random numbers billions of times.

Would Go be a good choice for this? I keep trying to learn Rust, but it's very difficult for me.

20 comments

[ 0.18 ms ] story [ 60.9 ms ] thread
c0balt
> Let's say I basically want to generate a series of random numbers billions of times.

That's a bit vague, if you only need pseudo random numbers consider a xorshift based approach, e.g., https://pkg.go.dev/github.com/db47h/rand64/xorshift. Xorshift is fast and simple.

> Would Go be a good choice for this? I keep trying to learn Rust, but it's very difficult for me.

If you need fast but don't want to write Rust or C(++), Go is a good choice for this. It will likely also have a good amount of examples and existing code available for most common problems.

speedgoose
You will most likely be IO bounded.
cpach
I don’t follow. Why do you think so?
speedgoose
Unless you write your numbers to /dev/null, your program will wait a bit for the storage, the network, or the next program.
machinekob
If you are not doing AI with custom Go code or low lvl close to hardware programming -> No. Every other instance -> Yes.
thrwwycbr
I mean, it's very hard to do and you have to build the GPU shader/processing pipeline yourself, but it's not impossible. [1] Gorgonia implemented most of the necessary libraries for tensor math for their alphago implementation.

[1] https://github.com/gorgonia/agogo

bjourne
Let's do some napkin math. A modern cpu runs at 4 GHz so it runs 4 billion cycles per second. Generating a random number takes, if we are being very pessimistic, 100 cycles. So we can generate 40 million random numbers per second. There's some overhead for seeding the random number generator too, but it's amortized over the large number of random numbers generated. You want a billion number so it takes 25 seconds. If Go is ten times slower (it isn't) than good C code, the job still takes only 250 seconds.
999900000999
So would you argue it's not worth it at the end of the day to switch to a faster language. I have a pretty quick solution in Go, but it's not scaling well.

Original solution was in Python, which is much slower.

bjourne
Yeah. Unless your Go code truly suck you won't see that much improvement.
sethammons
we've replaced production python and perl systems with Go. It is faster because the concurrency story and speed of Go is faster. On average, replacing services at scale (as in, billions of networked requests per day), we saw that Go reduced our fleet footprint by 10-20x. In the best case, we went from 70+ nodes running Twisted Python to 1 node running Go, so we bumped it to 3 nodes in case it failed.
999900000999
Was it hard to convince your team to switch to Go ?

What other languages did you consider?

Do you see a positive future for Golang devs?

Rust appears to of picked up a lot of its momentum. I vastly prefer Golang.

sethammons
We were building an smtp server and needed a good concurrency story. We tried for two weeks with gevent and got frustrated (monkey patching was misbehaving on low level network stuff). Someone on our team got the server mostly working over the weekend in Go with no prior Go experience. We made the case to the powers that be and were greenlit to finish the smtp server in Go. Go quickly became our default language at the company, undoing the recent decree that we were to be a Python shop.

There was some early talks about going to Java instead, but Go had too much momentum and folks didn't want AbstractFactoryFactory over-abstraction and craved the more simple Go approach having just experienced the pain of Python Twisted (which was aptly named).

As for the future of Go, all signs point to continued increases in adoption. I think it hits the sweet spot for backend, networked services given the languages I've tried. It is easier to learn than Rust, has a better concurrency story than Python and Perl while out performing them by an order of magnitude, it is easy to navigate unlike Ruby, and is simpler than what standard Java devs push for and uses less memory. And the deployment story is great; just a single executable.

999900000999
Thanks for the answer!

I built a small project in Go and I found it to be a very fun experience.

The syntax is very easy compared to Rust.

Any tips for finding a higher level (200k+) role with Go ?

logicalmonster
If the problem you're trying to solve is literally generating randomness, you might look into custom hardware. I think some companies have devices to measure subtle variations in the output of tiny laser beams, as one example of how pure randomness can be quickly and continuously found.
cpach
Is that really faster than the kernel’s CSPRNG?
[deleted]
(comment deleted)