Nice writeup. Especially interesting to see that CloudFlare is using Go for its DNS service. I run a DNS hosting service (https://www.SlickDNS.com) and currently use tinydns from the djbdns suite in my name servers, but I'm writing a drop-in replacement in Go. The goal is to make it much higher performance by using multiple goroutines (tinydns is a single process).
Have you done any analysis to see if tinydns was actually a bottleneck? The work of an authoritative DNS server is extraordinarily simple, and tinydns capitalizes on that.
Under normal load tinydns' performance is more than adequate, but under DDoS attack the fact that it's a single process is a definite limitation when it's running on a multi-core machine.
By default tinydns opens its database file for every request which can lead to I/O errors under very heavy load. I've patched it (http://tinydns.org/one-second.patch) to only open the database every second, but with my Go rewrite I can easily keep the database in memory and reload it when the disk file changes.
You meant “could theoretically be” – you're gambling that you'll have development resources and expertise to make optimizations which aren't possible in Go, that those optimizations will actually matter to the critical path, and that you're more likely to reach the point of being able to make those optimizations than the Go team will be able to optimize further.
In the absence of something showing a strong, unavoidable disadvantage for Go, that's a lot to take on faith.
CloudFlare is awesome. They're living out one of my fantasies of productizing really well done front-end web serving infrastructure, which I love to see. They appear to have gathered solid team. And they're doing it with Go!
I'd love to see those libraries (sans proprietary modules) show up on GitHub!
I'm in NYC this week if you want to grab coffee. Always interested in talking to smart engineers interested in GoLang. Email: matthewatcloudflaredotcom cc: jennatcloudflaredotcom and we'll see if we can set something up.
"The guarantees needed to avoid leaving the server in a bad state when handling panics would be impossible without the defer mechanism Go provides."
I'm only passingly familiar with defer, but I understand it to be equivalent to RAII in C++, Python's with statement, Common Lisp's unwind-protect, and others - does is actually provide something more, and if so, what?
It's not at all equivalent to RAII, or really any of those other examples; it's a way of clearing up the control flow of a function that needs cleanup work before it returns, but it is a fussy and error-prone way of expressing scoped resource access.
"Defer" is nice to have, and because it does less than scoped acquisition, it's easier to repurpose for other jobs; that's kind of thematic of Golang --- simple, orthogonal advances over C/Java/C++; a distinct lack of "theoretical" ambition.
Go's "defer" is not equivalent to RAII. It is function-scoped rather than block-scoped and has semantics based on mutating hidden per-function mutable state at runtime. For example:
func Foo() {
for i := 0; i < 5; i++ {
if Something() {
defer Whatever()
}
}
// ... the compiler can't tell how many
// Whatever()s run here ...
}
Compared to RAII as implemented in for example D with its "scope" statement, "defer" has much more complex semantics, inhibits refactoring since moving things to function bodies or inlining function bodies silently changes semantics, and cannot be optimized as easily, because of the dynamic aspects. IMHO, it has essentially no advantages over RAII and many disadvantages.
> defer inhibits refactoring since moving things to function bodies or inlining function bodies silently changes semantics, and cannot be optimized as easily, because of the dynamic aspects.
As someone who has written and reviewed hundreds of thousands of lines of Go code, I haven't observed this to be the case in practice.
RAII doesn't fit into Go, philosophically, as it lets you trigger hidden functionality on the creation or destruction of data structures, whereas a deferred function can only be run if there's a defer statement there in the code (where you can see it).
In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures. This brings a huge benefit in terms of readability and transparency.
Anyway, I'm not sure why we're comparing defer and RAII, because they're generally used for different purposes.
> As someone who has written and reviewed hundreds of thousands of lines of Go code, I haven't observed this to be the case in practice.
Sure, a lot of suboptimal design decisions don't cause problems in practice. That doesn't change the fact that they're suboptimal, and in this case lead to worse performance.
> RAII doesn't fit into Go, philosophically, as it lets you trigger hidden functionality on the creation or destruction of data structures, whereas a deferred function can only be run if there's a defer statement there in the code (where you can see it).
I'm focusing more on RAII as implemented with "scope" in D; whether stuff runs explicitly or implicitly is an orthogonal design choice (although I prefer implicitly running code since you need finalizers anyway in any GC'd language, including Go—so you might as well embrace it). With the "scope" statement you also always have to explicitly call the destructor, but in a lexically scoped way.
The main thing I find suboptimal with "defer" is the choice of dynamic mutable state as compared to lexical scoping.
> In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures. This brings a huge benefit in terms of readability and transparency.
This appears to be a helper function used exclusively by the standard library to handle file descriptor closing (incidentally, the one issue I've had with Golang's concurrency model).
> This appears to be a helper function used exclusively by the standard library to handle file descriptor closing (incidentally, the one issue I've had with Golang's concurrency model).
But it's part of the public API. You can add a finalizer to any object. The semantics of Go say that finalizers are run automatically when the GC reclaims an object. So this statement is wrong: "In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures." It would be more correct to say "idiomatically, in Go people tend to prefer calling functions explicitly, and 'defer' encourages this."
I think the fact that it's used by the standard library to close file descriptors is actually really illustrative: you need finalizers in a GC'd language, otherwise you'll leak resources. Not all resources are stack-scoped. So implicitly running functions on deallocation is a necessary evil. You might as well embrace it in your language design.
It may be part of the "public API" solely because it needs to be made available to several different components of the standard library, which is itself at pains to implement itself primarily in Golang.
I don't see why it's relevant that the standard library as opposed to user code needs it. The standard library is a library like any other. It needs finalization functionality because you always need that functionality in a GC'd language.
File descriptors are just one case of resources that need finalization functionality to not leak: the same applies to GPU textures, X server resources, etc. etc.
You don't need finalization functionality in a GC'd language. If you imagine a language that lacks finalization functionality but has automatic memory reclamation, things turn out okay. Finalization functionality isn't something that sane programs depend on -- garbage collection of memory makes sense because if you run out of memory or allocate and remove pointers a lot of stuff, the garbage collector can naturally kick in and find you some more memory to use. If you allocate a bunch of file handles, does the garbage collector kick in when your OS tells you that you've run out of file descriptors?
> You don't need finalization functionality in a GC'd language. If you imagine a language that lacks finalization functionality but has automatic memory reclamation, things turn out okay.
Not in fault-tolerant message passing systems, to name just one obvious example. Suppose that you put a bunch of file objects into a buffered channel, and then the goroutine that was supposed to receive those objects panics. Your program wants to recover from panics with recover(). Who closes the file descriptors in those channels? Nobody owns them yet: they were in a channel and the goroutine that was supposed to receive them died.
You might be able to solve this by handing out references to the channel to another goroutine that is supposed to clean up the file descriptors, but this gets really complicated. This sort of thing is why Go is GC'd in the first place. It's much easier to just have the GC clean up the file descriptors in channels in which one endpoint has gone dead, and that's the sort of thing finalizers are for and I assume it's the reason that finalizers were built into Go.
one option is for the goroutine to defer a cleanup closure that closes every file in that channel. Panics will cause all deferreds to be called all the way up to the deferred which recover()s.
Another option is to crash the app instead of excessive recover()ing. (obviously there are good reasons to use runtime error handling, but imho fewer than what one would think).
Finalization is not needed but it is nice to have. For example releasing shared IPC resources.
> Finalization functionality isn't something that sane programs depend on
One definitely wouldn't want to use it _if they don't have to_, but sometimes there is no choice. Finalization is one of those things one does reluctantly because they end up having to use shared memory for example. Or standard library want to help you not leak file descriptors.
> Garbage collection of memory makes sense because if you run out of memory or allocate and remove pointers a lot of stuff, the garbage collector can naturally kick in and find you some more memory to use.
Well collection of anything unused and limited probably makes sense because otherwise you'd run out of them eventually. Garbage collection doesn't magically add RAM sticks to the machine though. If you've used all the memory and still hold references to all the objects, (in a GCed language), there is nothing GC can do. [Well I guess you can have weak refs like in Python...].
> If you allocate a bunch of file handles, does the garbage collector kick in when your OS tells you that you've run out of file descriptors?
Well in a high level language that has GC you'd expect to also probably deal with File _objects_ not file _descriptors_. In that case I would expect those _File_ objects when and if they are GC-ed to also close their file descriptor appropriately. That is where being able to have finalizers helps. Because a finalizer of a File object would close the file descriptor.
I only saw this implemented in OS based on Native Oberon, where everything on the OS was GC enabled, from file handles to GUI widgets.
If your applications needs to communicate with the outside world in a OS implemented in a systems language without GC support, then the GC needs a little help to give back those resources back to the OS.
Golang is perfect for DNS: its concurrency model matches that of DNS extremely well, its standard library is heavily informed both by Unix and by Internet protocols, and its fast and simple. That's unsurprising, since Golang's charter was to modernize C/C++, and DNS is essentially an expression of 1980's C code style.
Though handling DNS requests over UDP is pretty easy and fast in almost any language. TCP connections are much more pleasant to handle by spawning a goroutine per connection. I do assume using goroutines is slower than multiplexing though, but still plenty fast.
I do assume using goroutines is slower than multiplexing though, but still plenty fast.
Surely, the goroutine model is just an abstraction built on multiplexing? As in, it uses the exact same syscalls that a multiplexing server written in C would use, except instead of storing state explicitly in a connection structure, it's stored implicitly in the goroutine's stack. The main (only?) real difference is code structure and thus the dispatch mechanism to where in the connection handling code processing should resume: In the goroutine, you're literally storing the instruction pointer and jumping straight to it. In the hand-written case, you'll probably go through some buffering code and an explicit state machine.
Basically, Go and its goroutines save you reinventing the connection state capture and resuming code for every single protocol you implement - it comes for free as part of the language. (you can make C do similar things with makecontext/swapcontext plus some extra plumbing code of course) Performance wise it should be a pretty close call: resuming a goroutine will presumably cause a branch prediction miss (due to the change in function return pointer), whereas the multiplexed code will need to wind its way through the state machine logic - almost certainly also a branch prediction miss. CPU cache behaviour should likewise be similar (you're storing the same state either way).
> Performance wise it should be a pretty close call: resuming a goroutine will presumably cause a branch prediction miss (due to the change in function return pointer),
Also you have to reload the register file when the goroutine scheduler wakes the goroutine up, and save it when the goroutine goes back to sleep.
This is true, but I'd be very surprised if you could beat the memory usage and performance of a while loop with epoll using goroutines. At least in the role of a DNS server. Now I want to test this...
A language perfect for almost every kind of server would not allow a single connection handler disrupt every other connection handler. A single goroutine can corrupt the heap shared by every other goroutine.
I'm not an expert, so take this speculation with a grain of salt, but I'd guess this is how it encrypts something in such a way as to require two people:
1. Generate a key, and use it to encrypt and sign your payload. Nothing fancy here; just plain old symmetric encryption and authentication.
2. Use Shamir's Secret Sharing to split the key into pieces. You need any two pieces to reconstruct the key. This is where the magic happens:
Thanks for the link to SSS. I'd never seen it before, and it's a really, really cool algorithm. One of those things that as soon as you see it you think "of course!"
We're building something similar for https://harrow.io/ which acts as the secret store, it's a Redis API compatible database with a strong auditing layer, and baked in crypto, it requires a single key to start and unlock the server, but we didn't know how to split the key to keep the secrets out of a single dev's hands. Now that we do, we'll build something based on the SSS algorithm and YubiKeys (which we use for all other tokens)
For millions of secrets and tens of people holding those secrets, there's a much more space-efficient solution. There's an added benefit that hiding each secret is plain vanilla RSA encryption.
Victor Shoup discovered a relatively straight forward way to take an RSA private key and apply Shamir secret sharing in a way that commutes with RSA public key operations. Just like Shamir secret sharing, each of the N shares of the secret is a point on a random polynomial on a finite ring (in this case, the set of quadratic residues modulo the key's public exponent). Each of the N secret holders, using their secret share, can generate a public share. Any T of the N public shares can be used to invert the RSA public key operation.
Shoup's threshold RSA signature scheme (http://www.shoup.net/papers/thsig.pdf) when used for RSA encryption caries all of the caveats that using the raw RSA algorithm entails (use OAEP, etc.)
I'm really surprised Shoup-Cramer encryption is more famous than Shoup's threshold RSA algorithm.
The "seamless binary upgrade" part is intriguing (unless it's just a reference to the fact that binaries are self-contained (which is indeed wonderful)).
We use a supervising process [1] that has bound the sockets and the pass in the file descriptors to Go process. When the new process is ready to take connections it sense a signal to the original process to shut down.
I do part time IT and I have repeatedly wanted a more programatically friendly DNS and DHCP solution for our small network. dnsmasq doesn't have a great high-availability story, and BIND+ISCDHCP can be quite complicated to configure for high-availability (plus BIND doesn't seem to have convenient options such as expand-hosts from dnsmasq).
Does this mean Go is ready to use to make a 'simple' forwarding/caching DNS server with the ability to have DHCP update local hostnames? How about DNSSEC validation?
I wonder how Lua fits into the picture at CloudFlare with this Go infrastructure, since CloudFlare is also a heavy user of Lua [1] as well and key financial sponsor of LuaJIT [2].
Every request that passes through CloudFlare goes through Lua. It has how we look up and apply a customer rules to a recess. We just recently realized a non-blocking logger option for Nginx [2] using Lua and are in the process of developing a new KT library [3] in Lua.
I would say that C, Lua, and Go are our three primary systems languages. We also use PHP for some of our frontend UI. Dabble in Python in places it makes sense, but usually not in the critical path.
of course it does, but it does it in a way that aims to provide an overall good performance for all apps. But it doesn't mean that's the best performance you can get.
C doesn't control memory. brk() does. There are various malloc libraries out there to improve the way malloc manages memory.
But ideally you don't want to allocate memory at all. You want to have memory preallocated and preferably accessing a page that's still in cache. You can control memory pages in C because it doesn't manage that for you.
64 comments
[ 3.1 ms ] story [ 58.2 ms ] threadBy default tinydns opens its database file for every request which can lead to I/O errors under very heavy load. I've patched it (http://tinydns.org/one-second.patch) to only open the database every second, but with my Go rewrite I can easily keep the database in memory and reload it when the disk file changes.
https://github.com/halayli/lthread
You meant “could theoretically be” – you're gambling that you'll have development resources and expertise to make optimizations which aren't possible in Go, that those optimizations will actually matter to the critical path, and that you're more likely to reach the point of being able to make those optimizations than the Go team will be able to optimize further.
In the absence of something showing a strong, unavoidable disadvantage for Go, that's a lot to take on faith.
I'd love to see those libraries (sans proprietary modules) show up on GitHub!
Seconded, I'll throw my website's name into the hat of "websites saved by cloudflare."
I was really skeptical of them at first but I'm really glad I decided to give them a try.
I'm only passingly familiar with defer, but I understand it to be equivalent to RAII in C++, Python's with statement, Common Lisp's unwind-protect, and others - does is actually provide something more, and if so, what?
"Defer" is nice to have, and because it does less than scoped acquisition, it's easier to repurpose for other jobs; that's kind of thematic of Golang --- simple, orthogonal advances over C/Java/C++; a distinct lack of "theoretical" ambition.
As someone who has written and reviewed hundreds of thousands of lines of Go code, I haven't observed this to be the case in practice.
RAII doesn't fit into Go, philosophically, as it lets you trigger hidden functionality on the creation or destruction of data structures, whereas a deferred function can only be run if there's a defer statement there in the code (where you can see it).
In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures. This brings a huge benefit in terms of readability and transparency.
Anyway, I'm not sure why we're comparing defer and RAII, because they're generally used for different purposes.
Sure, a lot of suboptimal design decisions don't cause problems in practice. That doesn't change the fact that they're suboptimal, and in this case lead to worse performance.
> RAII doesn't fit into Go, philosophically, as it lets you trigger hidden functionality on the creation or destruction of data structures, whereas a deferred function can only be run if there's a defer statement there in the code (where you can see it).
I'm focusing more on RAII as implemented with "scope" in D; whether stuff runs explicitly or implicitly is an orthogonal design choice (although I prefer implicitly running code since you need finalizers anyway in any GC'd language, including Go—so you might as well embrace it). With the "scope" statement you also always have to explicitly call the destructor, but in a lexically scoped way.
The main thing I find suboptimal with "defer" is the choice of dynamic mutable state as compared to lexical scoping.
> In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures. This brings a huge benefit in terms of readability and transparency.
http://golang.org/pkg/runtime/#SetFinalizer
But it's part of the public API. You can add a finalizer to any object. The semantics of Go say that finalizers are run automatically when the GC reclaims an object. So this statement is wrong: "In Go, the only way to execute a block of code is to make a function call. There are no constructors, destructors, or any other kind of side effect to allocating or deallocating data structures." It would be more correct to say "idiomatically, in Go people tend to prefer calling functions explicitly, and 'defer' encourages this."
I think the fact that it's used by the standard library to close file descriptors is actually really illustrative: you need finalizers in a GC'd language, otherwise you'll leak resources. Not all resources are stack-scoped. So implicitly running functions on deallocation is a necessary evil. You might as well embrace it in your language design.
SetFinalizer feels like a low blow, here.
File descriptors are just one case of resources that need finalization functionality to not leak: the same applies to GPU textures, X server resources, etc. etc.
Not in fault-tolerant message passing systems, to name just one obvious example. Suppose that you put a bunch of file objects into a buffered channel, and then the goroutine that was supposed to receive those objects panics. Your program wants to recover from panics with recover(). Who closes the file descriptors in those channels? Nobody owns them yet: they were in a channel and the goroutine that was supposed to receive them died.
You might be able to solve this by handing out references to the channel to another goroutine that is supposed to clean up the file descriptors, but this gets really complicated. This sort of thing is why Go is GC'd in the first place. It's much easier to just have the GC clean up the file descriptors in channels in which one endpoint has gone dead, and that's the sort of thing finalizers are for and I assume it's the reason that finalizers were built into Go.
How can the goroutine clean them if it is dead?
Another option is to crash the app instead of excessive recover()ing. (obviously there are good reasons to use runtime error handling, but imho fewer than what one would think).
> Finalization functionality isn't something that sane programs depend on
One definitely wouldn't want to use it _if they don't have to_, but sometimes there is no choice. Finalization is one of those things one does reluctantly because they end up having to use shared memory for example. Or standard library want to help you not leak file descriptors.
> Garbage collection of memory makes sense because if you run out of memory or allocate and remove pointers a lot of stuff, the garbage collector can naturally kick in and find you some more memory to use.
Well collection of anything unused and limited probably makes sense because otherwise you'd run out of them eventually. Garbage collection doesn't magically add RAM sticks to the machine though. If you've used all the memory and still hold references to all the objects, (in a GCed language), there is nothing GC can do. [Well I guess you can have weak refs like in Python...].
> If you allocate a bunch of file handles, does the garbage collector kick in when your OS tells you that you've run out of file descriptors?
Well in a high level language that has GC you'd expect to also probably deal with File _objects_ not file _descriptors_. In that case I would expect those _File_ objects when and if they are GC-ed to also close their file descriptor appropriately. That is where being able to have finalizers helps. Because a finalizer of a File object would close the file descriptor.
If your applications needs to communicate with the outside world in a OS implemented in a systems language without GC support, then the GC needs a little help to give back those resources back to the OS.
Though handling DNS requests over UDP is pretty easy and fast in almost any language. TCP connections are much more pleasant to handle by spawning a goroutine per connection. I do assume using goroutines is slower than multiplexing though, but still plenty fast.
Surely, the goroutine model is just an abstraction built on multiplexing? As in, it uses the exact same syscalls that a multiplexing server written in C would use, except instead of storing state explicitly in a connection structure, it's stored implicitly in the goroutine's stack. The main (only?) real difference is code structure and thus the dispatch mechanism to where in the connection handling code processing should resume: In the goroutine, you're literally storing the instruction pointer and jumping straight to it. In the hand-written case, you'll probably go through some buffering code and an explicit state machine.
Basically, Go and its goroutines save you reinventing the connection state capture and resuming code for every single protocol you implement - it comes for free as part of the language. (you can make C do similar things with makecontext/swapcontext plus some extra plumbing code of course) Performance wise it should be a pretty close call: resuming a goroutine will presumably cause a branch prediction miss (due to the change in function return pointer), whereas the multiplexed code will need to wind its way through the state machine logic - almost certainly also a branch prediction miss. CPU cache behaviour should likewise be similar (you're storing the same state either way).
Also you have to reload the register file when the goroutine scheduler wakes the goroutine up, and save it when the goroutine goes back to sleep.
1. Generate a key, and use it to encrypt and sign your payload. Nothing fancy here; just plain old symmetric encryption and authentication.
2. Use Shamir's Secret Sharing to split the key into pieces. You need any two pieces to reconstruct the key. This is where the magic happens:
http://en.wikipedia.org/wiki/Shamir's_Secret_Sharing
3. Encrypt-and-sign each piece with a secret key derived from its owner's password/passphrase using a secure KDF like scrypt.
4. Throw away the keys, and put those encrypted pieces on disk.
Now you need passwords from any two people to decrypt the secret payload. Cool, right?
Victor Shoup discovered a relatively straight forward way to take an RSA private key and apply Shamir secret sharing in a way that commutes with RSA public key operations. Just like Shamir secret sharing, each of the N shares of the secret is a point on a random polynomial on a finite ring (in this case, the set of quadratic residues modulo the key's public exponent). Each of the N secret holders, using their secret share, can generate a public share. Any T of the N public shares can be used to invert the RSA public key operation.
Shoup's threshold RSA signature scheme (http://www.shoup.net/papers/thsig.pdf) when used for RSA encryption caries all of the caveats that using the raw RSA algorithm entails (use OAEP, etc.)
I'm really surprised Shoup-Cramer encryption is more famous than Shoup's threshold RSA algorithm.
[1] https://github.com/cloudflare/circus/commits/cloudflare2
Does this mean Go is ready to use to make a 'simple' forwarding/caching DNS server with the ability to have DHCP update local hostnames? How about DNSSEC validation?
[1] http://blog.cloudflare.com/pushing-nginx-to-its-limit-with-l...
[2] http://luajit.org/sponsors.html#sponsorship_perf
Here are the most recent lua libraries we have open sourced... [1] https://github.com/agentzh/lua-resty-lock [2] http://github.com/cloudflare/lua-resty-logger-socket [3] http://github.com/cloudflare/lua-resty-kyototycoon
But ideally you don't want to allocate memory at all. You want to have memory preallocated and preferably accessing a page that's still in cache. You can control memory pages in C because it doesn't manage that for you.