I would think that once you start working with even half that many ips it starts to make sense to use a trie-like data structure that shares common pieces of the ips.
That doesn't help much if IP addresses are randomly distributed, as they are in a lot of our use cases. You use tries generally to do range and prefix lookups, not so much as a storage trick.
Also, we run on platforms like iOS, where we have very severe memory constraints (the NetworkExtension is allowed 15MiB of total RAM). Given that we juggle a lot of IP-related data types, even modest optimizations help.
That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
Mostly we optimize at the application layer (e.g. we do lazy on-demand WireGuard tunnel configuration, to minimize how much state sits around idle). We also have a small fork of Go with some space-optimizing changes: https://github.com/golang/go/compare/master...tailscale:tail... (most of these changes get upstreamed, albeit delayed by one release cycle).
Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off.
That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
Network code tends to be performance critical, so the more you can fit into a cache line the better. Exact numbers depend on the application, of course, but those memory overheads can give you a very serious performance hit in cache-heavy applications.
Size is a bonus though, the primary push seems to be all the other issues. Either way imho, needless bloat should be shorn regardless of the ability to accommodate it.
How the heck does one even turn an IPv4 address into 28-56 bytes?
Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator.
"It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to"
> The underlying type of a net.IP is just a []byte
> ...
> It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
Thanks, that's a bit interesting. I only know a little bit of Go, but I would have assumed you could just use an array ([4]byte) directly without a slice ([]byte) on top of it. Guess not.
It's less that you need to store many of them at the same time, but that it increases the amount of garbage collections. If you can create half as much garbage per packet or request, that's double the time until you hit the threshold for gc invocation.
No, the time isn't really what I'm referring to—ip addressing is complicated. But it shouldn't require having to work around the language to do this kind of thing.
I'm not sure how this is a "work around the language" rather then the standard library in Golang is allocation happy for IP types. I imagine this would look very similar in almost any language.
This type doesn't appear to support IPv6 Address Zones which was the most complicated part of the post. Actually from my limited googling Rust doesn't seem to support Address Zones at all in it's standard library.
It also doesn't include the ipv6 zone scope. Adding that would take it to at least 28 bytes, and still have to handle all the interning problems for cheap equality.
Interning is to reduce memory usage. I will agree the uintptr trick is a work around for the lack of weak refs in golang but the code isn't that different then what you would write if you have a referenced counted cache.
Year and a half of work. Impressive. I wonder if it’s actually worth it for a company to spend such amount of time on stuff like this. Nevertheless, I really appreciate it’s open source.
Cool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything.
On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
> Cool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything.
Why would it be better to use tailscale.com?
It has nothing to do with Tailscale. It's just an IP address.
This should point to AF_INET flag which you pass when creating a socket, and means that the address family is the internet (IP) address, as opposed to e.g. a domain socket, or some other transport.
Reading the description of the stdlib type, I can't help but be reminded of the statement that the go way of solving a problem is to ignore real-world problems and half-ass things for the sake of a simplicity that merely pushes the complexity elsewhere. It certainly does seem to apply here too. Glad this library exists now though.
That's a pretty lazy criticism given how much of the most modern and popular networking and system software is written in Go (Kubernetes, Docker, all the related infrastructure, etc).
The "net" standard library is still very nice and works extremely well.
The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
And Facebook was written in PHP. That hardly means that PHP doesn't have flaws or quirks or drawbacks.
And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them.
As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
I remember a time when Facebook would make a quite buggy appearance for a year or two. Must have been around 2015.
Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
> networking and system software is written in Go (Kubernetes, Docker, all the related infrastructure, etc).
Kubernetes and Docker are not networking. They orchestrate some networking components that do the actual network processing in the kernel (iptables, ebpf) or other high performance user space c/c++ (dpdk, etc).
The standard library IP type has worked for numerous programs for a literal decade. Just because a specialized application can do a better job by leveraging, again, decades of real world experience with the language doesn't invalidate that or mean it's half-assed.
I think your statement reflects more on your own biases than reality.
I am not saying that the type is unusable. Plenty of high profile software has been written with C strings too yet we almost universally recognize them today as a mistake.
What I'm saying is that the design flaws of net.IP, the absence of which would have benefited the whole ecosystem, fall into a familiar pattern that can be learned from. It is good to see that addressed, even if by third parties.
If you can look at any widely used API that has existed for an extended period of time and not find a single tradeoff or hindsight flaw, you aren't looking hard enough.
I think being able to compare or hash IP addresses is a pretty baseline feature for an IP address type. I don't think that's some kind of unpredictable hindsight or tradeoff.
I can't tell if your're being sarcastic but it's hard to find a program that uses UDP and doesn't at some point store IP addresses in a hashmap. Many more will do the same for per-IP rate limiting. I think those are reasonable use cases for go.
Either strings or (in the common case where I know I'm dealing with v4) uint32's. In fact, I probably do v6 stuff with uint32s too (I'm never working with an arbitrarily large v6 prefix).
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
I'd assume the lackluster selection of operations and how deeply nested some of the information is within the type. Definitely not the most ergonomic thing.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.
I think it's more likely that numerous programs have worked around the issues of the built in type for a decade and/or changed scope/modified functionality to avoid it.
I had to work around one of the issues here in my first use of the built in ip type - I needed to parse ip addresses and pass them to a command that needed a tcp4 or tcp6 flag depending on the address type.
I really wonder whether there is an executive at Google who has some connection to revenue for Google and how control of Golang ties to that. And, whether Brad leaving makes them really nervous. Google has never, AFAIK, invested in Jetbrains, but they care a lot about Android Studio and Kotlin. It's interesting to see this kind of post and wonder if this kind of work makes Google nervous at all. That would be a good book I'd like to read in 10 or 20 years.
Perhaps I missed something (and I don’t know Go at all), but if you’re keeping a background map in-sync for the pointer equality stuff to work out, doesn’t that imply it’s not actually 0-allocation? Or is this a go’ism where allocations refer to some subset of things?
It's a mild tradeoff the post glossed over: if you're handling IPv6 addresses with a zone specifier, the first use of a particular zone specifier will cause an allocation to intern that string.
Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
I don't quite understand why you need the messy/unsafe string interning, instead of a simple (mutex-protected) map with the m[string(b)] optimization that avoids allocations, or josharian's sync.Pool approach? https://commaok.xyz/post/intern-strings/
A simple map was discussed in the post. The problem it has is that it has unbounded growth. The sync.Pool approach is a best effort to avoid allocations only. If its API was changed to return sentinel pointers rather than the string, it wouldn't have the guarantee that Intern(x) == Intern(x) when x == x.
Thanks. I re-read this paragraph from the original article and it makes more sense now:
> use a zone mapping table, mapping between zone index integers and zone name strings. That’s what the Go standard library does internally. But then we’re left susceptible to an attack where an adversary forces us to parse a bunch of IP addresses with scopes and we forever a bloat a mapping table that we don’t have a good opportunity to ever shrink. The Go standard library doesn’t need to deal with this, as it only ever maps interfaces that exist on the machine and doesn’t expose the integers to users in representations; its net.IPAddr.Zone field is a string.
The uintptr / SetFinalize trick to implement "weak" references is pretty cool, but I'd be curious to hear why
type IP struct {
hi, lo uint64
z string
}
with sentinel strings like "\04" and "\0\06" for IPv4 and IPv6-no-zone was rejected. Sure, this 8 bytes larger for the implicit string length, but its still not terribly large, and the string comparisons in the normal IPv4 / IPv6 case are very cheap (just length and pointer equality checks). And it doesn't require opening the unsafe jar. Was the extra 8 bytes enough to disqualify it? Or are the authors worried about the fact that this may allocate on each parse of an IPv6-with-zone?
Yeah, that would be the obvious representation if we were willing to go to 32 bytes. (on 64-bit)
But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are.
And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that.
But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
A scoped IPv6 address contains a network interface name, which is used to disambiguate which network a link-local address belongs to.
It’s relatively common to encounter them in some network configurations where the routers use link-local addresses rather than global addresses in their RA packets. They are also used in SLAAC, because a device can configure a link-local address without knowing anything about its environment.
[ I think in the past there might have been an idea to use them to disambiguate site-local IPv6 addresses, but network interfaces are not sufficient in that case, and site-local addresses were problematic in lots of other ways, so they have been dropped from the specifications. ]
> Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake.
It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it.
(There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
I know nothing about Go or IP, but i do know Erlang where atoms (= interned strings) are a little bit of a security risk because it makes it very easy for an attacker to cause out-of-memory by flooding an endpoint that converts input strings to atoms with random data.
Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
> Basically, it’s playing unsafe games behind the Go garbage collector’s back, hiding pointers in untyped uintptr integers so Go will be forced to eventually garbage collect things which then causes the finalizer to be invoked to step in and either clean up its lies or clean up the map.
I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest.
Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
I don't understand why SetFinalizer+resurrected could prevent a memory block which will be collected from being collected.
Update: never mind, I re-read the docs of SetFinalizer to get the explanation:
SetFinalizer sets the finalizer associated with obj to the provided
finalizer function. When the garbage collector finds an unreachable block
with an associated finalizer, it clears the association and runs
finalizer(obj) in a separate goroutine. This makes obj reachable again,
but now without an associated finalizer. Assuming that SetFinalizer
is not called again, the next time the garbage collector sees
that obj is unreachable, it will free obj.
101 comments
[ 3.5 ms ] story [ 160 ms ] threadthat's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?
... Tailscale...
You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off.
That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator.
UTF-16, perhaps? LOL
It's covered in the blog post.
> The underlying type of a net.IP is just a []byte
> ...
> It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
[0]: https://doc.rust-lang.org/std/net/enum.IpAddr.html
https://news.ycombinator.com/newsguidelines.html
On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
Why would it be better to use tailscale.com?
It has nothing to do with Tailscale. It's just an IP address.
A neutral package name seemed best.
The former looks like the name of a Go type (which is what it is) and the latter looks like a domain name or something.
I'll even give you guys leading lowercase, which breaks the usual HN title conventions. Don't let it go to your head.
The "net" standard library is still very nice and works extremely well.
The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them.
As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
Kubernetes and Docker are not networking. They orchestrate some networking components that do the actual network processing in the kernel (iptables, ebpf) or other high performance user space c/c++ (dpdk, etc).
I think your statement reflects more on your own biases than reality.
What I'm saying is that the design flaws of net.IP, the absence of which would have benefited the whole ecosystem, fall into a familiar pattern that can be learned from. It is good to see that addressed, even if by third parties.
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.
It wouldn't surprise me one bit if this ended up in the stdlib for Go 2.0.
I had to work around one of the issues here in my first use of the built in ip type - I needed to parse ip addresses and pass them to a command that needed a tcp4 or tcp6 flag depending on the address type.
Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
I could clarify in the post.
> use a zone mapping table, mapping between zone index integers and zone name strings. That’s what the Go standard library does internally. But then we’re left susceptible to an attack where an adversary forces us to parse a bunch of IP addresses with scopes and we forever a bloat a mapping table that we don’t have a good opportunity to ever shrink. The Go standard library doesn’t need to deal with this, as it only ever maps interfaces that exist on the machine and doesn’t expose the integers to users in representations; its net.IPAddr.Zone field is a string.
But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are.
And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that.
But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
In either case, having the interning wrapped up in a reusable library is great.
Who uses IPv6+zones and why?
I'm confused what zones are about. They don't seem to be part of IPv6 packets. Where do they come from? Why does IP type have to deal with them?
Still unclear why and where do zones come from.
https://en.wikipedia.org/wiki/IPv6_address#Scoped_literal_IP...
Apparently it's possible to embed interface name into an address:
fe80::1ff:fe23:4567:890a%eth2
The article also points to an alternative solution - embed interface index into an address:
fe80:3::1ff:fe23:4567:890a
It’s relatively common to encounter them in some network configurations where the routers use link-local addresses rather than global addresses in their RA packets. They are also used in SLAAC, because a device can configure a link-local address without knowing anything about its environment.
[ I think in the past there might have been an idea to use them to disambiguate site-local IPv6 addresses, but network interfaces are not sufficient in that case, and site-local addresses were problematic in lots of other ways, so they have been dropped from the specifications. ]
[0] https://github.com/golang/go/issues/43615
The programmer's mantra.
It's in a PR here: https://github.com/WireGuard/wireguard-go/pull/11/files
Which mentions:
> Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake.
It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it.
[0] https://github.com/OJFord/terraform-provider-wireguard/blob/...
(There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest.
Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
Update: never mind, I re-read the docs of SetFinalizer to get the explanation:
Or how F-Secure is shipping secure USB keys with bare metal Go firmware.