HN Explorer

Ask HN: Good Go codebases to read?

167 points by ra7 ↗ HN
I'm a Python programmer and I just started learning Go. What are some good Go codebases to read? I feel it helps me learn better if I look at some existing high quality open source code.

There was a similar post about Python a couple of years ago (https://news.ycombinator.com/item?id=9896369) and it helped me tremendously. I'm looking for something similar with Go.

Thanks!

52 comments

[ 2.8 ms ] story [ 87.3 ms ] thread
sagichmal
HashiCorp put out good stuff. CoreOS can be hit or miss; Torus was nice, etcd is a wreck. I like https://github.com/go-kit/kit and https://github.com/oklog/oklog. Definitely avoid Kubernetes and Docker -- they're Go by committee, look and feel like transliterated Java.
zalmoxes
Seconding the Hashicorp projects. I learned a lot about HTTP from reading the Vault code.

go-kit and oklog are also fantastic. Peter Bourgon has written a good article about Go best practices and his open source projects really aim for clean, readable code.

Also join the go slack if you haven't yet.

amouat
I know very little Go, but I have seen the Kubernetes code and my impression was "I bet these people were Java programmers in a past-life". You could tell by the multiple layers of abstraction, a lot of which felt unnecessary.

Having said that, Kubernetes is a solid piece of engineering, so they are clearly doing something right and I don't want to denigrate their hard work.

cyphar
Kubernetes also came from the same engineers that worked on Borg (Google's internal Kubernetes), which is written in C++. I imagine a lot of the constructs and design/layers were inspired by their experience in Borg.
smarterclayton
Some of the style adopted for Kubernetes mirrored both previous Java experience (inside Google and out), but most of it was simply observation of other large Go codebases.

I don't think much of the layering / structure has anything to do with Borg except in a few areas:

1. Scheduler predicates

2. The massive ball of yarn that the kubelet started off as and was unable to achieve escape gravity on (until CRI landed)

Everything else was focused on pure efficiency and is only slightly biased by general experience building large software projects (pass dependencies down, avoid complicated abstractions, rigorous style enforcement). A lot of that leaked as Kube grew into a larger and larger community and more people contributed.

The other abstractions people complain about mirror traditionally hard problems (API version stability with conversion across semantic changes, general API surface area management). Other than that we decided not to build on relational stores but a partitioned document model (etcd, inspired by chubby, but not fundamentally different from any other object store), I'd say the code is mostly just a reflection of being good enough to maintain now and refactor later. Pretty code is easy when it's a few people writing it. Harder when it's a team of tens / hundreds

amouat
> Pretty code is easy when it's a few people writing it. Harder when it's a team of tens / hundreds

Yeah, this is a good point that I was overlooking. It's easy for me to be critical when I feel the pain digging through multiple levels of abstraction to find out how something works, but I didn't the feel the greater pain of trying to manage tens or hundreds of developers working on the same codebase.

rileytg
i second the k8s and docker comments. took me a long time to realize this isn't how i should be doing it.
barakm
I'm flattered. If you like the Torus style, https://github.com/coreos/go-tcmu is a package I'm quite proud of (SCSI interactions wrapped in a nice pure-Go interface) and then a plug for https://github.com/cayleygraph/cayley which is a little rougher but a nice community and large codebase I maintain.

Bolt (also here mentioned) is one of my favorite beautiful codebases. https://github.com/mdlayher is a prolific author with lots of good protocol implementations. https://github.com/chihaya/chihaya is a fun project if you're into BitTorrent.

mmgutz
Standard lib source is good. I gained a lot of insight from Dave Cheney's blog: https://dave.cheney.net.
trey-jones
This would have been my suggestion. Read the standard library. It's mostly easy to understand and well documented. In general if you have a question of how a problem should be solved, this is a good place to refer.
ereyes01
+1 on the standard lib. It's done so nicely in most places, that it's easy to convince myself to take a minute and read implementations of routines I'm reading godocs for, even though the docs are 100% clear... cuz it's actually a pleasant read.

In contrast, I've worked in places where I needed a drink or 3 before I can build up the gumption to open my text editor :-)

victorquinn
Camlistore is great: https://github.com/camlistore/camlistore

Many of what is now the Go standard lib came from Brad Fitzpatrick's work on Camlistore.

cyphar
I'm not sure I agree. A lot of camlistore's idioms don't feel like "normal", idiomatic Go. Since it was still being written while the standard library and language was evolving, I get the feeling when reading the code that it's more like doing archeology of the Go language.
philsnow
IIRC it also still has a non-standard build process (go run make.go) for the same reason (it was being written while the library/language were in their infancy).
tylerchr
I really enjoyed studying the BoltDB source[1], which is Ben Johnson's popular key/value store library. I found it to be clear, idiomatically written, and well annotated.

[1]: https://github.com/boltdb/bolt

beliu
Humble suggestion / shameless plug: try using Sourcegraph to read through some of the excellent codebases mentioned in other posts. This is basically our prime use case (I'm one of the creators): reading through and understanding code structure with jump-to-def / find-refs / symbol search in your browser. I'd suggest starting with the main function and tracing through the static structure of the codebase from there:

https://sourcegraph.com/github.com/golang/go/-/blob/src/net/... https://sourcegraph.com/github.com/mholt/caddy@master/-/blob... https://sourcegraph.com/github.com/btcsuite/btcd@master/-/bl... https://sourcegraph.com/github.com/camlistore/camlistore@mas... https://sourcegraph.com/github.com/boltdb/bolt@master/-/blob... https://sourcegraph.com/github.com/go-kit/kit/-/blob/example...

dorianm
I was trying to understand the go source code the other day, my process was:

`git clone https://github.com/golang/go` then `cd go`, and `tree .`, `git grep ...`, `vim ...`, etc.

Anyway, couldn't find how to get the tests to pass, but found this:

https://golang.org/doc/codewalk/markov/

And a little explanation: https://golang.org/doc/codewalk/codewalk/

Also: https://golang.org/doc/install/source

In terms of codebase, kubernetes and upspin seem pretty good too:

https://github.com/kubernetes/kubernetes https://github.com/upspin/upspin

nijaru
I imagine upspin is one of the best examples because Rob Pike is one of the main contributors
dorianm
Funny thing is that I couldn't get up spin to run :).

But yes having a global namespace is a genius idea.

swordbeta
Thanks for sharing! Just wanted to let you know it doesn't work on firefox android.
sargun
Do you have any plan on supporting C?
deepanchor
I know Go is mainly used for systems and infrastructure level software, but I also know that some companies are using it at a higher level of abstraction for their application servers as well.

I'd really like to see more examples of these types of codebases but can't seem to find examples anywhere.

ryancouto
Standard lib 100%. And Dave Cheney's personal website/blog posts. You don't need anything else.
themihai
The standard library is quite good and idiomatic. I particularly like net/* and io)