Ask HN: How to understand the large codebase of an open-source project?

I usually start by running cloc and sloccount to get an idea of the metrics of it, languages line of code estimates etc...

I progress to looking at the tests if there are any. They usually give an idea of how the authors expect things to work.

Once I have browsed some of the tests, in particular integration tests I start following how they work through the code. Your IDE of choice will help out here or failing that use ripgrep, ack, the silver searcher, searchcode server (note I run this so I am biased), sourcegraph.

One thing that I have found especially valuable is running something to determine the cyclomatic complexity of the code. Knowing which parts are complex is a good way to determine where you should focus your time.

The analytic approach is a bit more awkward, since you have no specific goals and need to make them up yourself. So you could pose questions like how a specific behaviour of the application comes about ("why does it do that when this happens?", "how does it do X?") and then try to answer those comprehensively, systematically (a format that works well for me is short snippets of code interleaved with explanations and arguments).

A bottom-up approach is generally easier, because your questions will give you information at the bottom (like specific application messages), which are generally easy to find (ag, grep). A good IDE can be helpful for navigating the code and finding call sites, especially in projects written in dynamic languages where such analyses can become kinda annoying. (However, in more awkward code bases analysers like PyCharm are quickly overwhelmed and are unable to resolve indirections)

Top-down is in my experience less useful, because there are far too many choices on each level for most applications, and the first few layers are generally the least interesting and most arcane/fragile and difficult to follow along (things like initialization sequences).

The most difficult projects are typically those relying on multiple languages, code generation and runtime mutation (reflection, on-the-fly UI generation, overly dynamic Python code are typical examples). Another frequent obstacle is excessive abstraction and indirection (implementing something that could be done in a few lines of easy to understand and reason about C using multiple C++ templates spread out over a bunch of files and a healthy dozen of advanced language features is an almost archetypal example).

Then I try to go down the main code path of some examples or the primary binary if available and just check out out how things are called/done around there.

Then run an example through callgrind and visualize the call graph in kcachegrind to get an idea of how often things are called and where and where the heavy lifting happens. That last step is optional and really depends on the type of project.

Then I use my code editor and lots of searching and call site lookups to get a better idea of how things are used.

I enjoyed reading it many years ago.

1. Read some code

2. Try to understand how it works

3. Repeat

Here is what I do:

1. Try to figure out what the code might be in advance using the information you have. (For example: I know nothing but the fact that it's a spreadsheet. Then figure out in your mind how the basics of a spreadsheet might work.)

2. Now read a little bit of the code. Compare with what you were thinking. If it matches, go to 3. If it doesnt match, figure out why by reading the code and by thinking more.

3. Repeat

Note the two processes are relatively similar because step 2 of the former process is a little bit like step 1 of the later process. Just try to focus on figuring out first, read second. Figure out first, read second. It's an active approach which makes you work more, and the more you work, the faster you go - or some benefits of that sort.

I actually wonder if people do that.

Also use version control to identify the most commonly edited files in the project. These are usually the files that are doing all the work (80/20 rule) and you likely need to know of them.

git log --pretty=format: --name-only | sort | uniq -c | sort -rg | head -10

When you trace your own usage footsteps like this, it's often amazing how much goes on behind the scenes that you never realized.

- Count the lines of code with find | wc, get a sense for what's there, and what language it's written in. The biggest file in the project is usually worth a look -- it is often where the "meat" is. Read the function names.

- Use the program. grep for strings that appear in the UI in the source code. That's a good place to start reading. Read function names.

- strace the program. What system calls does it make when? ltrace is also sometimes useful, although it also gives a ton of output.

- Look at header files. Understanding data structures is often easier than understanding code.

- Look at commit logs. Those are hidden "comments". And reading diffs can be easier than reading code.

- Do a "log" or "blame" on the file. How has it evolved?

- Start reading main(). This often reveals something about the structure of the program. Even just finding main() in many programs is a good exercise :) Sometimes it's a little hard to find.

- Make sure to build it. And if you can, look at the build system. How is it put together? Most build systems are pretty darn unreadable. I don't really know how to read autoconf, and GNU make is tough too. Forget about cmake :) But sometimes this can help.

I haven't gotten that far with this, but I tried uftrace recently and like it:

https://github.com/namhyung/uftrace

You can think of it like a dtrace that knows about every function in a C or C++ program.

-----

I want to try some kind of code explorer thing. I saw this in a CppCon video and on HN:

https://www.sourcetrail.com/

And older ones like:

https://www.sourceinsight.com/

But somehow I get by with Unix tools. I think this is because I feel like building the project in a way to accomodate the source browsers might be a big pain.

Counterpoint: I think the hardest part of understanding a project is usually the build system :-) I don't have too much of a problem with reading C, C++, Python, or (sometimes) JS code. Volume is always a problem, but I can read a specific function pretty easily. But the build system is where things get ugly, in my experience.

Also, reading multi-threaded code requires some special consideration. grepping for every place that threads are started is a good idea.

* Read any developer contribution docs.

* Glean what info you can from the layout and naming of the source tree.

* Peruse the code and any comments and see what does what.

* Read the unit tests to see how things are expected to work.

* Peruse the issues list to see what's breaking.

* Try to get a feel for how the contributor(s) think by reading any public blog posts, etc.

If none of those approaches yield any insight, don't blame yourself; maybe instead look for a different OSS project to contribute to.

Use a static analyzer to build a graph of the codebase.

Build an adjacency list and a graph of the imports; and topologically + (…) sort.