HN Explorer

Ask HN: Recommended resources to learn the Linux kernel and OS theory?

408 points by non-entity ↗ HN
So recently I did a couple of minor patches on the FreeBSD and NetBSD kernels and played with some linux kernel. It was the first time in a a few years, I've been excited about programming.

Unfortunately I'm still completely lost. It seems that there's so much to learn and every kernel update breaks whatever you just finished writing. I see terms in unfamiliar with and often find myself googling basic OS concepts.

Are there any recommendations reading materials to get a better grasp on OS theory and / or the Linux / UNIX kernels and programming for them?

83 comments

[ 3.4 ms ] story [ 174 ms ] thread
yourbandsucks
The O'Reilly book Understanding The Linux Kernel is fantastic. Starts you off in the deep end with memory management/addressing and fans out from there. Notably excludes networking, as the book is long enough without it.
DanAtC
Any suggestions for Linux networking? I have Understanding Linux Network Internals but it’s a bit dated.
yourbandsucks
I think it comes down to just understanding IP and TCP in general. Which I don't really (IP) or do sorta (TCP).

As far as kernel architecture, the rest of the system can work with the assumption "writes go into a buffer, reads come from a buffer, and the networking subsystem does the right thing in between", plus some stuff about DMA optimizations.

weinzierl
Not very Linux specific but Tanenbaum's "Modern Operating Systems" is an excellent source for OS theory and very readable as well. It's an expensive book, but I found it worth its money.
jascii
I agree! I took Andy's class that he wrote the book for and it was one of the most influential classes I have ever taken. It is also one of the few books I have kept over the years and throughout all my travels.
Noe2097
Seconded. I would add "Structured Computer Organization", as a preamble book. I read both as if they were novels - they really are interesting and written in such a way that can hook you just as a good story
lorenzfx
If you are still interested in FreeBSD, "The Design and Implementation of the Freebsd Operating System" is a fantastic book. It's a great guide in how a UNIX system and especially FreeBSD works and written by some of FreeBSD's most respected developers.
non-entity
I actually did get a copy of that! Unfortunately, I only have the ebook at the moment, which I find difficult to read
ashafer
If you want something more hands on you can try "FreeBSD Device Drivers: A Guide for the Intrepid" by Joseph Kong. It is much easier to read, and my guess is you ran into it while making those BSD patches.

He also has a short book on building rootkits for BSD, which is a very fun read and demonstrates how things like system calls work.

If you want a guide to reading actual kernel code, here is a great resource explaining where some of the foundational pieces are. With a tool like cscope, reading the FreeBSD kernel is actually pretty easy.

http://www.watson.org/~robert/freebsd/reading/

Hope that helps!

mjb
This is a great book, and while FreeBSD and Linux differ in a lot of ways, there are enough similarities to make it worth reading even if you're only interested in Linux.
bytematic
OSTEP is the best book on Operating systems easily. http://pages.cs.wisc.edu/~remzi/OSTEP/
apacheCamel
Not sure if it is the best book overall, but I used it for one of my college classes and I think it is a great book for learning much more about operating systems. And it is available for free online so my college student wallet loved it!
yodsanklai
Not Linux, but I'd recommend this lab from MIT.

https://pdos.csail.mit.edu/6.828/2014/overview.html

I completed this one, but they may have more recent versions. They give you some initial code that you need to complete, as well as tests to check your work before going to the next step. It covers a lot of material. It takes some patience and it's not for complete beginners but it's a great fun project. Not sure, but I'd say it took me 1 or 2 weeks full time.

geofft
Yes, there is just so much, and it is not thoroughly documented. "BPF maps" in particular are a) a special feature of a special feature that's Linux-specific, not anything that generalizes across OSes, and b) a feature for userspace anyway, not for kernel internals. But even longstanding kernel-internal features aren't well documented (e.g., the other day I was trying to figure out what struct file's f_version does, and I think there's genuinely no docs for it.) So honestly I think the answer there is to not feel bad about not knowing everything.

The basic trick of dealing with the kernel is becoming comfortable working in a large codebase most of which you don't understand, and figuring out how to find what you need. Honestly, git grep is one of the best tools here. Get some practice finding some specific thing and where it's implemented, e.g., find a syscall (git grep SYSCALL.*foo) and trace what it calls. Find the definition of a structure inside include/ and see who uses it. Get comfortable with the kernel's OO-ish system of operations structs, and get some practice tracing both "this function makes a generic call, here's a sample driver that implements it" and "this is an implementation of a generic function, here's the syscall that calls it."

Beyond that, reading https://lwn.net 's articles is invaluable, partly for the clear prose coverage and partly for the breadth of what they talk about. (You don't need to pay unless you care strongly about this week's updates - you'll learn plenty from reading articles a week behind - but support them if you can, they're an important resource.) Again, you're not going to follow exactly why e.g. Google wants a new syscall for "restartable sequences" on your first read, but you'll get a sense of what is involved in adding a syscall, how various concurrency models work, what other kernel features are relevant, etc.

What sort of OS concepts are you finding yourself Googling? I will say that actually doing a college OS class is what made things like virtual memory management click in my head. It's an intensive approach but writing code in a much smaller kernel than Linux is a valuable way to understand concepts without being drowned in real-world optimizations and edge cases and portability.

non-entity
Perhaps I should've said more. I do often end up googling basic OS stuff, especially related to memory (software interrupts still confuse me though, I understand the concepts, but never the implementation), but even more so have been my ignorance of hardware level concepts. For example trying to patch the code for a network card the other week, I learned what watchdog timer is, I've never heard of that prior.
geofft
Working through an academic OS class where you implement interrupt handling is super helpful for that - hardware interrupts and software interrupts are similar but not quite the same and actually writing the interrupt handlers makes some things clear.

I can endorse MIT's 6.828 (I've taken it and taught a version if it), which (at least before this semester, they seem to be redoing things) has a lab about setting up interrupt handling: https://pdos.csail.mit.edu/6.828/2018/labs/lab3/ It's entirely possible to complete the labs on your own, and doing labs 1-3 is worthwhile, but it's also a significant time commitment.

Watchdog timers fall in the category of "there is so much" - they're not crucial to how network drivers work, but drivers in a production OS have to care about every feature that might interact with them. :(

livueta
> The basic trick of dealing with the kernel is becoming comfortable working in a large codebase most of which you don't understand, and figuring out how to find what you need. Honestly, git grep is one of the best tools here.

That's a very good point. If we're recommending tools, cscope (especially if you're a vim/emacs user) or opengrok (if you're not or prefer a web frontend for other reasons) are super helpful for navigating and comprehending large, mysterious codebases.

DTrace/eBPF are also valuable tools for grokking kernel functionality.

hatsubai
One important part about learning how the Linux kernel works is understanding the details of the system's interface the kernel provides. In my opinion, there is no better book out there than the "Linux Programming Interface" by Michael Kerrisk: http://man7.org/tlpi/

It provides extremely detailed information about everything going on in Linux, as well as example programs and exercises to help you further your knowledge. While it doesn't get deep into kernel theory like Tanenbaum's books tend to do, it will provide you with a greater understanding of how things work, IMO.

tbrock
This book is great. Seconded.
whowhatwhy
This book is great, so is Advanced Programming in the Unix Environment, which covers much of the same material but details the differences in standards and implementation. I'd also add `Linux Kernel Development` by Robert Love.
aidos
£50 on kindle! Looks like a great book, but that’s punchy. (Maybe that’s standard for a text book like this, not something I would normally buy)
geofft
It's a huge book (1552 pages), much larger than even your average OS textbook. It's more like a Linux encyclopedia.
sruffell
I was going to suggest this book when I see I'm late to the party, so I'll just have to add my voice to the chorus.

"Linux Programming Interface" is one of the best technical books I've ever read.

koolba
That’s a fantastic book.
gravypod
These are some good resources on the things that go into creating your own kernel:

  - http://www.brokenthorn.com/Resources/OSDevIndex.html
  - http://www.osdever.net/bkerndev/Docs/intro.htm
For learning about Linux specifically:

  - https://0xax.gitbooks.io/linux-insides/content/index.html
ndesaulniers
Join us in /r/kernel. As a mod I'm always helping folks with questions like this, and our community frequently beats me to the punch.
non-entity
Didn't know that existed. Will definitely check it out.
qntty
If you want to understand Linux systems programming, I've never found anything better than the book The Linux Programming Interface. For Linux internals, Linux Kernel Development by Robert Love is good. For something more removed from current systems, The Design of the UNIX Operating System and Operating Systems: Design and Implementation are good.
duckqlz
Beyond the exceptional mentioned texts dedicated to the subject; my favorite resources are:

lwn.net

github.com/torvalds/linux

pm215
My somewhat off-beat recommendation is "Lions' Commentary on UNIX 6th Edition with Source Code". The sources to a 1976 version of Unix written in an archaic dialect of C and targeting a long-dead CPU architecture are obviously not of immediate relevance to the modern world, and may well not be to your taste unless you already have some sympathy with the idea of retrocomputing. But if you do: 6th Edition is small enough that you really can read through the whole set of sources and understand pretty much how the whole thing is put together; the basic bones of the design are not so far away from Linux, so it gives you a workable conceptual model of what the shape of the Linux sources are and what the various important parts are; the commentary is really good, explaining the inscrutable but nudging you to figure things out yourself too; and overall it is practice in reading a large volume of somebody else's code, which is something you're going to be doing all the time if you work on Linux or any modern kernel. You probably also want a more theoretical and general book too, of course.
jiveturkey
And you can witness the glory of swtch() and the original "You are not expected to understand this" comment.

Highly recommended.

alfiedotwtf
I've commented below... there's an old book called "Linux Core Kernel Commentary" which is based on the Lion's book. It's a great read.

There's a Xinu book which is similar, but without the accompanying source code.

throwawaypls
Linux Kernel Development By Robert Love and Operating Systems Concepts by Silberschatz are great resources.
waddlesplash
The OSDev wiki can probably teach you anything you need to know about "OS theory" (and practice, mostly on x86): https://wiki.osdev.org/Expanded_Main_Page

Depending on your style of programming, I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals.

Have you looked at Haiku (https://www.haiku-os.org/)? We have a very well organized and commented modular-monolitic kernel, and a pretty active development team with a wide range of experience levels. I'm more than happy to help you (or anyone else) learn OS development!

Some example Haiku kernel code:

* "load_image_internal()", which is responsible for creating processes: https://github.com/haiku/haiku/blob/master/src/system/kernel...

* a recent change by me to replace a global lock with two local ones: https://github.com/haiku/haiku/commit/37eda488be1c9fee242e8e...

non-entity
I have heard of Haiku,but haven't looked at it in a while. However, I have heard great things about it and would certainly be interesting in learning and contributing
waddlesplash
I and most of the other developers hang out on Freenode#haiku, and of course on our mailing lists. Come say hello and get involved, indeed. More than happy to teach :)
jordz
I really love to see this kind of stuff within the community. I'd really love to take you up on this offer sometime :)
waddlesplash
Hey, please do! Onboarding new people and teaching skills is fun, and learning them while working on a project people actually use for day-to-day work is even more fun.

I idle in #haiku on Freenode, so ping me there and I'll usually reply soon enough. :)

xz0r
> I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals.

Could you also tell why ?

waddlesplash
The Linux kernel is really, really large, and changes extremely rapidly, to the point of breaking such "experiments" (as the original poster already noted.) It's also rather loosely commented and, aside from a handful of core systems, not so well documented.

The BSDs do much better here; they have excellent manpages for almost all kernel APIs, manuals for how the internals work, and very good source code commenting and organization practices. Haiku is not as good as the BSDs on manpages and manuals, but that's the only real way we differ from them (in this respect.)

kjs3
Complexity.

Start with something where you can substantially wrap your head around the whole thing, like XV6 or Minix, and it's vastly easier to look at chucks of something like Linux and see "oh, this fits into the mental picture here". For me, in the mists of the old days, learning Unix V6 out of the Lyons books (very slim volumes of annotation...wonderful stuff) make it easy to understand what was going on in V7, which made 4.2BSD and progeny accessible, which made SVR[234] somewhat weird and different, but still conceptually understandable, which made most everything Unix-ish afterword (including Linux) something you still had the vernacular to understand.

Now...to be fair...this leaves out a lot of really valuable OS knowledge. There's no IBM large system OS (e.g VM/370->z/OS, VM/CMS, TPF). No Seven Dwarfs OS (DEC VMS, Burroughs B-series, DG Nova, CDC NOS, etc.). Nothing for the really interesting Lisp or Smalltalk machines, the Perq or the Mesa/Cedar workstations. No interesting small systems OS: OS/2, classic MacOS, OS-9, much less the Windows NT family. Much less any of the embedded/RTOS world.

But you can still learn alot.

okl
Function with 9 arguments, 200 lines. Mix of abstraction levels. Basically untestable. Could use RAII.
waddlesplash
> Function with 9 arguments

That's nothing, let me show you some of the ones in the VM...

> Mix of abstraction levels

What are you referring to here?

> Basically untestable.

Unit-testable, no. Integration-testable, absolutely; POSIX testsuites are not so hard to write. And this function is called (tens of) thousands of times per hour while you are using the OS, remember.

> Could use RAII.

It is already using RAII reference and lock objects, what improvement do you think could be had here? There are no exceptions in the kernel, so that's not a concern.

If this is your criticism of Haiku's kernel code ... go take a look at some of the equivalents in the BSDs or Linux, then. We are by far and away better than those by your metrics (especially as, you know, those are C and this is C++.)

But do these specific metrics matter? Probably not. "9 arguments" is not a mark against a function in all cases, neither is length or abstraction mixing. Those can and usually are bad, but kernels are a different ballgame altogether.

no-dr-onboard
> modular-monolitic kernel

This sounds like a bit of an oxymoron. What is a modular-monolithic kernel?

linguae
The Linux kernel, as well as the FreeBSD kernel, supports the loading and unloading of kernel modules, which are compiled binaries that can be linked to the kernel while it is running. These modules are still in the address space of the kernel; the term "monolithic" refers to the fact that all portions of the kernel, even kernel modules, are running in the same address space. This is different from a microkernel like Mach where various operating system subsystems such as networking and the file system are running as separate user-level processes (thus having their own address spaces).
waddlesplash
"Monolithic" refers more to the way drivers and the kernel interact, i.e. if drivers live in the kernel address space, can read/write all kernel memory, call functions in other drivers, etc., then it's a monolithic kernel. If instead each driver is its own process with address space, and calls into the kernel via syscalls like any other process, then it is a microkernel.

"Modular" refers to linkage and overall design. The Linux kernel is mostly "statically" linked (there is DKMS but it isn't so widely used) with most major systems in the "kernel" itself, and something similar is true in the FreeBSD world.

Haiku, on the other hand, has a rather small (2.2MB binary on i386) kernel with no drivers built in at all (not even PCI); and loads all its drivers as relocateable shared-objects. This means unloading and reloading kernel modules at runtime (or boot) is very easy; so as long as you don't crash or hang the system, you can keep reloading drivers as much as you like without rebooting.

I know macOS also uses dynamic kernel modules, but I'm not as familiar with that system.

kernyan
https://github.com/s-matyukevich/raspberry-pi-os

Matyukevich page on GitHub has lessons with accompanying code to develop a Linux OS for raspberry Pi. The author goes into kernel, processor initialization, interrupt handling, scheduler, implementing syscall, and virtual memory.

I like his approach for these reasons 1. Minimal workable code, 2. Points you to entry function in linux repo 3. Line commentary of those entry function

Note that it's also a somewhat short read as he focuses on the practical implementations instead of the theory.

I myself was hoping that the author continue developing the chapters on file system, drivers, and networking but seemed to have been on hiatus.

gamescodedogs
Years ago in school, I decided to figure out how the Linux kernel works. So I've downloaded the sources and open them with a notepad++. After an hour I've figured out that I have no idea what am I reading in the code, close the notepad++ and never open it back. Buy the sources are still awaiting in some old directory at my home's PC :)
whydoyoucare
"Operating Systems: Three Easy Pieces" is a more modern book to OS concepts, explained with real source code as opposed to algorithms, and is regularly updated.

http://pages.cs.wisc.edu/~remzi/OSTEP/

(Online version above, you can also order a print copy).

0x262d
I took this class (although not from him). it's a good textbook and was pretty much sufficient.
snoocat
This is what I learned from, along with some xv6 labs. Just wanted to say that both Remzi and Andrea were exceptional professors and their book is clear and concise.
jiveturkey
The old standards.

Stevens to start. If you don't already know Stevens' thoroughly first, I wouldn't go deeper. The kernel is not separable from the userland-facing API.

I'm not convinced you actually need to / want to understand actual internals. But assuming you do, next is Tanenbaum (maybe start with the newer "Modern" book).

"Operating Systems: Three Easy Pieces" also seems well regarded, although I haven't read it myself.

The FreeBSD book is great, if you're specifically interested in BSD. There's also a video lecture you can buy on McKusick's site. I had the pleasure of an on-site presentation and it is absolutely fantastic.

ekr
Start with something smaller, such as lk.