HN Explorer

Ask HN: Looking for a resource on Linux kernel module development

93 points by matt3210 ↗ HN
I'd like to break into this area in my next role and I'm looking for resources to learn about this over the next few weeks so I can try to get a role in the area. My context is embedded linux.

EDIT: I don’t know what I don’t know so I’m looking for a “point in the right direction” and anything the fine people of HN think is a good resource

45 comments

[ 2.5 ms ] story [ 136 ms ] thread
yxre
I am also learning about the linux kernel, but not in an embedded context. Here are some good resources I have found.

https://man7.org/tlpi/ This book is considered the best overview of linux kernel interface. Its old, but still gives a broad coverage of topics.

https://lkml.org/ Browsing LKML is really helpful since it shows you where current development is happening and how kernel devs think. Some conversations get very advanced and nuanced.

https://kernelnewbies.org/ is a good starting place for starting to compile and deploying kernel builds

Other than that, reading and tracing the source is probably the most productive thing to do for an experienced dev to do

jfdi
What kind of resource are you looking for?

Have you already gotten started with your LLM of choice, and need something more?

matt3210
I don’t know what I don’t know, so I need a “point in the right direction”.
SanderNL
Just here to note that it is amazing I can just copy-paste the request and get a summary with all your guys’ suggestions.

Don’t forget "Embedded Linux Primer" by Christopher Hallinan.

synergy20
it's a good book, but it's 13 years old.

in fact, most of linux kernel books are aged these days.

scrollaway
You could look for existing active current or ex kernel module developers and pay for a couple hours of their time to guide you through the weeds and give you early pointers. This is my approach when breaking into a new field if I have access to people.
matt3210
Good advice, I'll keep an eye out for experts willing to teach
beeburrt
Maybe Joseph Kong's books?

Designing BSD Rootkits

https://nostarch.com/rootkits.htm

FreeBSD Device Drivers

https://nostarch.com/bsddrivers.htm

Playing Games With Kernel Memory ... FreeBSD Style

http://phrack.org/issues/63/7.html#article

matt3210
Very nice, a BSD prospective helps a lot!
spacemadness
A complete tangent here, but is there any modern Phrack like zine? Other than Phrack of course which seems to have had a brief rebirth with their October 2021 release.
mfrw
Linux Kernel Development by Robert Love is something I would encourage going through if you are unsure about what you don't know.

Although it's based on 3.x kernel, it still is pretty relevant.

Good Luck!

nottorp
Semi related piece of advice:

Also get a random raspberry pi and a simple peripheral for it: some 1 wire temperature sensor or something on i2c.

Then ignore the existing drivers and do your own. Do a character device and have a read from it trigger a read in kernel and copying of the data to user space, maybe in human readable form.

Should be good practice.

And for general embedded Linux development you also want to take a look at Yocto.

matt3210
This is great advice thanks!
realo
Yocto is fine, basically the "traditional" way of doing embedded systems (e.g pSOS or vxworks style) .

However, I found great joy in using Nix for more modern development t of higher-end embedded devices.

If your target platform is powerful enough to accommodate NixOS, you might want to give it a try.

nottorp
Nix seems to advertise itself as something to do reproducible builds with, which is damn nice.

But what's the extra effort if your hw vendor doesn't support it? The examples on their homepage look more like cloud computing not embedded.

smashed
Yocto has made great progress into fully reproducible builds too, just fyi..
realo
You are totally right... Nix is cool for advanced CPUs (i.e. x86, armv7, aarch64, etc) and much less so for microcontrollers.

But if your platform can run NixOS ... then ... let's see...

With nix I have a file that exactly describes the git sources (pinnable on precise changesets if required) for all my components (including custom kernel patches, device drivers, etc).

I use a nix config to create a fully GPLV3 compliant rootfs and another set of nix config files in a separate repo to create our device specific intellectual property, in a separate encrypted image different from the rootfs.

The nix config files easily describes everything i need to fully support Secure boot, as well as code to unseal a TPM luks password used to decrypt the image that contains our intellectual property for that device (basically the application).

All this is fully reproducible, so I can provide our clients with the nix config files for our rootfs, tell them to use nix and they can reproduce our exact rootfs image to run in the device , for GPLv3 compliance. Of course , then the device is "broken" , because our IP will not be decryptBle anymore. But the GpLv3 does not mandate full functionality when running foreign images...

Now I am absolutely certain all this can be also done with Yocto... but with less simplicity ... less joy ... :)

I used pSOS in the past, got burned when Integrated Systems was sold... then stayed away from proprietary OSes as much as I could. Used Debian with XEN later on... then Yocto in many products, then back to a Debian-like OS and now , finally, using Nix. It just works.

yjftsjthsd-h
> Of course , then the device is "broken" , because our IP will not be decryptBle anymore. But the GpLv3 does not mandate full functionality when running foreign images...

I thought v3 was specifically made to prevent tivoization? And that sounds like you even used the same implementation as TiVo.

realo
Well... you get to reproduce yourself and run everything covered by the gplv3 (same exact versions, same binaries even with Nix) on the device. You get everything you need to reproduce the device's rootfs. It will boot your image, if you disable Secure Boot. You can install your own UEFI certificates if you want to enable Secure Boot with your image. You even get the complete development environment for free (Nix).

Our IP is obviously not covered by the GPLv3.

What is the problem?

smashed
Yocto is fine and very powerful. I work with it daily, producing a "just enough os". It's amazing to produce a full OS with fully traceable sources, versions and licensing of everything, even CVE patch tracking.

If you work long enough in embedded Linux you will encounter it for sure.

I would advise against starting with yocto for a beginner. It is very complex, most tasks look more like weird magical incantations than actual code. It will discourage many, as things will feel completely incomprehensible for months.

You kind of have to know the problems it solves to appreciate it.

In another comment I pointed to OpenWRT and Raspbian. They have real world devices they target, you can git clone, build and flash real devices to get you started and see real results faster. Their build systems do not have all the complexity of layers and are easier to follow.

synergy20
I would suggest beginners to start with buildroot.
smcameron
You don't really even need a device to get started. The first driver I ever wrote consisted of 128 bytes of memory that you could write to and read back from. After I got that working, I realized I could change the 128 to 1.44M, implement a couple ioctls and have a reasonable RAM disk substitute for a floppy drive (this was SCO OpenServer back in the 1990s, and it didn't have a loop back device.)
oasisaimlessly
With VFIO, there is increasingly little need for writing in-kernel device drivers. You can do everything in userspace with better DX and without having to deal with kernel API breakage.

If you want to try out VFIO, you could try writing a userspace program that interfaces with some simple PCI device, like a serial card.

matt3210
Good point, can you point me in the right direction for more info?

Edit: A cursory glance suggests this is for connecting existing devices to virtualized guests, how does that replace kernel modules for hardware components?

oasisaimlessly
Paaaing hardware to VMs is the "killer app" for VFIO, but VMs (e.g. QEMU) are just normal userspace processes. In other words, VFIO can delegate control of a device to any userspace process, but it's most commonly used for VMs.

DPDK [1] is an example of a non-VM VFIO user.

[1] https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

synergy20
for userspace drivers check out UIO.

VFIO is still mostly about virtual machines, as a beginner I won't touch it for now.

I will stay in the kernel for a while before doing userspace drivers.

tux3
You could try looking up the tasks of the Eudyptula Challenge.

More than half the value was in sending mails and getting feedback, but it's still a great introduction that gradually gets you used to more and more complicated kernel dev

There are solutions lying around on the internet for the various tasks, but I'd advise not looking. The kernel is sparsely documented (some areas, rigorously undocumented), so a bit of experience searching and figuring out how to get the information on your own is genuinely valuable

bitcoinmoney
I think it’s not there anymore ?
tux3
The challenge is closed, but it's really just a series of emails with tasks for you to do. You can find a list of the tasks online, only there won't be someone to verify your solutions since the challenge is closed.
smashed
Since your context is embedded Linux, I would recommend following Raspbian development for general purpose SBC embedded Linux or OpenWRT for smaller, more specialized devices.

Both systems have code bases and build systems that will build root filesystem and kernels for a wide range of devices.

By following development of a device (aka target) that interests you, you will see how drivers are stabilized with various patches applied over time.

wilburm
Since you’re in embedded Linux, I’d suggest some of the bootlin learning material. Something like an i2c or spi driver might be good.

QEMU is nice for learning because you can more easily debug kernel code with gdb.

cushychicken
Bootlin’s training for embedded Linux and driver development is second to none.
Roark66
I second advice to focus on embedded systems, but I'd stay away from raspberry pi. I'd much prefer something you could debug easily from outside. There are plenty of arm chips one can debug with openocd and gdb. Sometimes being able to debug a running kernel from the outside can make a huge difference when troubleshooting.

If also choose some class of device and get to know it. Many Linux drivers are very similar across a class of devices. For example CSI cameras on top of V4L. Thankfully the code in the Linux kernel is very readable. Also it is well documented online.

I definitely recommend a good ide too.

westurner
- "The Linux Kernel Module Programming Guide" (2023) https://news.ycombinator.com/item?id=35782630

- Still trying to find the How To Build a Formally Verified Kernel Module from Zero tut

Edit:

- /? Kernel module https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

- "Linux Kernel Module written in Scratch (a visual programming language for kids)" (2022) https://news.ycombinator.com/item?id=31921996 (EduBlocks does Scratch with Python blocks and/or text code)

- "How to Discover and Prevent Linux Kernel Zero-day Exploit using Formal Verification" (2021) [w/ Coq] http://digamma.ai/blog/discover-prevent-linux-kernel-zero-da... https://news.ycombinator.com/item?id=31617335