Ask HN: How to learn proper systems programming?

Find an open source project.

I can't give you specific advice, but in general it sounds you are already on the right track. You are basically exposing yourself to this new stuff, it raises questions that you want to have answered and you reach out to a community to know more. The stuff they wrote you don't understand YET is part of the journey and if you continue to practice work in that domain, you will be able to connect the dots. TL;DR My advice is, be fair to yourself, accept it's a new domain and be resilient and continue despite the difficulties. I hope a systems programmer will give you more specific advice.

If you're interested in systems programming with rust then i can recommend "Rust in Action" by Tim McNamara [1].

You've found some pretty good resources already by the looks of it though.

[1] https://www.manning.com/books/rust-in-action

Find the kind of system you'd be interested in working on and start building one yourself. For me it was a game engine + an editor for it, but it could be anything you find the motivation to build.

Start with a book or a few texts/tutorials on the subject and begin building. Along the way you'll find the questions and choices involved. Find the answers from the internet or books. This is when I'd recommend some open source projects (not earlier) to see how they solved the specific problems. If you just go into an open-source project you won't have an understanding of the problem, just the answers, so it won't help you nearly as much.

> I had a Rust job for around half a year, where people build web servers and came from a C and C++ background

I don't think this is system programming rather network programming.

You should try embedded or try to write drivers

Better yet write a mini OS your own that interface with hardware

Edit: Wow, I'm getting down voted. I guess political correctness

A couple of books that are often recommended for understanding how to make robust software:

- Release It! (https://pragprog.com/titles/mnee2/release-it-second-edition/)

- Designing Data-Intensive Applications (https://dataintensive.net/)

I would suggest finding an open source project of interest and taking a deep dive into its code and documentation to understand how it works and why it was built that way.

Which reminds me, this should help with that: The Architecture of Open Source Applications (http://www.aosabook.org/en/index.html)

Some books I learnt off and still seem to be around

Modern Operating Systems by Tanenbaum is a good theory book - this will probably answer your questions about flushing etc

for down and dirty:

Advanced Programming in the UNIX Environment

TCP/IP Illustrated, Volume 1 (2 and 3)

There are two books that taught me how systems work.

- One system in isolation - Operating Systems: Three Easy Pieces. Covers persistence, virtualisation and concurrency. This book is available for free at https://pages.cs.wisc.edu/~remzi/OSTEP/

- Multiple systems, and how data flows through them - Designing Data Intensive Applications. Covers the low level details of how databases persist data to disk and how multiple nodes coordinate with each other. If you’ve heard of the “CAP theorem”, this is the source to learn it from. Worth every penny.

More on why these two books are worth reading at https://teachyourselfcs.com

First off, don't worry. You already have all the necessary tools you need.

Although the language you use to learn system & network programming doesn't matter much, it is better if you use C or C++ to practise and learn. This is because the kernel itself is written in C and exposes system calls that can be used directly from a C/C++ program. That said, "The Linux Programming Interface"(I am personally reading it) is a really good book. It talks a lot about how one should go about using system calls to get things done by the kernel. Make sure to read a little every day and try out the examples by writing C/C++ programs.

I recently realized that TLPI doesn't talk much about why are things the way they are(a very good example would be virtual memory and related stuff). You should refer some theoretical book for this. I suggest you go with "Operating systems" by Deitel & Choffnes.

Read man pages and practise using the libc/kernel APIs. For example, if you want to know about flushing, read 'man 3 fflush'. This might be needed when you want to flush all the input/output data that has been buffered by the C library before you can get fresh input from stdin. For example, if prompts are buffered, you definitely don't want to "scanf" before you have flushed the buffers. If you want to learn network programming, read chapters related to socket and refer 'man 2 socket'.

You will eventually get to a point where you will be able to connect all the dots(APIs) and be able to figure out what exactly you will need to get some problem solved.

Finally, don't learn for a future job. Learn for yourself. This will help you in the long run.

There are a few good YouTube channels of strong system programmers doing live coding. Watching videos takes time, but you can pick up lots of techniques, big and small, by watching people work.

Since you're looking at rust, https://www.youtube.com/user/gamozolabs/videos could be a good fit.

Pick an operating system, e.g. Haiku(BeOS), Plan9, Net/Open/Free-BSD. Download the source code and dig in. Read the source, the comments, commit logs and email threads. IMO Linux has gotten too big and complex due to having to support so many processors and devices.

Most existing systems programming is in C/C++. Rust is new and there isn't much battle hardened code out there.

What worked for me is trying to develop my own kernel. This might be too low-level for you, but it helps immensely to dispel the magic around how code gets executed by the CPU and the OS. You'll learn how the OS achieves protection from user programs, what are interrupts and how does the OS handle them, what are processes and threads and how does the OS schedule their execution, how virtual memory works (segmentation/paging), how the program is laid out in memory (code, data, heap, stack), how runtimes like the C runtime manages memory and links your code to OS syscalls, how dynamic linking works, what is an ABI, what is a calling convention, how does the compiler, the linker, and the OS know what each should do? How memory mapped file access works, etc. I can keep going, but you get the picture.

From there, you'll know where to go next based on what you've learned so far.

My recommended reading list is:

[1] Operating Systems: Three Easy Pieces https://pages.cs.wisc.edu/~remzi/OSTEP

[2] Intel Software Developer Manuals (especially volume 1 and 3A) https://software.intel.com/content/www/us/en/develop/article...

[3] OSDev wiki https://wiki.osdev.org

Start using your operating system directly instead of relying on libraries to do things for you. Learn Linux system calls and use them for everything. A great way to do this is to compile your C code in freestanding mode and with no libc. You'll have to do everything yourself.

It is easy to get started. Here's a system call function for x86_64 Linux:

https://github.com/matheusmoreira/liblinux/blob/master/sourc...

With this single function, it is possible to do anything. You can rewrite the entire Linux user space with this.

The Linux kernel itself has a nolibc.h file that they use for their own freestanding tools:

https://github.com/torvalds/linux/blob/master/tools/include/...

It's even better than what I came up with. Permissively licensed. Supports lots of architectures. There's process startup code so you don't even need GCC's startfiles in order to have a normal main function. The only missing feature is the auxiliary vector. You can just include it and start using Linux immediately.

You can absolutely do this with Rust as well. Rust supports programming with no standard library. If I remember correctly, the inline assembly macros are still an unstable feature. It's likely to change in the future though.

Take a look at the APUE course: https://stevens.netmeister.org/631/

Hello there I have quite similar background. (10 years experience, in backend and native android development). Most of the comments already contain good advice. Systems Engineering takes time. I followed long route. I initially took some course in digital logic, and computer architecture and than later took following OS courses.

1. https://pdos.csail.mit.edu/6.S081/2020/schedule.html (check video links and do all the lab assignments)

2. https://www.youtube.com/watch?v=dEWplhfg4Dg&list=PLf3ZkSCyj1... (based on old MIT 6.828)

3. Networks Review (self study)

Somebody mentioned Intel manuals in the comment (those are really helpful)

These courses helped alot. I would suggest to take some firmware and device driver development course as well. My conclusion is that, these are tough skills and the learning process can be accelerated if you can find some entry level position in a small company which do this kind of work.

If you know folks in your company are writing things you are interested in reach out. Most of those people are happy that someone is taking interest in what they are doing. The information that they will provide will be meaningful and valuable. Use what you learn from those conversations and leverage it in your daily job functions to better position yourself for future roles/jobs/projects.

Just to check my own knowledge. Do you mean opening another socket for logs?

If not, what is a channel?

From personal experience, what made low-level concepts "click" for me was looking at documentation for linkers and the various ELF tools. Once I could see the "ends", it gave me the inspiration to dig into the "means". Ymmv, of course.

If you want to learn Rust for systems programming, first learn C and write something non-trivial in it. Then run it through Valgrind [1] with your test suite. You can also try the sanitizers (AddressSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer, ThreadSanitizer, etc.).

This will teach you what Rust is saving you from and help you understand how it does it.

[1]: https://www.valgrind.org/

> I had a Rust job for around half a year, where people build web servers and came from a C and C++ background. Half of the stuff they wrote I didn't understand (flushing, opening another channel just for logs so we don't fill up the other ones etc. etc.)

I think it's worth mentioning that the majority of topics involved in this type of webserver work wouldn't necessarily be covered by a systems programming book. There's obviously overlap between all of systems programming, networking, and concurrent/distributed systems, but if you plan to focus on web servers, I'd pick up texts on the other topics as well.

Years ago I modified the Postgresql source code (8.xx).

It's something I was terrified of doing.

But once I got in there and started poking around, I realized it was just ordinary plain-vanilla C code. Not C++. Just C code.

With my local copy, I started to hack pg_dump to do something special that we wanted at the time. Even after 30 years of coding, I'm not that especially good of a programmer. But I ended up getting our own special version of pg_dump that did what we wanted at the time and it went into production dumping hundreds of gigs of data every day!

But what I'm not, is afraid. I'm not afraid to try anything.

And that's what it takes to do deep, systems level programming.

Don't be afraid.

Those bits are just bits. And it's just code... and most of it was not written by wizards. Just ordinary people like you and me. Don't be afraid man.

Clone the repo and setup a workable build environment and start tinkering and compiling and running to see what happens.

You would be totally shocked to find out what you can actually achieve.

Those books will only go so far.

    bool cond1 = ...;
    bool cond2 = ...;
    if (cond1 || cond2) {
      sleep(1000);
    }

    What a fool can do, another can.

    Considering how many fools can calculate, it is surprising that it should be thought either a difficult or a tedious task for any other fool to learn how to master the same tricks.

    Some calculus-tricks are quite easy. Some are enormously difficult. The fools who write the textbooks of advanced mathematics – and they are mostly clever fools – seldom take the trouble to show you how easy the easy calculations are. On the contrary, they seem to desire to impress you with their tremendous cleverness by going about it in the most difficult way.

    Being myself a remarkably stupid fellow, I have had to unteach myself the difficult, and now beg to present to my fellow fools the parts that are not hard. Master these thoroughly, and the rest will follow. What one fool can do, another can.

Back when started coding, system programming were books like these ones,

https://www.atariarchives.org

https://archive.org/details/pcinternsystempr0000tisc

Granted they are probably too old, but the concepts of what is actually systems programming is there, you can then get hold of an Arduino or Rasperry PI like device and do your own little OS or bare metal game,

https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/o...

http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Ass...

Or maybe trying your hand at compilers with https://www.nand2tetris.org or given the similarities of Rust with ML, maybe dive into Tiger Book (https://www.cs.princeton.edu/~appel/modern/ml).

Or maybe

https://www.manning.com/books/rust-in-action

https://www.apress.com/gp/book/9781484258590 (Rust for IoT)

Follow-up question: how do I learn systems programming on Windows? I know it’s pretty easy to just set up a Linux VM or use WSL2, and I’m prepared to do that if I must, but it would be even easier if I could work in the same Windows environment I normally use. However, from what I’ve seen, practically all guides to systems programming seem to start off by assuming you’re using Linux.

(And by the way, thanks OP for asking this question! I’ve also been wanting to learn systems programming, but I haven’t gotten around to asking yet. And all the suggested resources look fascinating… there goes my university vacation!)

I've also been trying to learn system programming for a while and what did the trick for me was finding a way to use the C interfaces that UNIX provides.

I've tried to learn C probably 4 times now and I just don't like it. But then I came across LuaJIT ffi which, very easily allows you to use whatever shared library and call whatever syscalls directly and that was a game changer!

After that I decided to test ziglang, which a big part of it's design decisions is interoperability with C, and I'm in love with it! It really feels like anything I would need C for I can do in zig.

If Rust if your jam find a way to call the linux syscalls directly from Rust, not just using a cargo library, but actually importing the appropriate headers and successfully doing an epoll or something.

It will feel like suddenly the man (3) pages all make sense and are extremlly useful!

Good luck!

Get a job in the field. Some places might be willing to take a chance on you if you are a good generalist engineer.

Also, make the case that many things that you learn while doing frontend programming like asynchronous programming are skills that will port over just fine.