Ask HN: What should a systems/low-level software engineer know?
Tired of the hype on the web/js (every week new library to learn about), I am thinking to switch career in the next 1-2 years, and move lower in the stack. I am into Rust in the last 3-4 months and I really enjoy it, but I would like to learn more, what would you consider essential books/papers/resources that a system's programmer should know?
230 comments
[ 3.7 ms ] story [ 253 ms ] thread[0]: https://people.freebsd.org/~lstewart/articles/cpumemory.pdf
Jokes aside, the guy asked how it's been useful and it's not terribly convincing that your first (and only?) thought was interview questions.
I, honestly, don't know if it will help me in terms of my career because at my current job, it's not something I can put to use. However, I want to learn this because it is fun (for me) to understand things close-to-the-metal.
And yes, the title isn't hyperbole. You can skim the parts describing specific tooling, but the fact that the description of the memory hierarchy and how it works hasn't been internalized by every programmer is a travesty.
This comes up incredibly frequently, and having it be second nature will benefit you.
A lot of the time, this gives you a starting place for how your inputs and outputs should look, what your method signatures should be. For example, if your data can can fit in a uint8_t, why waste bytes on a uint32_t?
This isn't the be all and end all, especially since modern architectures use larger word sizes/RAM is plentiful/bandwidth is cheap, but it can be a helpful way to frame the conversation when you're architecting software (especially network protocols).
I'm an ME who went into embedded systems, so I'm not sure what qualifies as low level for a web developer and if checking timing using an oscilloscope or saleae is feasible for the type of work OP wants to do. But even just looking at the assembly would have made things obvious in my example.
EDIT- This made me realize that a rudimentary understanding of the assembly language for your architecture will be very valuable. Doesn't have to be enough to actually write any code in it, but it's great to be able to take a look at what was generated when things are acting weird.
Without that though, performance is really really bad. The inner loop of a motor controller is no place for them...
Anyway a bit toggle tells all, and if you're making speed optimizations you should be checking that things actually got faster.
So I agree, knowing how data is stored is critical.
I prefer to generate them, so I would add to that "tools for generating C code and state machines".
(Tangent: A guy who worked on Verilog and FPGAs told me that the "whole idea" is state machines... and they are hard to get right.)
When is it appropriate to allocate memory from the heap? If you're in a rendering or audio processing routine in realtime context, avoid it at all cost.
Think about which parts of the code could profit from optimization. Does learning an assembler for a specific platform pay off or can the compiler do a sufficiently good job with -O3? Use profilers to identify performance bottlenecks.
Think about portability. Does the code have to compile with ancient C89 compatible compilers [Mine has to, and I would be excited to see a Rust to C transpiler]? Can you choose your compilers by yourself? Are they provided by your customers?
Only reason to go lower in stack is if you really ~like~ love hardware and exploring nitty gritty of how things work. In that case, pick any OS book and get ready to go down the rabbit hole :)
I am bored of the churn associated with stuff higher up the stack. I would prefer to master a craft rather than chase new fads every few years.
Also, you won't be required to work on pet side projects, push them to Github, blog about them and post Show HN comments to build up street cred for future job interviews.
But I have no idea how one breaks into this field. Maybe that's worth an Ask HN or a Stack Exchange post.
I learned enough to know when to call IBM for support, but with that experience it wouldn't be too difficult to find another job managing one.
I remember 12+ years ago, my old boss bitching about working on an AS/400, then he'd hire his brother (who worked on them for years) to come out to do very simple stuff for quite a high premium.
I just wish they weren't so proprietary, but at the same time, I am glad they are. It's been a love hate relationship.
Now I work with an AIX system. Unix under the hood, but IBM still has a stranglehold on it. Both tanks though.
- Understanding how to write purely event-based programs using e.g. epoll since that will avoid lots of complexity and potential problems compared to throwing threads at everything.
- Floating point representation and gotchas ("What Every Computer Scientist Should Know About Floating-Point").
- Unit-testing as part of designing interfaces to get low coupling, sane ergonomics, and forcing some thought to error handling instead of just working in the happy-path case.
- Interrupt- and thread-safety (races, mutexes, deadlocks, orderings etc.)
- Storing state in maintainable ways (i.e. not just dumping C-structs to disk).
- Understanding basic computer architecture concepts like virtual memory, caches, and pipelines.
- Know what can be achieved with OpenMP and GPU programming using OpenCL/CUDA etc.
- Write great commit messages (What is the problem being solved, why this way, etc.)
- Basics of database and distributed systems theory like ACID properties and Lamport timestamps.
- Using regular expressions instead of lots of a complicated set of if-statements.
- Understanding networks regarding error sources and latencies.
With that being said, your list of things to know are all things I have to know for my job (these are things almost all programmers should know, in fact) and I would not consider myself a low-level or systems programmer, just an application programmer.
This takes in the definition of a regular language as a set of regular expressions, and generates C code for finite state machine to parse the language. You can visualise the state machine in Graphviz to manually verify all paths, making it much easier to spot hidden corner cases, while being a lot quicker to code than a big pile of if statements.
The other points are solid, although I have to admit I have never used regular expressions on a microcontroller
But then someone has a bright idea that e.g. a global cache to share some state between all threads would be a good optimization, or that threads need to be able to store some global settings, etc. and complexity related to threads start to creep in to your application.
And I would also add fuzz-testing to your list.
If you go that path, bear in mind that you will not be involved in creating clever algorithm. When safety is involved, you need to program very simple and easy code. The complexity lies in how all the parts and ECUs are interacting with each other. On the upside, you do not need to constantly learn new languages and library, and you accumulate expert knowledge which is interesting for companies.
But that's not the point. C and C++ are in practice used completely differently, so writing C++ using C concepts with some additions rarely happens in the wild.
But the major point is that there are almost always safer or more ergonomic ways to do things using C++ features that are not present in C.
Because sometimes that's annoying, things can get less type safe, but in your head, there's a C++ program that you're representing.
But in reality, why C is still the best programming language for large projects (IMO) is exactly that the programmer is allowed to choose a suitable structure, such that the program can fulfill the technical requirements. Other languages force the project into a structure that somehow never fits after a couple thousand LOC.
What good programs are written in C that don't have well-structured memory management the way C++ does it with RAII?
Edit:
"But in reality, why C is still the best programming language for large projects (IMO) is exactly that the programmer is allowed to choose a suitable structure, such that the program can fulfill the technical requirements. Other languages force the project into a structure that somehow never fits after a couple thousand LOC."
Yeah, this doesn't make any sense. The reason is, C++ doesn't impose anything on your program structure that C doesn't, while C, with the limitations it has, imposes a tax on all sorts of ways of structuring your program.
For example, you can't practically write a program using a futures library (such as the Seastar framework) in C. And every program you write in sensibly written C can be translated to C++. The exception might be really really small-scale embedded stuff that doesn't allocate memory.
MISRA C standards, popular in embedded projects especially automotive, ban the use of memory management altogether.
The whole point of RAII is that the compiler manages it for you as far as it can. This is impossible in C because you have to do it manually. You might end up writing malloc() at the top and free() at the bottom of functions but that's the opposite of RAII.
If you're running in an automotive environment, you're probably real time; that is, you have to finish your processing before, for example, the next cylinder comes into firing position. You have to hit that, for every cylinder of every rotation of the engine, for any rotation rate that the engine is capable of reaching. You can't be late even once.
Now in the processing you have a malloc call. How long will the call take? Depends on the state of the heap. And what is that state? Depends on the exact sequence of other calls to the heap since boot time. That's really hard to analyze.
Yes, you can get a memory allocator that has a bounded-worst-case response time, but you also need one that absolutely guaranteed always returns you a valid block. And the same on calls to free: there must be a guaranteed hard upper bound on how long it takes, and it must always leave the heap in a state where future allocations are guaranteed to work and guaranteed to have bounded time.
And, after all of that, you still have a bunch of embedded engineers scratching their heads, and asking "explain to me again how allocating memory at all is making my life easier?"
So embedded systems that care about meeting their timings often allocate the buffers they need at startup, and never after startup. Instead, the just re-use their buffers.
Then all the implications like exceptions and needing to implement copy constructors, move constructors, etc. in each little structure.
As to what C project doesn't just emulate RAII: Take any large C project and you will likely find diverse memory management strategies other than the object-oriented, scope-based one. Also, other interfacing strategies than the "each little thing carries their own vtable" approach. The linux kernel is one obvious example, of course.
But I also want to reference my own current project since it's probably written in a slightly unusual style (almost no pointers except a few global arrays. Very relational approach). https://github.com/jstimpfle/language. Show me a compiler written in RAII style C++ that can compile millions of lines of code per second and we can meet for a beer.
> The reason is, C++ doesn't impose anything on your program structure that C doesn't
Of course you can write C in C++ (minus designated initializers and maybe a few other little things). What point does this prove, though?
It wasn't really implemented for performance, and maybe the language is more complicated -- no doubt it's a lot slower. On the other hand, I can look at any function and see what its inputs and outputs are.
I've mostly optimized for clean "mathematical" data structures - basically a bunch of global arrays. This approach is grounded on the realization that arrays are just materialized functions, and in fact they are often the better, clearer, and more maintainable functions. If you can represent the domain as consecutive integer values, of course. So I've designed my datastructures around that. It's great for modularity as well, since you can use multiple parallel arrays to associate diverse types of data.
But anyway, your language looks impressive I must say.
I don't know enough about D to really comment on it (I read the black book on it 9 years ago, and the examples crashed the compiler), but... it has better compile times, right? There's a module system? I'd have to look at the D-with-no-GC story, figure out how hard it is to interop with C++ API's. I think I'd have better feelings about D (or C++) if it didn't have class-based OOP.
Treating C as a sort of "portable assembler" is a lot better, although it runs into UB problems (see DJB on this subject).
I think there is a world of wild and crazy C++ (like, boost::spirit-grade, or std::allocator-using) that you're imagining, that is not what I am thinking of. If you took C, and added vector<T> and hash_table<K, V>, added constructors and destructors so you don't have to call cleanup functions, you'd get a language which most sensible non-embedded C programs would map to, which then maps upward to C++.
Maybe some templated functions like std::min<T> and std::max<T> and add_checking_overflow<T> would be nice to have too.
Edit:
An example of where I think properly written C diverges from portable assembler is things like here: https://github.com/Tarsnap/tarsnap/blob/master/libcperciva/d...
It depends on whether you think ELASTICARRAY_DECL is within the scope of portable assembler. (It gives you type safety!) (And I don't know what advanced assembly languages can offer in terms of that -- maybe they do too.)
But it does in C: https://godbolt.org/z/zdTKiE
https://blog.directededge.com/2009/05/21/c-and-c-are-not-the...
Json is not exactly a subset because it supports representation of numbers of any precision. JavaScript will convert to an IEEE 754. But it's up to a json deserializer to decide how many decimals to use. I think.
Then pick whether you want to start from the "front" or "back" of C++; i.e. learning the language in chronological order or in reverse. C++17 idiomatic style is very different from the 99 style that most existing C++ is written in.
I would suggest picking a codebase to work on and learning its style and subset of C++ first.
EDIT. The book also comes with a collection of excellent projects, which the authors have made freely available online at http://csapp.cs.cmu.edu/3e/labs.html
>before you rewrite your C book, perhaps you should take the time to actually dig into it and learn C first (inside and out.)
which is uninformed, as Zed wrote Mongrel and Mongrel2 in C. Saying he doesn't know C is ludicrous. He might have a different approach to C, but then argue this view instead of claiming your way is the only way. The author of that blog post is saying the book is bad because it is not the way he writes C. Not because it is objectively bad.
Also, replies to that post like "Just for info K&R stands for "Kernighan" and "Ritchie", please, don't compare the bible of C with "just another book on C". It is a blasphemy." are hilarious. People are just parotting "read K&R!!" off of each other. The term Stockholm syndrome is overused but it is very appropriate for people who think C is actually good.
I think that C is actually good, and that C++ is a Scooby-Doo sandwich of fail. But this is an opinion concerning a particular domain (low-level programming) that doesn't carry over to anywhere else.
Also 21st Century C has been recommended, and it goes into tooling as well.
> as a self-respecting software engineer
Then you probably already know more about C than you realise: if, while, for, etc. all work the same as most other languages.
What you're probably not used to is needing to define functions in advance, pointers, and memory (de)allocation.
If you learn by doing, then with google and github, you can create a few simplified versions of unix utilities (cat, ls, grep -F).
If you're on Windows, I recommend using a Linux VM, WSL, or Cygwin - it's easier to setup than the C tooling in Windows (and you're stealth learning something else).
Once you know C, you can then move onto C++ (I stopped at C).
Agreed. APUE is perhaps one of the best book available covering SUS, POSIX layer in detail.
Nice. I'll check it out.
But all the fun stuff happens when you start talking with your OS, so get a book about that too. If you are planning to develop on Linux, Michael Kerrisk's The Linux Programming Interface is excellent. Much of it will be familiar to someone used to shells and the terminal, but there will be plenty of new ideas too, and even the stuff you know will get a much deeper perspective.
I'm french and studied in the UK. I'm sometimes lost when my colleague use french technical terms, i have to ask the concept behind it to be able to identify the english term which i learnt during my studies.
> However, as an English speaking engineer, I found many of the discussions rather clumsily written. I'm guessing that it was translated from the German by someone who doesn't thoroughly understand the subject matter.
It's worth picking up some basic gdb skills. Use of ddd can help with this. On windows you can use VS for most of this of course. Picking up the basics of valgrind will also help you.
Get comfortable with the preprocessor. Get comfortable with Makefiles. Get comfortable pulling in library headers and binaries as needed.
Errr....
I was (mostly) a C programmer for over a decade, but that's about all I can think of right now!
--edit--
And someone below has just triggered me - FFS use stdint.h!
You absolutely need to know in C++ too. Modern C++ just gives you tools to express ownership in code. Rust goes way further and gives you compile time correctness of your lifetime handling
Pointers, stacks (one in ever 23.7 bugs is a stack smashing bug), bit bashing and endianness, types and coercion at the byte level (see also: pointers, bit bashing), C strings, the stupid rules about when a variable's value is actually written to memory that need to die in a fire, memory allocation/clearing/copying/ownership/freeing, ALWAYS CHECK RETURN CODES, what the heck an lvalue is.
This book is fun:
http://shop.oreilly.com/product/0636920033677.do
Then you should learn Unix. From an understanding point of view, I think it's probably better to learn something like FreeBSD, NetBSD, Xv6 ... Linux is very pragmatic, and very general, and so it doesn't have the purity that smaller, more focused or curated systems have. Once you have a handle on Unix, look at other OSes: Plan9, Minix, FreeRTOS, L4, etc.
Then networking: I suggest starting with IP, TCP, ARP; then step down to the physical layer: Ethernet, Token Ring, 802.11, hubs, and switches; then static routing, RIP, OSPF, and BGP; maybe look at mesh routing. Then some application layer stuff: DNS, NTP, SSH, NFS, LDAP, HTTP, etc. Reading the RFCs is really valuable, and they're remarkably accessible up until say 2500 or 3000 or so.
Security: symmetric and asymmetric crypto, Kerberos, SSL, SSH, OAuth, etc. Read up on pen testing, social engineering, defensive programmings, fuzzing, etc.
Databases: both relational and otherwise. SQL. Wrap your head around how a filesystem and a database are the same and how they're different.
Messaging: some sub-set of protocol buffers, Cap'n'Proto, Avro, Thrift, XDR; brokers vs. p2p; pub-sub vs. directed. There are hundreds of system you can look at here: pick a few that look different.
Learn about complexity analysis, distributed consensus, locking, concurrency and threads.
so far as tools go, you need to understand a debugger (how to use it, and how it works), packet capture and analysis (Wireshark is good), profiling and performance analysis.
That's probably a decent coverage for the software side. The exact focus will differ depending on embedded/real-time vs enterprise, etc.
From the hardware side, I think it's worth starting with an 80's or earlier, 8 or 16-bit system, and learning how it works to the digital logic level. What a simple microprocessor actually does: fetching, decoding, execution units, etc. A Z80 or 6502 or similar system is a pretty simple circuit, and it's worth really grokking how it works.
From there, you can move forward to more complex CPUs, newer memory architectures, newer buses, etc. But it's much harder to dive straight into a modern x86 or ARM CPU and try to understand how it works.
It's a this point that reading Drepper's memory article, and the "everything you should know about latency" article(s), etc, really start to be useful, because you've got a solid grounding in what's underneath them.
You don't need to do this all at once, or before you start working more on backend or systems level code: I'd guess it took me close to 10 years to feel like I had a decent grasp on most of it.
- Embarrassingly low-level (chip developers): OS, C/VHDL/Verilog, MA, DS, GPH, EE, Apathy, SE
- The in-betweeners (kernel, device, storage developers): OS, C, MA, DS, GPH, Apathy, Crypto, SE
- Low-Low-level systems: OS, C, DS, GPH, Apathy, Crypto, SE, FTDS, MSG, NW.
- Low-High-level systems: C/Rust, DS, GPH, Apathy, HDS, Crypto, SE, FTDS, MSG, NW, TL.
- Unicorn rock-star systems hacker (perhaps only a handful of these creatures exists :-): Multiple OS, C/Rust/JVM/Erlang/Haskell, Multiple MA, DS, GPH, EE, Crypto, FTDS, MSG, NW, HDS, SSE, TL.
Abbreviations used:
GPH = Good Programming Hygiene
EE = Electronics and Electrical Engineering
OS = Operating System
MA = Modern Assembly
DS = Data Structure
FTDS = Fault-Tolerant Distributed System
MSG = Messaging System
NW = Networking
TL = Toolings (the darlings of UNIX)
HDS = High-Density Systems
SE = Security Engineering
You have to watch what compiler flags you use, like someone turning off stack cookies, not using clang's sanitizers. Check out https://clang.llvm.org/docs/AddressSanitizer.html it would have prevented the Heartbleed vulnerability if it existed at the time.
You need proper bounds checking everywhere! You should fuzz your code with something like AFL and if you don't have the time to setup test cases for it, just send your program random junk and see if you can get it to seg. fault.
Multithreading is hard, and detecting multithreaded bugs is even harder. Random monkey testing can sometime's help find these, but they are very hard indeed. Monkey testing is just literally if a monkey was smashing your keyboard and your program was open, what would happen?
Know the most vulnerable function calls by taking a look at banned.h from Microsoft's SDL. It looks like it's no longer on the official website, but the author put it up here Take a look at https://github.com/x509cert/banned/blob/master/banned.h. Sometimes you can't avoid using these functions but know why they can be considered bad.
A proper Makefile is your team's best friend. The same can be said with one build machine for your whole team. It is easier now than ever to build and distribute it between your team with Docker. This is just a personal opinion, but I think downloading all the exact library versions you need once and putting them in a Docker image will save your team some pain in the future.
If that was true, then so would Valgrind have, and Valgrind was in wide use at the time.
It was however more complicated than that. If my memory serves me right - OpenSSL had its own memory management.
Focus on C and follow up with an understanding of the underlying assembler code. Haven't played with Rust for about 1 year, so I don't how how mature it is these days.
Also have an idea of how to debug the code. Debugging on embedded is very different to debugging a web/js app.
Low-level systems programming is hard and can be a bit boring at times. I agree that debugging is very different from debugging on higher level systems. Most of the time an oscilloscope or an LED is the best debugging tool. Also, in well written low level code you tend to have more code making sure that everything is okay than you have actually doing stuff, which can be tedious. It takes a certain kind of personality to enjoy it.
[edit] I have the LDD and https://www.amazon.com/Linux-Kernel-Development-Robert-Love book. I got confused about the authors name :)
The author Robert Love has done a tremendous job on Linux Kernel Development book. Every aspiring "In-betweeners and Low-Low Level engineers" [0] should read this book. It's awwwwespiring.
And please read LDD too!
[0] https://news.ycombinator.com/item?id=18882486
I would never ship a product today without remote signed upgrade ability.
All devices that plugs into a windows PC can use Windows Update for example to upgrade firmware. Any device that sits at a remote location probably ought to have a GSM modem in (adding only $1 to the cost of the device) for tracking uptime and firmware updates. Any device which offers an API to other devices should have 'provide firmware update' as part of that API.
Obviously security is a concern with updates, but by signing update files, and making sure downgrades aren't allowed, the security benefits of being able to patch vulnerabilities outweigh the disadvantages of the device manufacturer being able to produce evil update files.
Those who say that your lightbulb/toaster/USB hub don't need automated software upgrades are naive. Software on them will typically consist of many libraries totaling perhaps hundreds of thousands of lines of code. Security vulnerabilities will be found in that code, and even if the security of this particular device isn't of concern, it can be used as a jumping off point to attack other networked devices or for data exfiltration.
What about the $1-2/mo network charge? Or battery life considerations? I've got sensors that run for 2-3 years on a CR123a battery. That power profile isn't going to support a GSM connection.
￡0.96 | DIY KIT GSM GPRS M590 gsm module Short Message Service SMS module for project for Arduino remote sensing alarm https://s.click.aliexpress.com/e/b4tPrIy0
Lots of providers are happy to provide worldwide service for free, as long as you pay $10 per gigabyte. I generally budget 1 check-in per day, of about 250 bytes, so a coin cell can easily power it for a few years with a total service cost for 2 years of just a few cents.
2. Learn about data structures and memory handling (pointers) https://www.geeksforgeeks.org/data-structures/
3. Try to understand how an OS works (e.g. linux) https://github.com/tuhdo/os01
4. Buy an Arduino or alternative with some I/O. https://www.arduino.cc/
Just make sure you don't buy the `international edition` of the book, it's unreadable due to the amount of errors in the exercises.
Then you can have "low level" as in close to the metal, for example drivers that talk to a hardware. That being said even these are (and should be written mostly) at fairly high level (especially in the user space) and it's only a relatively small part that has to mess with HW specifics.
And then of course there's some embedded IoT stuff that has also different layers of software although for some people working anywhere in the embedded stack would qualify as "low level".
Generally speaking you'll want to lean on a few core technologies and APIs such as POSIX (on linux), have a decent understanding of the von neuman architecture and a solid grasp of C/C++. Having a decent idea of how different subsystems such as network or TCP/IP stack works and such can be useful depending on the domain.
That's exactly my point as well. Well said - ha at least better than I would be able to explain myself.
http://www.brendangregg.com/
The things you talk to will fail, you should be prepared to deal with those failures.
Electrical Engineering is basically being able to read drawings and choose components for your application.
Signals I found fun, but I can imagine doing a digital application and this being difficult.
FMEA comes with the job IMO, but a good tool to have.
The pay sucks, but it is much more enjoyable work than web development. There's a lot less BS to deal with.
I really can't tell whether an entry-level systems programmer gets paid better than an entry-level web programmer. But it seems to me that in web programming, you hit a wall at about ten years, where more experience quits translating into more pay. In embedded, you can find at least some jobs where 30 years experience gets you more pay than 20 years experience.