If your domain of discourse is the computer itself or the functionality of the computer in the world (store/compute/ communicate) than it is system programming. Else it is application programming.
It has nothing to do with the actual programming language.
I think anyone's opinion on what makes something "systems programming" has a lot to do with what it meant when they learned the term. The quotes from Stroustrup and Alexandrescu, for example, very much call out language features.
There are languages that are clearly better suited for systems programming. You might call these "systems programming languages." Would you write a file system driver in PHP?
> There are languages that are clearly better suited for systems programming. You might call these "systems programming languages." Would you write a file system driver in PHP?
I wouldnt but that sounds like something we would all read about on HN.
I actually wrote a filesystem driver in Perl using FUSE for my final year of university. It seemed like it would be much less painful when it crashed, given that the objective was to prototype an idea rather than ship a product.
Your phrasing suggests (I might be wrong) that (a) the expected answer is no, in your optinion, and that (b) Python is a relevant factor. If that is so, I wonder if you would consider an equivalent program written in C to be systems programming.
(My personal opinion is that the whole discussion is moot, a "systems programming" category is ill-defined and not needed.)
My personal definition of systems programming a programming where one calls a lot of OS APIs directly. Which would make the AWS deployment Python script (or even C program) example not s.p..
> To summarize, what we call “systems programming” I think should be called “low-level programming.” Computer systems design as a field is too important not to have its own name. Clearly separating these two ideas provides a greater conceptual clarity on the space of programming language design, and it also opens the door to sharing insights across the two spaces: how can we design the system around the machine, and vice versa?
Alternate definition whose source I can't recall:
Systems programming is everything that isn't application programming.
Maybe it's time to update that with:
Systems programming is everything that isn't application programming or low-level programming.
Slightly more seriously, systems programming seems to me to be approaching the lowest levels, so including "not low-level programming" would give the wrong impression that it includes the really high-levels ones.
Maybe not the thread to post this, but I've seen too often "systems programming" be intertwined with arrogance and gatekeeping. As if these skills are special and inherently more difficult to pick up by developers without them. But tbh all software engineers are programming abstract machines at various levels. Systems work is just a different abstract machine to grok.
My education is in relevant skills (CompE), but the attitude I keep running into in this domain has really started to disgust me, even if it isn't everyone. When it's people I've seen in leadership, that's enough :/
There isn't one. The only source is the hivemind of systems programming elitism. I've seen it permeate every systems engineering shop I've encountered.
I get where you're coming from, but I don't see how me joining a company could cause its leaders and employees to exhibit the sort of arrogance and gatekeeping I'm describing.
My limited experience with bare metal programming tells me they are not. This is just a different abstraction layer to learn. Often low level abstractions are much simpler and less broken than high level programming. Even if something goes wrong, you can plug in a logic recorder / oscilloscope / led (!) and in minutes you know what's wrong. But this doesn't work if you have 10 semi-broken layers below.
I have 15+ years of experience in hl programming and almost none in low level, yet I feel like at home with bare metal stuff. And it is a lot of fun as well.
To some degree, yes. They allow to write software much faster, but in order to be truly proficient, you need to know a lot of what's in the layers below anyway. And when writing in a high level language, there are many more layers below you than when you write a device driver. So overall you have more things to keep in your head, and more things to learn. But if you do, you gain ability to move faster. So it is not necessarily worse, but just a trade-off.
Also, at a high level, you typically solve different, bigger/harder problems. Someone may use Python to build a statistical model, but using a simple language that even kids use at school, doesn't mean the task would be easy.
This is also reflected in salaries here where I live - embedded devs don't earn more than application devs.
Hah no they aren't. There isn't anything inherently harder about low-level software than webdev or web services. I'm pretty confident people from those domains could pick up systems engineering on the job if the employer is committed to basic teaching and onboarding (which you should be regardless of domain.)
Bjarne Stroustrup: "Systems programming came out of the field where you had to deal with hardware, and then the applications became more complicated. You need to deal with complexity. If you have any issues of significant resource constraints, you’re in the systems programming domain. If you need finer grained control, then you’re also in the systems programming domain. It’s the constraints that determine whether it’s systems programming. Are you running out of memory? Are you running out of time?"
These constraints lead to the level of complexity which many programmers are unable to deal with.
Another source of the complexity is hardware itself -- it is asynchronous and opaque.
By that definition, you can end up doing systems programming in any domain. Optimizing browser performance, optimizing Postgres throughput. All of that is dealing with constraints and synthesizing knowledge from the layers of abstraction your problem is built on. Hardware is just a different set of constraints and lower level programming runs into these problems more often.
But there's literally _nothing_ inherently special or impressive about low-level systems programming or programmers :) I'm pretty confident good problem solvers from the domains I mention above could learn to work on it as easily as anything else.
And my original comment was about a common _attitude_ I've observed that is the opposite of what I'm saying. And you aren't really making any points against that haha
That is exactly what article was arguing for. I'm not in the business of discussing semantics.
My point is that the bar in the 'systems programming' is indeed set higher than in many other domains.
I'm not a systems programmer, but I've seen competent in their own domain programmers taking on tasks in more complex domains and failing where I didn't and I've been myself on the failing side.
And please refrain from personal attacks, I myself love sharing knowledge with colleagues.
I don't see any personal attacks in my comment, outside of the fact that my original issue was with how a culture of people I've encountered carries itself.
I think it would be a unfortunate for people to start redefining what "Systems Programming" means because it is vague enough now as it is - and we don't have the luxury of being able to go back and update the term in the historic literature.
My understanding is that, in effect, "Systems Programming" is analogous to "Operating Systems Programming", or rather you might say (given the increasing size of operating systems these days) that "Systems Programming" is a subset of "Operating Systems Programming" that deals with the direct manipulation of memory in order to provide a useful abstraction for "user" software - remembering that most hardware interaction is via memory. Obviously, in order to do "Systems Programming" you need a "Systems Programming Language" that allows such direct access to memory.
These days, "Systems Programmers" would be those working on the Linux kernel, or microkernels, or graphics drivers (basically those who have complete access to memory) though those developing services one layer up might also consider themselves to be such (especially micro-kernal-based operating systems).
It is probably worth emphasising that "System" is a generic term that is used in a lot of different contexts. I think "Systems Programming" is used in the context of a "Computer System", while modern day cloud programming would be better described as "Distributed Systems Programming" as it is in the context of a "Distributed System" (though I've probably just annoyed a whole bunch of distributed systems researchers).
I think that what the author of the article is talking about is "System Architecture" (or perhaps "Distributed Systems Architecture").
> I think it would be a unfortunate for people to start redefining what "Systems Programming" means because it is vague enough now as it is
Indeed. When people call something "systems programming" when it is just e.g. server programming, it feels like when you are being told that "hacker" means the same thing as "cracker" because that's what most people think it means.
In my slightly angry opinion, "Systems programming" is one of those terms that are convenient because they are vague and look cool. The plural on "systems" is suspicious in itself. It's not just one system they are working on, but many. Impressive, isn't it?
The funny thing is, if you look up the definition of "system":
a regularly interacting or interdependent group of items forming a unified whole
... not so many people actually do that. I think programmers usually work on one thing, one component of a whole system (guess why you are part of a team?). I'm not sure about how uncommon the opposite is because I actually do that - I have worked at the driver level, OS configuration level and at the application level in order to make a product work - and I am not that much of an exceptional programmer.
I've primarily writing C++ for over 15 years. I don't think I'd ever heard the term "systems programming" until the last few years when Rust talked about it. I don't have a super formal CS education and most of my career has been in gamedev.
What is sytems programming? I don't know. I don't care either. It's an arbitrary label that is rarely used and doesn't affect anyone in any way.
Systems programming concerns the code between hardware and the application programmer. So pretty much kernel programming.
Application programming concerns the code between kernel and the ordinary user.
Essentially, systems programmers write software that application programmers can interface with. Application programmers write software that ordinary users interface with.
It gets fuzzy at the edges like most definitions, but that's how I think about it generally.
I've always thought of compiler programming as systems programming. I guess it fits your definition of being something application folks interface with in addition to kernel.
One litmus test that a former coworker applied was "how resource constrained is the application"?
I think conceptually this is a better test, more so than say if you have access to a raw pointer - because more important parts of how to handle core issues like scheduling, storage management, concurrency, parallelism and distributed computation manifest themselves in many forms.
I started my career doing C programming in a OS / kernel environment. In retrospect a lot of pieces of applications I've worked on are what I think of as systems, and many parts of kernels are more applications than systems.
How resource constrained is the Linux kernel when running on a modern x86_64 machine with 16 GB of RAM? Not very. So is Linux kernel development for this platform not "systems programming"?
If you're going to say something like "but the kernel should be fast when servicing syscalls", you are right, but that is not a resource constaint, it's a user requirement. LibreOffice should also be fast when reacting to my keypresses, but I hope we agree that LibreOffice development would fall into the "applications development" camp, if we have to put things into bins.
The truth is, we don't have to put things into bins. There is no need for a muddy, fuzzy term like "systems programming". If you're doing "kernel programming" or "embedded systems programming on resource constrained microcontrollers", you can just say that directly.
49 comments
[ 3.6 ms ] story [ 115 ms ] threadIt has nothing to do with the actual programming language.
I wouldnt but that sounds like something we would all read about on HN.
Personally I use go for everything.
Here is some stuff to get you started,
"The Cedar Programming Environment: A Midterm Report and Examination "
https://pdfs.semanticscholar.org/33ff/212245ff8af0359e911883...
http://www.projectoberon.com/
http://www.ocp.inf.ethz.ch/wiki/Documentation/Front
https://www.whoishostingthis.com/resources/modula-3/
http://joeduffyblog.com/2015/11/03/blogging-about-midori/
https://www.infoq.com/presentations/csharp-systems-programmi...
(My personal opinion is that the whole discussion is moot, a "systems programming" category is ill-defined and not needed.)
Alternate definition whose source I can't recall:
Systems programming is everything that isn't application programming.
Maybe it's time to update that with:
Systems programming is everything that isn't application programming or low-level programming.
;)
Slightly more seriously, systems programming seems to me to be approaching the lowest levels, so including "not low-level programming" would give the wrong impression that it includes the really high-levels ones.
My education is in relevant skills (CompE), but the attitude I keep running into in this domain has really started to disgust me, even if it isn't everyone. When it's people I've seen in leadership, that's enough :/
It's because they are.
I have 15+ years of experience in hl programming and almost none in low level, yet I feel like at home with bare metal stuff. And it is a lot of fun as well.
As for the systems programming please see my answer to a sibling comment.
And you misuse this word, try googling what it really means.
Also, at a high level, you typically solve different, bigger/harder problems. Someone may use Python to build a statistical model, but using a simple language that even kids use at school, doesn't mean the task would be easy.
This is also reflected in salaries here where I live - embedded devs don't earn more than application devs.
Bjarne Stroustrup: "Systems programming came out of the field where you had to deal with hardware, and then the applications became more complicated. You need to deal with complexity. If you have any issues of significant resource constraints, you’re in the systems programming domain. If you need finer grained control, then you’re also in the systems programming domain. It’s the constraints that determine whether it’s systems programming. Are you running out of memory? Are you running out of time?"
These constraints lead to the level of complexity which many programmers are unable to deal with. Another source of the complexity is hardware itself -- it is asynchronous and opaque.
But there's literally _nothing_ inherently special or impressive about low-level systems programming or programmers :) I'm pretty confident good problem solvers from the domains I mention above could learn to work on it as easily as anything else.
And my original comment was about a common _attitude_ I've observed that is the opposite of what I'm saying. And you aren't really making any points against that haha
My point is that the bar in the 'systems programming' is indeed set higher than in many other domains.
I'm not a systems programmer, but I've seen competent in their own domain programmers taking on tasks in more complex domains and failing where I didn't and I've been myself on the failing side.
And please refrain from personal attacks, I myself love sharing knowledge with colleagues.
I don't see any personal attacks in my comment, outside of the fact that my original issue was with how a culture of people I've encountered carries itself.
Perhaps I misinterpreted your comment.
My understanding is that, in effect, "Systems Programming" is analogous to "Operating Systems Programming", or rather you might say (given the increasing size of operating systems these days) that "Systems Programming" is a subset of "Operating Systems Programming" that deals with the direct manipulation of memory in order to provide a useful abstraction for "user" software - remembering that most hardware interaction is via memory. Obviously, in order to do "Systems Programming" you need a "Systems Programming Language" that allows such direct access to memory.
These days, "Systems Programmers" would be those working on the Linux kernel, or microkernels, or graphics drivers (basically those who have complete access to memory) though those developing services one layer up might also consider themselves to be such (especially micro-kernal-based operating systems).
It is probably worth emphasising that "System" is a generic term that is used in a lot of different contexts. I think "Systems Programming" is used in the context of a "Computer System", while modern day cloud programming would be better described as "Distributed Systems Programming" as it is in the context of a "Distributed System" (though I've probably just annoyed a whole bunch of distributed systems researchers).
I think that what the author of the article is talking about is "System Architecture" (or perhaps "Distributed Systems Architecture").
Should we talk about this as programming at all?
Indeed. When people call something "systems programming" when it is just e.g. server programming, it feels like when you are being told that "hacker" means the same thing as "cracker" because that's what most people think it means.
In my slightly angry opinion, "Systems programming" is one of those terms that are convenient because they are vague and look cool. The plural on "systems" is suspicious in itself. It's not just one system they are working on, but many. Impressive, isn't it? The funny thing is, if you look up the definition of "system":
a regularly interacting or interdependent group of items forming a unified whole
... not so many people actually do that. I think programmers usually work on one thing, one component of a whole system (guess why you are part of a team?). I'm not sure about how uncommon the opposite is because I actually do that - I have worked at the driver level, OS configuration level and at the application level in order to make a product work - and I am not that much of an exceptional programmer.
What is sytems programming? I don't know. I don't care either. It's an arbitrary label that is rarely used and doesn't affect anyone in any way.
Application programming concerns the code between kernel and the ordinary user.
Essentially, systems programmers write software that application programmers can interface with. Application programmers write software that ordinary users interface with.
It gets fuzzy at the edges like most definitions, but that's how I think about it generally.
I think conceptually this is a better test, more so than say if you have access to a raw pointer - because more important parts of how to handle core issues like scheduling, storage management, concurrency, parallelism and distributed computation manifest themselves in many forms.
I started my career doing C programming in a OS / kernel environment. In retrospect a lot of pieces of applications I've worked on are what I think of as systems, and many parts of kernels are more applications than systems.
If you're going to say something like "but the kernel should be fast when servicing syscalls", you are right, but that is not a resource constaint, it's a user requirement. LibreOffice should also be fast when reacting to my keypresses, but I hope we agree that LibreOffice development would fall into the "applications development" camp, if we have to put things into bins.
The truth is, we don't have to put things into bins. There is no need for a muddy, fuzzy term like "systems programming". If you're doing "kernel programming" or "embedded systems programming on resource constrained microcontrollers", you can just say that directly.
Specially jarring because it's out of nowhere. That conclusion does not follow from everything that comes before.