run0, which behaves like sudo, but works entirely differently and is not SUID. Run0 asks the services manager to create a shell or command under the target user’s ID, creating a new PTY, sending data back and forth from the originating TTY and the new PTY.
Is that even a problem? Any program can shell out to sudo, hence why you shouldn't set NOPASSWD in sudoers. Polkit takes in a request on an unprivileged interface, that request is evaluated in privileged code against the set of privilege rules, and then passed the proper capabilities if the rules allow. This includes a mechanism where it can, if desired, prompt a user to enter a password etc to prevent a rogue program silently acquiring root. But even in the worst case, the rogue program is not going to acquire any capabilities that you would not otherwise have as with sudo, and the breakpoint between privileged and unprivileged code is (in theory) more tightly defined and controlled.
I'm really starting to hate the sub-community in Linux that tries to constantly change it.
I don't want to learn a new network config alternative with every update (Ubuntu changed its net config tool again with 24.04). I don't want an immutable os. I don't want to learn to write new config files. I just want to do what I've been doing but with new packages. If there's a problem with something, just fix it. Don't throw out the whole thing.
I moved to FreeBSD and am happy for its reluctance to change. If there is any, it's usually offering something genuinely new to me as a feature and to boot I only need to learn about it if I need it.
Hardware support is much lower but it's worth it IMO. I had the same irritation with macOS. Every release breaking something essential that was part of my workflow and i didn't want to change. Eventually I did change but away from Apple.
To help myself to understand others better, I made some efforts to look and keep looking outside my bubble.
That's not simple though- humans tend to think THEY know better for all others while it's often not.
Say you don't want to have immutable distros, but many want. Android or Talos or even OS for network switches are nice examples.
Linux based solutions and products related community is wide and to keep up with evolution one needs to adapt and adopt changes.
We already see what happens for immutable/stale systems like FreeBSD - even TrueNAS is abandoning it, dying as predicted by experts since 2008-2010.
On other side - nothing stops you from keep using sudo or even choose disto vendor which doesn't use systemd, there is a choice, not thing set in stone.
Say I don't plan to switch to run0 in any near future - first it's not gonna be in any LTS disto, 2nd - it seems to be lacking LDAP based rules and very much likely other important features.
I really think you need some time away from a PC. You seem to be taking this incredibly personally. You can feel how you feel about the general public, but knowingly and willfully staying in a bubble is only detrimental to yourself. You will ultimately become what you hate. A person unwilling to move with the current change of society.
> A person unwilling to move with the current change of society.
Well if society goes towards fascism like the Netherlands, then yes I'm very proud of not moving along with that.
The best remaining option is to live side by side with those people and interact with them as least as possible, and hinder them as much as possible in their attempts at legislating, like using the first chamber to block things. Mind you, this is already the status quo in the US where the democrats and republicans use whatever means at their disposal to undermine the other.
We didn't really have this phenomenon in the Netherlands because we don't have this "the winner takes all" kind of political system but parties need to collaborate. But the rise of the extreme-right makes it impossible to cooperate any longer.
This is the main reason why I'm still on Slackware. Pat is keeping the same thing for the last 20-30 years. Sure, he had to introduce some stuff, like NetworkManager or PulseAudio, to keep up with the latest software versions, but every major change is postponed as much as possible. Hell, even systemd is not there yet, and I'm pleased about that.
I love it when new and better ways are found to do things.
I love it that Linux is constantly improving and moving forward.
I’m willing to accept along the way some things seem to be mis steps (I’m looking at you snap packaging).
I love it that improved network configuration systems are being adopted because network configuration is a pain.
I love systemd and when new stuf comes out from the systemd project I think “gee I’m glad finally someone is taking a wholistic look at and fixing that messy inconsistent evolved corner of Linux and replacing it with well thought out powerful and integrated solution.”
Bring on the change, change is the best thing about computing and software. I own vintage computers but wouldn’t want to live there.
I’ve been running Ubuntu boxes in prod since 12.x — there is no “improved” way to put an ip address on an interface besides writing something to a file in /etc, but every update this file changes, or it’s format does.
It’s bullshit and I wish it would end.
Alas, keeps the consultant bucks flooding in when we have to rewrite a load of cfgmgt to go to 24.04 I guess..
I just switch every system to systemd-networkd immediately. The same .ini sytax as for service files, and dependencies are easy to handle, e.g. on one system I have two physical Interfaces eth0 and eth1, I want two vlans on eth0, and then bridge one of those vlans with eth1 and then run a DHCP client on that bridge but at the same time assign an additional IP address. This is dead simple to describe with one .ini file per vlan/bridge. Seriously the first time I feel like I'm not fighting an archaic config syntax, fixing up crap in some post-hook.d script, or give up entirely on any config language and just have a convoluted script setting up everything manually.
It's also easy to explicitly express "weird" stuff like "run DHCP client and use all the config options except the default route". Seriously a couple times I needed to do dumb shit and was like "there's no way they let you do this" but no, there's a way to do it.
And I'm pretty positive the config files will stay stable over the coming years and any new networking features will get appropriate config options in newer versions.
Eh, I could. But I'm quite sure that the method of switching to systemd-networkd also requires the same if not much more maintenance as simply changing the ifconfig template every few years...
I really don't care about interface configurations that much. It's an annoyance, but one that's quite easy to fix. I think introducing yet another network configuration here isn't the answer, but my linux fleet just run k8s anyway and network config on the linux level is quite simple before we get into cillium/istio/etc :}
Sure, I made that choice at a time where after upgrading, Ubuntu suddenly wanted netplan, debian stayed with /etc/network/interfaces and I think fedora went with networkmanager. So I though f- it, they all have systemd so I'll give networkd a spin. And never looked back.
So maybe don't just switch out of the blue right now if you've got a working setup, but maybe keep it in mind for the next time. :)
> I love it when new and better ways are found to do things.
I do, too. But I also really hate it when those new and better ways make things worse for me. Systemd does that in a couple of important ways. There is even some network-related startup stuff that I can no longer make work automatically at all. For me personally, systemd is a regression, not an advance.
But I also recognize that the Linux world is not duty-bound to make sure it remains excellent for me, and I've pretty much given up on advocating for my needs in the Linux space. There's no point, particularly with the systemd crowd.
It's very fair to eschew change for negligible improvements.
But I've also seen the community defend terrible stuff just because.
Look at what happened with the init system. System V, fstab, etc was awful. Doing anything with a reasonable level of robustness was grotesquely obtuse and complicated. And yet it was "perfectly fine" to the greybeards. Alternative proposals were near zero.
I don't have a dog in the networkmanager/netplan fight. It could be that one is irrelevant; given history, I have a hard time trusting what I hear.
sudo has quirks for sure (which is why you see a number of alternatives).
Daemontools was pretty popular with greybeards, actually. But yeah, daemontools/runit/s6 and company have always been for handcrafted server setups, where the thing about init scripts and unit files is that they're a standard thing a package can supply and have work out of the box with minimal tweaking across distros.
Any serious challenge to systemd nowadays is probably going to have to at least offer compatibility with it. No one is going to rip it all out and start over again (again).
I so want to ditch Linux for bsd, but hardware support, both for my current laptop and future pain in searching for compatible hardware is the only thing putting me off. And my server needs cuda for my AI shenanigans, so probably no bsd there either ...
That's funny. It's like there are two camps, conservatives, and progressives. ;)
Jokes aside, I just think that life is constant change and the programming industry is a good example of that. Coding practices have improved a lot in the last few years and will continue to improve with new knowledge and new technology. Sometimes it's better to start anew from scratch than trying to adapt old code into new practices.
Btw this is not a young whippersnapper saying this. My first IT job was on FreeBSD and OpenBSD. I was a full-time FreeBSD user from 04 to ~10.
And I remember exacty this gripe back in 2011 when Debian was using one network config, RHEL another. Today I actually enjoy the progress made with systemd, and I'm that annoying co-worker who will give you crap for disabling SElinux.
Well, some things are improvements. But many are not, and are just changed for the sake of it.
I'm a fan of SELinux (and the similar mandatory access control on BSD) because it gives strong security but the user or admin keeps control. This is a much better solution than things like immutable OSes where the user can't control anything and just has to trust the developer of the OS.
What complaining. There weren't paved paths before. Whatever one person learned was different from how anyone else did it. Few of the tools had anywhere near the essential capabilities, serving o ly some tiny niche of the use cases in some tool specific limited way.
Look at all the different netsevs supported by systemd-networkd. https://www.freedesktop.org/software/systemd/man/latest/syst... This is a huge list of tools that required a huge assortment of tools to do before, few of which had even part way decent management & fewer still had good init scripts. You used to have to learn from 0 each time, with each tool. Nothing was alike, nothing was as capable, nothing was integrative.
Don't listen to these complainers, for God sake. Your life is too short to get pissed off about well built work together featureful tools being built in a mono-repo fashion.
I'm so tired of the sabotage, so tired of broadscale general refusenik attitudes. This post is absurd. There was nothing to learn before, everything was 100% special snowflake & distinct. None of it was great, all of it was limited. Systemd mono-repo is built of many small pieces, but they couple together and are 100x more learnable. What you learn today will work across whatever system you run across. It's such a a better world, and these "pry it from my cold dead fingers" attitude can keep to that path for all I care, but I wish they weren't poisoning kinds with absurd incorrect negativity & being such magnets for disdain. The world today is fantastic & you rarely see these folks with even an iota of appreciation for how good we have it, never a drop of balance. But hate sells, & unites, powerfully.
But I do. So I use one, and contribute to a project that tries to create one. Am I a part of some sub-community that wants constant change? Or do I just have an unusual use-case and want to support it?
The beauty of OS is that anyone can decide which tool to use, contribute to it, and even fork it.
> I don't want to learn a new network config alternative with every update (Ubuntu changed its net config tool again with 24.04).
That's really just Ubuntu's fault. Between upstart, Unity, netplan, and snapd, Ubuntu likes to go off and do its own thing for a few years before coming back to what everyone else picked in the first place
I was really hoping the next sudo replacement would borrow heavily on root as role[0] (if not being root as role). Feels like a missed opportunity to not use capabilities.
Capabilities aren't guaranteed to be present, and in a lot of high-security situations aren't available (though obviously you could say that about sudo too)
"Liblzma ends up dynamically linked to sshd because of a
systemd-related extension added by many Linux packagers that pulls
in liblzma as an unrelated dependency."
"openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma."
> - Linux packagers decide to patch sshd to use libsystemd for a notification, that could have been trivially done without this library.
Why was that? Would that "trivial" approach have broken the next time systemd made one of their incompatible interface changes, perhaps? Was using libsystemd the kind of thing the systemd maintainers recommended?
> And therefore, systemd is insecure?
Systems with systemd had a vulnerability that systems without systemd did not. So it certainly seems like systemd-the-system (not necessarily systemd-the-unix-process) is bad for security.
You're not making a very good point here. A lot of packages have a transitive dependency on liblzma - for example everything that depends on libxml since that depends on liblzma https://packages.debian.org/sid/libxml2. LZMA is a pretty widely used compression algorithm, I'm absolutely certain there's other juicy targets that depend on liblzma.
The reason why "only" sshd on debian/ubuntu was affected is that the attacker chose to tailor their exploit to those systems. Systemd was the vehicle, debian patching opensshd was what made this specific incarnation of the attack possible, but essentially, both trusted a widely used library.
> A lot of packages have a transitive dependency on liblzma - for example everything that depends on libxml since that depends on liblzma https://packages.debian.org/sid/libxml2.
Sure. But security-critical software like SSH would certainly think twice before bringing in such a huge and complex dependency as an XML parser.
> I'm absolutely certain there's other juicy targets that depend on liblzma.
You could probably make a system package manager (which has obvious reasons to depend on a compression algorithm) do something nefarious. But that would be a more complex chain of exploitation with more chance for things to go wrong. Most security teams put more attention on security-critical parts like SSH, and I think most people would agree they're right to do so.
First, getting rid of setuid (I guess you'd have to get rid of the whole thing, not just the permission bit) is not the same as making systemd an integral part of the OS.
Second, when even the package maintainers can make such "trivial" mistakes, something is wrong. You'd expect a component such as systemd to be much more trustworthy than some random library.
I'm not arguing against systemd, just that it seems to grow and grow, and is not the correct place for security. It security is obviously broken.
> First, getting rid of setuid (I guess you'd have to get rid of the whole thing, not just the permission bit) is not the same as making systemd an integral part of the OS.
It absolutely is. sudo allows you to execute code as another user. If you want to do that without giving sudo itself administrative privileges, this has to be done through the service manager, which creates a completely new, elevated process and handles communication with that. This is how it should be done (and BTW, this is pretty much how also the new sudo for Windows works). Now Lennart for some reason prefers systemd as this service manager - you might disagree with that choice, but then come up with a better one.
> Decoupling/single-reponsibility is sort of lesson #1 in software engineering.
Well said. What makes you think systemd does not do this? Have you ever even looked at systemd in any amount of detail? Do you think it is one big binary running as PID1 doing everything?
Package maintainers of a distro can do absolutely anything to a package. With zero input from upstream developers. Some distros have more tradition for patching software than others. An upstream like systemd (or openssh) can hardly be blamed for what others do with their software.
If it was "obviously" broken why was the xz backdoor such a shock to everyone? Do you personally audit the library dependencies of every tool you run, including core servers that come with your distribution? I think people don't do this.
Also, even before the backdoor was discovered, the systemd team were making libxz be dynamically loaded only in the cases where it was needed which would have killed the backdoor dead. There's some evidence that this might have actually caused the backdoor to be sped up and hence led to its discovery. Claims that systemd has bad security have to explain why it was already implementing practices that would have blocked the xz backdoor without it even being discovered. That seems pretty decent to me.
The point is that (even) the systemd maintainers do not vet their dependencies. As an attack vector, it is the (second?) highest level, yet they did not assume the responsibility. Everybody silently assumed they did, hence the shock.
> Claims that systemd has bad security have to explain why it was already implementing practices ...
No, they don't. It doesn't take away the fact that they did not check xz, and probably only few of their other dependencies.
That would be irresponsible and strengthen the case against run0 as a replacement. And let's be honest this time: the argumentation is that it's there to replace sudo.
> It doesn't take away the fact that they did not check xz, and probably only few of their other dependencies.
Xz is also used by Debian's package manager (both dpkg and apt). Or tar. Or the Linux kernel. Or... It already was a system library on Linux systems before systemd started using it as well.
Your post reads like you have no real idea what Xz is and how it is used.
Btw, do you think the Linux kernel developers are also clueless for using Xz?
Yes. You have literally just described the way it is insecure. It bundles a large amount of functionality under a single system, and therefore anything using that functionality is at risk. You seem to be suggesting that Systemd would be secure if you didn't use it, which is obviously fallacious. Anything is secure if you don't use it. Systemd offers this functionality, and did it in an insecure way. You cannot blame users for that. Saying that people shouldn't be using a certain part of Systemd is really the same as saying that part shouldn't exist to begin with. The conclusion is obviously that Systemd should be smaller to decrease the chances of things like this happening.
LZMA is a widely used compression protocol. The kernel uses it. xz - the compression tool that was affected gets used by the kernel build makefiles - they reference it in the build docs https://docs.kernel.org/staging/xz.html. It's absolutely fair from systemd to have this dependency and to use the trusted library that the most fundamental part of the underlying OS uses.
It was purely the attackers choice to leverage the exploit via systemd instead of injecting code in the kernel at build time.
Your speculation on what is right and what was fair is of no consequence to me. Their error was not simply using a compression library, it was creating a large central point of failure. If Systemd was smaller, it would not have caused this error. By being large, it made itself vulnerable. It made itself a target. It made other software insecure. These facts are inescapable. And you cannot justify this by simply saying they didn't do anything wrong right before the attack, or that packagers are to blame, or that other software might also be vulnerable, or anything else that doesn't address the core of the issue: Systemd created the circumstances needed for this to happen. They were warned of he risks they created, and chose to do so anyway. Now those risks have been made manifest – the inevitable result of a fundamentally flawed design.
Why is this an argument specifically against systemd, rather than all large software projects?
Linux kernel, gcc, glibc - all bundle "a large amount of functionality under a single system" - does this make their design fundamentally flawed as well?
I think a micro-kernel architecture would be better in many ways, security being just one of them. With GCC, it couldn't really be separated into logically distinct modules any more than it already is. All of the constituent parts essentially use the full functionality of the base compiler part, so there is little to be gained separating them. It's not as if the C compiler from GCC, could for instance, be written in such a way that it doesn't depend on GCC. GLibc is a large implementation, but the library itself doesn't necessarily need to be large. There are some very small libcs out there.
On the whole, I do not like monolithic software projects, but I can accept that they are necessary or beneficial in some cases. Systemd is simply a much bigger target than these other things because it is an especially bad example. It has many components which are only tentatively connected. It is also more fixable. Alternate init systems are used much more widely than something like Hurd to replace Linux. The laptop I'm typing this from runs GuixSD which ships without Systemd and I can hardly notice the difference. I doubt a different kernel architecture would provide such a seamless experience.
Given how common LZMA as a compression algorithm is, are you certain that your init system of choice doesn’t use it in any way? It’s a very common algorithm in network protocols, it’s a direct dependency of libxml, … - and if any part of your init system uses LZMA, then it just happens not to be affected because the attacker chose to target one specific system.
The attacker had essentially full control over a very fundamental library in the Linux ecosystem. They could have leveraged that in a hundred ways.
The attacker chose to target a very specific component of a very specific system. It was their choice, not some sort of technical requirement that made it impossible to use a different attack vector. Just as they chose not to target other Linux distributions that use systemd.
You’re essentially saying “I was safe because the attacker chose to ignore me.” That worked well this time, but it’s a pretty dangerous stance.
When a hacker chooses to attack something, that isn't random. They had to look at a lot of different pieces of software, and decide which would be the best to attack. The choice in this case was Systemd. In other words, if you are looking to do malicious things, Systemd is helpful.
Now I want you to imagine that every piece of software has a score, which tells you how useful it is to hackers. Systemd has a high score, and hence it was chosen for this attack. Your argument is that: there are other pieces of software with a high score so it's fine for Systemd to also have one, since without it there would be other things to attack. My argument is that we should reduce the amount of software that has a high score. Do you think my reasoning or your reasoning will lead to a more secure ecosystem?
The complaint is that the code architecture of systemd results in security problems for their _customers_.
The inclusion of a library to send notifications shouldn't have external dependencies, it shouldn't need them. The library is included in the customer's codebase at execution time, so it is a hole in the customer's security model. This immediately opens a supply chain attack vector (which is what we saw).
This is being taken as evidence that they shouldn't have responsibility for truly security sensitive code - the replacement of sudo.
Some of this is a long-term dislike for systemd and some representative bias. However, Systemd has missed the opportunity to make their client libraries safer.
Personally? I wouldn't have thought to limit the dependencies of my client libraries. It's a growth and project age thing. One moment you're on one side of a line, the next you have to skill up and do things differently.
Using systemd as intended shouldn't result in security holes for their customers.
"Systemd is bad because Debian patched OpenSSL to add an unnecessary dependency on a systemd library" is just not a good argument, sorry. Nothing about the "code architecture" of systemd caused the xz issue.
And again, Poettering's complaint with sudo is specifically about it being a SUID binary, so this discussion isn't even related to the thing you're accusing them to be hypocritical about... SUID is more than just "code running as root", it's "code running as root in an attacker-controlled environment". That last part is the important one.
By "code architecture" I mean the packaging of their client libraries.
This attack demonstrates that they should be tightly focused and have minimal downstream dependencies.
I don't have any experience with systemd, but typically, people will bundle _all_ of their client libraries into one .so and say "use that".
However, what needs to happen is there should be multiple .so's, one for each sub-API. At least there should be libraries for frequently used shim code (like notifications seems to be).
Then systems that need to push information don't need to pull in the dependencies for other parts of the overall systemd interface.
I can't think of a reason why pushing local notifications would require a compression library. The notifications should be information heavy already, so not very compressible.
As I said, it's a growth and project age thing. One moment you're on one side of a line, the next you have to skill up and do things differently.
How the team responds is what is important. That is why people are objecting to the SUID work. Not because sudo isn't a hole, but that the systemd team isn't considered responsive enough to take it on.
I know I'm taking lessons from this to my work. It's an unpleasant mirror for me to look in.
The complaint might be valid. The solution, to shoehorn yet another functionality on systemd will in no way reduce complexity or attack vectors, merely shift them, again, like with all systemd solutions.
What difference does it make if it's part of the systemd project or not? Do things suddenly become a more problematic attack vector when they're organizationally part of the systemd project instead of the sudo project?
here's a new tool in systemd, called "run0". Or actually, it's not a new tool, it's actually the long existing tool "systemd-run", but when invoked under the "run0" name (via a symlink) it behaves a lot like a sudo clone.
But they already ship pkexec together with systemd anyway via polkit, why are they again reinventing a wheel they already reinvented?
Unit files are a neat concept I don't want to miss again, but everything else done by Lennart seems to be an inceasingly stupid mistake born from hubris.
AFAIK privileges are an area that has an easy problem statement ("execute this command with that capability") and is fiendishly difficult to execute in practice. `sudo` alone has weird bits to set in the filesystem, magic users and all sorts of unhelpful implications - and it doesn't even lead to any particular security for single-user systems. Same-user code is a scary enough place to be running untrusted code.
Those sort of problems sound like the sort that get a lot of attempts which run into the complexity wall and halt. I think Amazon has one of the best implementations of a privilege system I've used and it is horrible.
Because pkexec has the same problems as sudo: it's a SUID binary. As Lennart says, the goal is to eventually get rid of SUID binaries altogether, as they are an inherent security risk. Replacing sudo with pkexec would not change much. In fact, pkexec has had its fair share of local root exploits over the past few years.
This seems like it won't break anything except really exotic scripts, I think it will probably be a good thing for at least the main target audience of systemd, id imagine it might somehow suck for others though.
Uh, depending on exactly how it's implemented, it could break a lot of things.
If all you are using sudo on is a personal (i.e. single user) laptop/desktop to install packages, this (along with other things like pkexec or doas) would seem to present no issues (and personally, from what I can see, I'd be happy to run `run0` on my personal systems!), but sudo does significantly more than that, as is called out by the systemd devs in the linked post https://mastodon.social/@pid_eins/112353324518585654
sudo supports not just LDAP (for multi-user systems), but include various levels of logging (including logging stdin and stdout of commands), apparmor and selinux profiles, the BSD and linux audit subsystem and more in a simple, easy to read and edit config format (this is just me reading from the `sudoers(5)` man page).
Whereas it seems `run0` won't have a `sudoers` file, but will instead be configurable (implicitly) via polkit, which uses JS to write policies (which I'd view as a much harder and error-prone system than the current `sudoers` format). It's not clear to me how much of sudo is tied to SUID vs. having a separate daemon (i.e. how much would have to be ditched vs. how much could be mapped over).
I do feel this is systemd moving away from traditional multi-user unix systems to being a single-user system (targeting the laptop/desktop case, or where sys-admins are the only users of the system, and it's basically a container host).
> If all you are using sudo on is a personal (i.e. single user) laptop/desktop to install packages, this (along with other things like pkexec or doas) would seem to present no issues
Of course, once distros start to say 'wait, why are we shipping 3 different privilege escalation systems again? Systemd is needed for starting units anyway, so lets just drop sudo and su'
sudo is not installed by default on many distros (e.g. if you give a root password on Debian install, then sudo is not installed, but if you don't, it is so you can admin the system), so unless systemd introduces something to break sudo, I don't think it'll go away (it just may get bumped down the list of important packages). su seems to come from https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/, which is all the truly core stuff, and I don't see su disappearing from there.
Not all distros ship pkexec with polkit. Polkit runs fine without pkexec present. This is kind of a non-suid alternative to pkexec. It's really more similar to a local-only ssh though.
“Systemd” and “expand” used in the same sentence….. all the systemd haters will be triggered like it’s the national rifle association shooting carnival.
In many ways systemd has actually become the operating system. It’s so pervasive that it certainly is more deserving of naming rights than gnu. “Systemd/Linux” makes more sense than “gnu/Linux”
People have been memeing this for a long time but if you know anything about the project you would know that one of systemd's explicit goals is to use the full capabilities of the Linux kernel to hell if it's not portable to other kernels.
I am looking forward to the day systemd will implement everything a typical Linux system needs. No more systems greater than the sum of their parts, just a Half-Life 2-style Combine of Microsoft sponsored Poetteringware, running a monolythic system on top of a monolythic kernel, until systemd rewrites the Linux kernel as well. The future is bright! \s
Seriously though, this seems to be a decent replacement for sudo if it works as seamlessly as it's described in the article. I still prefer the doas line of approach that simplifies the tool as much as possible, but I see the value of having such an important tool integrated into the existing system tooling, especially if it already includes everything but the kitchen sink.
It either has to be SUID or it has be a daemon running as root (or with enough caps to make the difference not matter). Adding a needlessly verbose configuration ecosystem doesn't change that. I imagine there's going to be some cool stuff this can do with homed and userctl, but it's not like the fundamental problem of "this program can grant root privileges" can ever go away.
The problem is not "this program can grant root privileges", it's that the setuid bit sucks.
Linux processes inherit a lot of state from the parent which means it's absolute hell to make a secure setuid binary. And at any time the Linux kernel can add a new feature which will be inherited by a child process, but that the process can't defend against because it wasn't even a thing when the code was written.
Running a binary at all also goes through a complex set of initialization steps a lot of programmers barely know exist, let alone are able to understand fully.
The tool doesn't try to do away with root, it tries to do await with the setuid bit. Meaning, "running a persistent daemon as root" is the intentional solution, and presented as the significantly better option for good security.
Just a reminder that there are plenty of systemd-less distros available. Also a reminder that those distros would have been safe from the nearly-solar-winds-level backdooring of Linux distros from XZ utils.
Not sure of the relevance of this comment, can you elaborate? Were you the one that caught it?
Our balls were inches from the bandsaw. Systemd made it possible to compromise SSH through an unrelated, single-maintainer lib that wasn’t even a dependency.
Edit: never mind, I see you are a systemd crusader.
It was in OpenSUSE Tumbleweed for a few days actually (RPM-based + rolling release + did the sshd patch). I was affected by it and it was fun watching the reliable ~100ms difference in `time /usr/sbin/sshd -h` with and without `TERM=foo`
Of course, the actual hacker was to blame, but systemd was implicated. The fact that the attacker was willing to settle for compromising just Debian and Red Hat systems indicated that they perceived the path from xz to libsystemd was the easiest way to effect the backdoor and that doing it any other way would have been too much work for marginally little gain (Red Hat and Debian systems being so common).
> Don't you find it ridiculous to blame the XZ backdoor on systemd, instead of the actual hacker?
This is a great argument against all computer security. If you believe in securing your computer, you're supporting hackers. Because if you ever believe that a lock has failed, you're saying the thief is innocent; that's how logic works.
Well, you conveniently ignored half of my point. Even without ssh depending on systemd (dependency introduced by distro maintainers, not systemd, mind you), a backdoor in xz can still exploit your system in a myriad of ways.
And secondly, I would say even if nobody is going to blame me, I would still secure my systems. Why? To protect my data of course.
Also remember that systemd-using distros like, say, Arch were also safe from the nearly-solar-winds-level backdooring because the backdoor targeted specific distros widely used as servers. Obviously the solution to security from backdoors is only using distros that aren't popular for servers
Wasn't the recent liblzma attack already exploiting the fact that systemd has its hands in pretty much everything? Wouldn't this expand further the attack surface of systemd and the systems that connect with it?
That's not a great summary of lzma. It was systems adding custom patch to ssh which used a systemd-related library which it didn't really need in the first place. It's a stack of issues that don't have much to do with systemd itself really.
But re. expanding the attack surface - unlikely. Systemd's primary purpose is to start processes with the right environment / permissions. systemd-run/run0 basically give you the tool to invoke that functionality with a terminal attached to it. That's smaller scope of extra code than sudo/doas deal with.
Isn't it a fault of systemd that libsystemd had a dependency on libxz? (because it implements too many things). It should have been possible to add the notification functionality using a tiny libsystemd-notify.
It's not a fault. They needed xz for some functionality and didn't want to split that library into multiple pieces. That's just a choice.
But either way, you could always do notification in a few lines yourself (probably as many as you needed to link that library in the first place). I've done multiple 3-line "implementations" in Python and Ruby in the past and never linked it for example.
sudo and su made sense when it was a multiuser time sharing system. You needed clear boundaries between each users of the system, and permission bits.
If I’m running on my workstation or desktop just let me run the damn thing. I don’t need an unprivileged user. On TempleOS you can modify the running system in ways you can’t on Linux.
Why do they have to do this? This is really, really stupid.
My issue isn't even that someone tries to replace sudo. That may or may not be a completely fine thing to do, depending on the state of sudo and what improvements can be made. But what makes me really upset is this completely unexplainable need to make everything part of one particular init system. There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
This is a completely insane way to develop software, instead of creating a new piece of software in a separate project they will force all their projects simultaneously onto all their users for absolutely no reason.
I am very glad to have jumped ship from systemd. It is particularly bad software created by a team of people who engage in very bad practices and a totally unhealthy view of software in general.
The single beat reason for doing this is to get a coherent complete system. This is what every other person here says.
You can not try to create a large coherent system and then tell people they shouldn't use that particular part. That is totally disingenuous. Systemd is DESIGNED to be an all or nothing deal.
And people are also telling you that tight integration is not a mandate. Having a bunch of stuff designed to work together does not mean that they're tightly coupled and can only work with their specific implementations.
We've all got confirmation bias, the trick is to be aware of it.
You aren't making any sense. All of the components of system D are separate programs, and yes they are maintained by the same general project and designed to integrate well with each other, but just because components are designed to integrate well with other components from the same project if they are present, that doesn't remotely mean that all of the components are required. And I mean, the proof is in the pudding, there are plenty of distros that do not use all of the components of systemd. Also, doesn't Linux have a long history of creating integrated suites of programs designed to integrate well with each other and used together? Like GNU?
The reverse is true: Trying to use a component like udev or logind or in the future, run0, will require the use of systemd. Either you use full systemd or you can't get any of the fancy stuff.
Gentoo folks have eudev or seatd but thats an uphill battle.
Why is this stupid? It’s just an option for how to configure a system that uses systemd to allow commands to be run in a privileged execution context without a suid binary. What’s wrong with having options?
Why don’t you propose a better solution? How would your non-systemd solution actually work?
> But what makes me really upset is this completely unexplainable need to make everything part of one particular init system. There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
The systemd developers are tying it to systemd because they are systemd developers. If somebody else made something like this, it wouldn't be tied to systemd. But somebody else hasn't made something like this.
Sudo isn't going to just go away (unfortunately). You can keep using your CVE-ridden setuid binary as much as you want.
> You can't use any GNU project without depending on GNU libc.
Of course you can. Unless they're depending on non-standard parts of libc, you can use any GNU project with other libc implementations (musl, dietlibc, ulibc etc).
I find this quite clever. Yeah yeah, "systemd is bad, it's a monopoly, yadda yadda".
Well, there's also a reason they are doing it, and many aspects of it (seatd, timesyncd, resolved, run), and more neat than cobbling together bash scripts. I like it
> But what makes me really upset is this completely unexplainable need to make everything part of one particular init system.
It is not unexplainable at all. In fact, the article explains it very well.
sudo allows you to execute code as another user. If you think about it, you could also replace sudo with ssh to localhost - just set a root password and allow root logins. Now, security-wise, this would obviously be a bad idea. Our current solution is to give the sudo binary itself administrative privileges, which is a slightly less bad idea, but still pretty bad.
Systemd already handles logins, so it is quite obvious that it could also handle this problem very well, and in fact, it already does: there is a tool systemd-run which you can already use. It will create a completely new process and will handle communication to it, just like ssh, but without the above downsides.
And I disagree with that. The 'sudo problem' is a good example why it makes sense to handle init and login by the same system (note I'm not meaning "same binary" here, in fact, they are separate binaries in systemd). The SUID approach for sudo has been a problem for decades, and it needs to go. I don't really care if it's done by systemd, or if we agree on another system like S6 that was mentioned in another thread - in the end, they work very similarly be replacing the old sudo with an IPC approach.
> systemd is a large beast -- no need to make it larger with unrelated things
I think I sufficiently explained why it is very much related.
> the Unix philosophy seems to be applicable here
Systemd very much follows the unix philosophy. It is not one big binary, but actually consists of dozens and dozens of tools communicating with each other through protocols.
> the result should be portable to non-systemd systems
People who just have an axe to grind with systemd really don't like when you point out that it's incredibly modular and follows the unix philosophy.
I think what people mean to say but don't have the words for is that systemd is an East coast school of thought project and folks prefer "worse is better" style tools.
> I think I sufficiently explained why it is very much related.
This:
| Systemd already handles logins
?
But I don't see why a bring-up/shutdown system should handle logins.
> Systemd very much follows the unix philosophy. It is not one big binary, but actually consists of dozens and dozens of tools communicating with each other through protocols.
That is fine. As long as this service (sudo replacement) is a standalone, separate daemon started by a systemd unit, I'm happy. If it's a core part of systemd itself then I'm not happy.
> Portable on what basis? POSIX?
Or Linux / glibc / musl, sure, why not. Some people (no, not me) want to run Linux w/o systemd. The point is that a sudo replacement service should be fairly portable to the universe of UNIX/Unix/BSD/Linux.
> Or Linux / glibc / musl, sure, why not. Some people (no, not me) want to run Linux w/o systemd. The point is that a sudo replacement service should be fairly portable to the universe of UNIX/Unix/BSD/Linux.
This fetish of "everything should just stick with libc and POSIX" needs to go. These standards have not evolved at all, they are decades behind and don't even remotely cover the necessary requirements for implementing a "sudo replacement service". Just stick with sudo then.
The article explains how this sudo replacement ties to systemd:
> But with one key difference: it’s not in fact SUID. Instead it just asks the service manager to invoke a command or shell under the target user’s UID.
I understand your frustration, but systemd isn't the first attempt to build an integrated system. It just happens to run on Linux. It isn't insane to develop software this way, from that perspective.
I was very surprised when I learned about it, but it does make sense (for s6 at least).
> There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
With s6, the idea is to replace a SUID binary with an IPC mechanism. That does make sense, since (parts of) the init system need to be running as root.
> I am very glad to have jumped ship from systemd.
> With s6, the idea is to replace a SUID binary with an IPC mechanism. That does make sense, since (parts of) the init system need to be running as root.
Because I'm pretty sure that most of the components are optional.
You did not even discuss the reasoning given for not using sudo to instead hop on your soapbox to say it's bad software with bad practices and that they are stupid.
It's annoying how in the the more surface level Linux communities there's 0 value in discussing systemd.
"1 million lines of code for PID0!"
The new thing is blaming systemd for that recent exploit even though distros were patching in the bug themselves.
People analysing the exploit determined that a new version of systemd was going to prevent the exploit vector so the exploit seemed to have been rushed out.
Isn't this just textbook FUD?
What I've noticed is over the years is systemd would have identified a gap in functionality.
Like systemd-homed having a solution for automatically encrypting home directory when the machine is suspended.
Is that a functionality that OSX has had for years? Yes.
But anti-systemd people will dislike it automatically.
Why are you bringing up random arguments I didn't even make?
No, I am a dedicated systemd hater ever since I spend over a month full time writing and debugging systemd services for work. Systemd (the init system) is just all around badly designed and executed, I have very little confidence in the developers and their technical abilities and their tendencies to expand into completely unrelated areas for seemingly no reason makes me quite concerned.
I wouldn't blame the xz exploit on them, it is very hard to call it their fault in any way. But I do think it is a symptom of a system which has grown far too thin and wide.
Because your post is the repeating cliches that are under every discussion about systemd.
You're essentially saying that the month you spent is enough for you to call it bad and the creators incompetent.
What qualifies you to make a determination like that?
There is never any actual technical reasons it's always about vague things like not adhering to UNIX philosophy, lines of code or it being badly designed (without any real architectural criticism)
This is an article about why they believe sudo isn't a good system. Where's your criticism of that from a technical / security perspective?
It's been about 10 years since systemd was adopted by Debian/Ubuntu/Redhat/Fedora etc.
Millions of deployments over the years. The companies that build and are paid to support for years with SLAs the operating systems are using it without issue.
>There is never any actual technical reasons it's always about vague things like not adhering to UNIX philosophy, lines of code or it being badly designed (without any real architectural criticism)
I did not mention the first two, so please do not pretend I argued that. For bad design look at transactions. That is really dumb and makes the system near incomprehensible. The documentation is bad, dbus is literally so bad they tell you not to use it without a wrapper. The terminology is very questionable and makes it hard to explain what a unit actually does.
But I don't even see that as the worst part. The worst part is that they fundamentally can't do basic software engineering, in the sense that they do not have a defined project scope. Everything is potentially a systemd issue and not once does anyone take a step back and say "maybe systemd" isn't the right place to fix that problem.
>This is an article about why they believe sudo isn't a good system. Where's your criticism of that from a technical / security perspective?
If you don't read my posts please do not respond to me. Look at the first post I made and carefully read it.
> If you don't read my posts please do not respond to me. Look at the first post I made and carefully read it.
I read your OP. It does not contain a technical / security criticism of run0. It's an angry, hand-wavey, vague rant against a project that took a design decision you apparently disagree with, but lacking any actual analytical evaluation of the thing up for discussion.
This sort of top-level post shows up on every single article that mentions "systemd", so you'll maybe understand why people tend to be dismissive.
>I read your OP. It does not contain a technical / security criticism of run0.
Yes, I literally say there is nothing wrong with the idea, so you going ahead and demanding I criticize the idea, is just absurd.
Really, this is completely bizarre. I even say that the thinking behind replacing sudo is fine, yet you are here complaining that I don't deliver technical arguments against something which I even told you might be completely valid to do from a technical perspective. Baffling.
Let me get this right: you see an article on a new thing, which you have no problem with, but have an angry rant in the comments section anyway? And now you're baffled by people's reaction to that?
I'm not sure there's much point engaging further, I hope you have a good rest of the day.
I can use GNU bash on NetBSD with no other GNU software installed. I can install GNU coreutils on Alpine Linux (complete with musl libc instead of glibc). In fact, it's possible to just install a single part of GNU coreutils but not the rest - ex. Alpine packages just sha512sum as https://pkgs.alpinelinux.org/contents?branch=edge&name=coreu... (not sure why). I don't think I've seen it done, but you could build a Linux distro that used glibc and gcc but no other GNU software (busybox coreutils and ksh shell, say). GNU has their own kernel, but is predominantly used on other OSs. They want to build all the pieces, but you can opt in or out of all of them, and they're all portable. In contrast, if you want to use, say, run0, you must run systemd as PID 1, you must use journald, and the whole stack only runs on Linux. So yeah, that is actually different.
> I think that portability is a deliberate anti-goal of systemd.
Yes, and that is one of the things I dislike about it. (In fairness, the list of things I like about it and the list of things I dislike about it are both fairly long.)
> No, you must run something on pid 1 that implements the spec, similar to how musl can be used instead of glibc - they both implement the same spec.
> Run0 expects pid 1 to behave a certain way, much like my web browser expects web servers to behave a certain way.
If there's only one implementation, then it's not portable. If a webapp uses a web API that only Chrome implements, it's not portable regardless of whether Google published a spec for their non-standard behavior. There are dozens of web servers and web clients that all speak HTTP, there is one systemd.
> But what makes me really upset is this completely unexplainable need to make everything part of one particular init system. There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
You should look at it differently, and then it'll make perfect sense.
systemd has long stopped being just an init system. It's a system tooling suite. When thinking about systemd don't think "PID 1", think "Linux New System Software Suite". It's a big umbrella project in the style of Gnome and KDE.
For example, systemd-boot is a perfectly normal bootloader that's just been systemd-themed. It has a "ctl" tool, has the same command-line aesthetics as other systemd-group tooling, and so on. It's not in any way dependent or even interested in the init system.
>systemd has long stopped being just an init system.
I think that this has always been the core criticism of anyone who objects to them. Besides systemd, the init system, just being very poorly thought out.
Riddle me this: can I create new mount files inside a systemd unit and have them activated to mount the locations specified? Do I need a daemon reload, for this?
This doesn't answer the question. Also generators are started very early, before other units have been started, so if your system is already running and now you want to generate those units, depending on the state of the other units, they don't really help.
>I don't think having an unit that generates units at runtime is an officially supported use case
Are you sure? Can you tell me how I would find out?
> This doesn't answer the question. Also generators are started very early, before other units have been started, so if your system is already running and now you want to generate those units, depending on the state of the other units, they don't really help.
That's the point. Any situation in which you have a system modify itself at runtime is a recipe for a headache. So you do your auto-generation first, then work from a stable state.
> Are you sure? Can you tell me how I would find out?
Aside from that this kind of recursion seems like a great way to get weird problems, and that generators exist for this exact thing, the whole design of systemd discourages this kind of trickery. Units are just supposed to start a command and little else.
Maybe somebody made an official pronouncement on this somewhere, but my personal take on this kind of thing is that it's a bad idea, anywhere, not just systemd.
>So you do your auto-generation first, then work from a stable state.
You can't do auto generation if that generation depends on the output of some units. Generators can not solve that problem.
>Units are just supposed to start a command and little else.
What a bizarre thing to say. No, that is not just what units are for. I think you are severely misinformed about what systemd is. Units are supposed to take care of ordering accept IPC, define how to handle failures and manage devices.
That comment alone makes me believe that you just do not know what you are talking about at all. It completely misses why systemd is designed the way it is and what it tries to accomplish.
> You can't do auto generation if that generation depends on the output of some units. Generators can not solve that problem.
True
> What a bizarre thing to say. No, that is not just what units are for. I think you are severely misinformed about what systemd is. Units are supposed to take care of ordering accept IPC, define how to handle failures and manage devices.
I mean that part of the point of systemd is that units mostly work out to ExecStart=/usr/bin/binary, and an unit isn't supposed to contain the arbitrary jank one can put into a SysV script.
IPC and the like is an explicit systemd feature, not something you improvise behind the scenes and then expect to work anyway.
> Units are just supposed to start a command and little else.
Units are not just for services.
From systemd.unit(5) man page:
> A unit file is a plain text ini-style file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a watched file system path, a timer controlled and supervised by systemd(1), a resource management slice or a group of externally created processes.
Instead of just hating based on assumptions it would be useful to actually familiarize with the thing you are critizising
> But what makes me really upset is this completely unexplainable need to make everything part of one particular init system
systemd is not init system, its and umbrella project for various core system components, which includes an init system.
> There is absolutely no reason to tie your new sudo replacement to systemd
well, in this case there is good reason to have it interact with service manager in general
> But with one key difference: it’s not in fact SUID. Instead it just asks the service manager to invoke a command or shell under the target user’s UID [...] Or in other words: the target command is invoked in an isolated exec context, freshly forked off PID 1, without inheriting any context from the client
strictly speaking it's not 100% coupled to systemd specifically, it most likely uses systemds d-bus API, which is part of their "portable and stable" APIs and as such could be implemented by other service managers https://systemd.io/PORTABILITY_AND_STABILITY/
> This is a completely insane way to develop software, instead of creating a new piece of software in a separate project they will force all their projects simultaneously onto all their users for absolutely no reason
The couplings between systemd projects are not that tight, you can pick and choose which parts you want, they explicitly are not "forcing all their projects" to consumers. The minimal systemd build has only init, journald, and udev, which is not exactly sprawling huge. All the rest of the projects are fully optional, and I believe quite many of them can actually work without systemd-init
Half the people here tell me that systemd is great because it tries to create an integrated complete system, the other half tells me I can just pick and choose.
Both can be true though, it can create an integrated system where everything systemd-* works together, but where you can replace any systemd-* you don’t like with something else
The point of an integrated system is that interactions between components enhance the system as a whole. This is obviously incompatible with an easy replacement of components.
This is literally just basic systems engineering. I don't even know what you are arguing here. The more tightly integrated a system is the harder it is to replace individual components.
You can have a well-integrated system without having tight coupling between components. Interfaces are a thing, and a high level of integration just means having a good collection of interfaces between well-defined conceptual components with well-defined capabilities. The actual software that implements those components is entirely separate.
I don't know what you are arguing. Replacing a component in a tightly integrated system means that the component has to be compatible to that tightly integrated system, meaning lots of assumptions and replication of functionality.
This means components have to tightly conform to the components they are replacing, which obviously decrease modularity as these components are harder to maintain for available system configurations.
This isn't about whether someone can rewrite a part of systemd, but whether you can freely mix and match, which tight integration works against. Obviously
The more tightly integrated a system is, the more involved the contracts between the components become. I seriously didn't believe that a single person here would disagree with that.
A component that depends on one simple interface is far easier to replace than one which depends on twenty complex ones. This seems like the most basic stuff.
You can replace most systemd components but in that case you have fewer features and probably more code overall. If you run the whole systemd bundle you get more features.
> systemd is not init system, its and umbrella project for various core system components, which includes an init system.
The point being made is that it directly depends on the init system part.
> well, in this case there is good reason to have it interact with service manager in general
I disagree! There's no reason to not have it be its own daemon with its own configuration and looser bindings to the rest of the systemd ecosystem (e.g. through dbus and other protocols). KDE applications do this a lot, where they take advantage of other KDE components if present. This is strictly a philosophy thing and not a requirement for achieving what he proposes.
> their "portable and stable" APIs
"we're portable if you reimplement our APIs", idk how this is an argument. They consistently make very little effort to be compatible with (or provide fallbacks for) what's already there. Having stable APIs is nice but there's a reason most of the interfaces on that page don't have alternative implementations: They solve questionable problems and provide no tangible benefits over the methods there were before. Yet projects feel compelled to hard-depend on them...
> The couplings between systemd projects are not that tight, you can pick and choose which parts you want, they explicitly are not "forcing all their projects" to consumers.
In practice they are. The primary reason why everything is shipped as a single project is because that makes it easier to make available in different distros (they just enable everything), allowing it to become the "de facto" standard since it's available everywhere. There's a lot of projects that solve some of the systemd tools' in sometimes better ways, that never see the light of day because they don't have a trojan horse to ride in with.
There's a reason why projects like elogind exist, because there's only a very select few systemd tools that work without systemd at all
It's not unexplainable. Any init by design is supposed to spawn new processes with the given environment, permissions, capabilities. When you run "your-init-cli start foo", you're starting a new service process. run0 is just one step away from that, because it connects the terminal to that process.
It's less "a new thing has been created with systemd" and more "a user interface was exposed for an existing functionality".
>This is a completely insane way to develop software, instead of creating a new piece of software in a separate project
I appreciate and respect the KISS sentiment of doing one thing and doing it well, but oftentimes I also think open source programs' lack of integration with each other at a fundamental level is also one of its biggest downsides.
Being a Windows wizard unlike most folks around these parts, seeing and using the tight integrations between all the Windows subsystems is frankly marvelous and I wish Linux could have something like that.
Instead you get awkward state management issues. Will the child really inherit everything it needs from the parent or has systemd forgotten to transmit something? If I chroot and then run0 will that process also run in the chroot? What about systems with older versions of systemd?
I imagine these questions are what might make this project non-trivial, but surely achievable.
> If I chroot and then run0 will that process also run in the chroot?
That's an interesting question and it's not quite clear to me what should be the answer. For example, one legit answer might be that it would not work at all (in which case you would need to use some other tool that fits your needs), or another would be that the policy can decide what to do in that case.
But wouldn't you agree that it is better security hygiene to explicitly opt-in to inheriting properties when going to an elevated security level, instead of opting out from them?
I expect that when I run "sudo rm /foo" it should try to affect the same file as "rm /foo". Especially when I just checked, without sudo, that /foo was the correct file.
If there is a non-error effect in running sudo rm /foo, then indeed it should have the same effect as rm /foo as root in the chroot environment.
But using sudo, or even root, in a chroot environment isn't good security practice in the first place. For example, the environment's /etc/sudoers, /etc/shadow, /etc/passwd can all be different (but actual user ids are shared), while the environment could mknod the root filesystem device, mount parts of the host under /usr and /home and quite possibly elevate its access to the host filesystem, in which case sudo in a chroot environment would interact with the host in perhaps ways that are not directly constrainable in sudoers in the first place.
I imagine it's possible to carefully construct the chroot environment and craft /etc/sudoers in a way to avoid this, but I wouldn't even try.
In a better contained system implemented with namespaces (as in docker) sudo should work as expected. I imagine e.g. with docker you could have a service that responds to the system dbus messages, either as a separate daemon in the container or something even provided by docker itself. Or, maybe the requests from the container could be forwarded to the main systemd, with information about the context the request comes from, and let it choose how to move forward.
In the end I don't think I agree that a sudo replacement exchanging messages directly with the systemd is the best way to go. There could be another daemon, run by systemd (or not), that handles these messages. I suppose in the case of dbus there is no "dbus firewall" to limit the access to dbus as one can limit access to Unix domain sockets with plain old permissions, though, so perhaps the security benefits would be minimal in the dbus architecture.
Thats fine, and lord knows we probably need a replacement to sudo.
However, sudo needs to be user friendly and fail safe with decent information as to why its failed. Something that service files historically didn't do.
But, the way it's supported also needs to change, it almost certainly needs to be decoupled from systemd's release cycle.
I hope that we have all learnt from early systemd, and that we all won't take a "lets piss on each other's chips" approach. I'm too old you you lot to start flame warring over stuff you'll never actually fucking use.
doas exists but isn't universally available but already solves this problem. sudo has too many features and permits excessive configuration, but it also has the convenience of ubiquity. Inventing a third thing tied to systemd is absurd and unnecessary.
systemd has been a net positive for the linux ecosystem. remember when you had to write bash scripts to start, stop, restart services and handle any other signals you want to send it? nowadays it's a unit file (basically just an ini file) away with relatively straightforward API. and you can actually declare startup dependencies and other useful relationships past just "prepend a number signifying when it should run globally to the front of the filename". it's provided an extensible platform with which higher level orchestration frameworks like ansible / ignition can easily templatize services or other system configuration.
since the beginning of systemd people have moaned about how complex it is and how we're reinventing the wheel. yet time and time again the people actually working on the project show that the solution they've come up with is the result of the problem they're facing on a daily basis. it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back", maybe think about whether or not you have a legitimate issue not being addressed by the solution proposed or if you are just getting rage baited by a headline.
It is much more than just a re-implementation of launchd. I can't grab a link right now, but Lenart wrote a long blog post about the philosophical decisions behind systemd, and he goes into detail about what he liked from launchd and what he did not like. It is called "rethinking PID 1"
Yeah, basically I've found that the people the more vocal against systemd are either not really knowing how it works behind the scenes, and just criticizing for the sake of it (or because other people do so), or criticizing from an ideological point of view (do one thing and do it well). They see systemd as an octopus, not following the unix ideology. Which I don't really agree tbh
The root of the problem is sudo, and su more generally. This derives from *nix, or mainstream versions today, requiring a super user, a manifestation of the problem with monolithic kernals. A microkernal may be a stepping stone to improving this but even this is not a solution. The only solution is no kernal space, no privileged user(s), all processes negotiate independently with each other.
No the root of the problem is the large attack surface systemd is creating by tightly coupling a ton of tools together, I agree about the microkernel idea completely though.
systemd isn't a single piece of software. It's a collection of software with an unified theme, like KDE or Gnome.
Attack surface-wise, I don't think there's much difference between "sudo" being a part of the systemd package and not. Either way there's "sudo" code to be targeted, which package it's part of is just a technicality.
I like to compare Systemd to GNU. Pretty similar scale, similar "take over the world" levels of adoption in Linux (GNU's libc is in nearly everything, all the other GNU tools are ridiculously common), adds lots of attack surface to what used to be single-purpose simple tools, etc. Just about every criticism of Systemd applies just as well to GNU.
So aim your ire at the distributions who (I agree) cocked this one up. "Take a library dependency to implement basic functionality" is not a systemd mentality, it's pernicious throughout software development - see leftPad as another example.
It's one of many things to consider. Think of it as sandboxing, or attack surface reduction. Should we expose everything to everything else, or should it be on a need-to-know basis?
IIRC, xz was used by a systemd library, and that systemd library got added to sshd so it could tell systemd when it had started or something like that. SSH itself doesn't use xz.
My Ubuntu /usr/sbin/sshd already links to libz, liblzma, liblz4 and libzstd. I don't see why linking to libxz would be so outrageous. All-in-all, ldd reports 26 libraries.
They attacked the weakest link, and systemd was just a small pawn in that game. Sure, a smaller attack surface is better, but it's not like OpenSSHd has a small attack surface even without libsystemd. Not even in projects with a similar possibility of obscure "test data."
> Except for libz, they are only linked indirectly though libsystemd.
Ah, that invalidates my point re. obscure test data. Sloppy use of ldd. (I'm guessing it would be much harder making such an attack on a crypto library.)
They didn't need to use the library to make use of the systemd notify mechanism, which is simple to interface and quite a nice feature in the first place.
The issue is that it's not just ideological. "Do one thing and do it well" is important because if you want to port software to another platform, it's a lot easier to port a single dependency component over to make it work than it is to port over the entire framework.
This is a serious problem and it makes it way harder to make things cross platform.
Systemd was written specifically for Linux, hard depends on a list of features provided by the Linux kernel and leverages them to do its work. Porting it to another kernel is a rewrite. Lack of portability is in this case a design tradeoff.
Sure that is the case for systemd itself but it's not the case for most projects that happen to use things systemd provides.
There is very little benefit for most userspace software to tie itself to systemd and by extension linux when otherwise it could be portable to any unix or unix like platform. Especially when an alternative, portable solution already exists and is well established.
The argument was that this design of systemd makes it hard to write applications that are portable, so that systemd is effectively a very big net negative to the open source ecosystem, because it causes massive fragmentation.
I think this is a really old statement. I’ve used systemd since it was made default in Fedora back in 2011 (Lovelock, anyone?) - at what point am I qualified to have an opinion, after having used it personally for 13 years and professionally for 8.
systemd is scary for 3 reasons.
1. it is inscrutable. Debugging it is nearly impossible, so you had better hope you don’t get a buggy release, especially with how hostile the devs can be.
2. it is large, and growing. Lots of things it claims are optional are in reality: not really. This is fine until uou get something that really doesnt work well (systemd-resolved is consistently the largest reason I have connectivity issues, wether it be because it interferes with docker inter-container networking or because it needs to time-out when trying DNSSec to continue- or if it fights with my vpn provider for power over my resolvers etc). Due to my distro being very tied to it: I gotta keep using it and working around it.
3. The interface is irreplaceable. Why are their 13 init systems? Because init is a closed scope. To be a sucessful init you need to spawn processes and do it cheaply. Supervising processes after start? Noble, and there were implementations that could do that (CDDL licensed SMF from Solaris for example). However we have already reached the state where it will be literal man-decades of work to replace systemd as we will need to make any replacement bug compatible with systemd itself. Its the ultimate show stopper.
The implementation is the reference. Which is a large departure now from what came before.
> you can actually declare startup dependencies and other useful relationships
In theory, yes. In practice, I had a lot of trouble ordering things correctly in non-trivial cases.
> it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back"
I can only speak for myself, but I don't want the abysmal sysvinit scripts back. I just want a simple process supervision suite which is true to the UNIX way of doing things. The sysvinit/systemd dichotomy is false.
Runit and s6 are both very real alternatives to systemd. They lack a ton of features, but they are a way to reliably run services. They do use shell scripts, but not for the reasons sysvinit did. They are extremely simple and have a very small attack surface as a result. Runit itself is a spiritual descendant of daemontools which predate systemd by great many years.
The problem with systemd is that it's a mediocre solution to many problems. UNIX deserved better.
In a similar vein, systems like openrc use "shell scripts" as well, but can generally be written declaratively[1]. This provides greater flexibility when it comes to creating one-shot services or services that require a little setup before running, as you can just re-define the start() function, rather than requiring one to make a separate shell script for it, or dependency tree, like you'd have to in systemd land.
"sysvinit" is an ill-defined concept, anyway, as every distribution had their own scripts and tooling around actual services, and sysvinit was generally only responsible for starting getty and launching the distribution's actual service system. How initscripts were created and how you managed them depended significantly on the distribution.
The other pieces are also pretty excellent. Ifup gave me no joy & was very limited. Systemd-networkd is a wonderful option with vast & great capabilities, that meshes well with the init process & it's style. Systemd-timesyncd is fine, I dunno, works for me. Systemd-journald is probably the weakest of the batch but mostly because it's not very ambitious; I loath that there's no way to really deal with super-active programs hogging all the logspace, short of configuring a second journald instance for that program, which ain't no joy to setup. Systemd-home is a cool set of features for portability. Systemd-nspawn is a bit ahead of its time & looks a little weird now but was a very novel & powerful thing to have built in to most systems. Systemd-resolved is a pretty good local DNS that handles mdns and much weirder cases easily. Systemd-boot is a really nice easy to work with uefi boot loaders that's worlds easier to deal with than uboot or gasp grub.
All of these are taken em or leave em. But they're good expectations to have. I love the bazaar model, but Linux used to have so few common expectations, used to manage various bits so poorly. And now there's a much better base of capabilities, which have universal patterns of management that tend to apply to them (how etc files are laid how, being dbus and maybe varlink accessible). Linux hadn't been growing; when you came to a random systems you expected enormously little and typically got it, and what extras were available were scattered/random and often not particularly high quality or capable software. Systems has extended a much bigger base of competency & capability. That we can plug in a USB drive and systemd-home can create an isolated dynamic user out of that & let that user securely run their environment there is a neat as heck expectation. That our network manager supports setting up such a wide range of bridges and tunnels and taps and tuns is fantastic, is excellent. I don't have words for people turning their noses up at this better world; this is so much more competent & capable a world than where we were, gives us so much we all can now take for granted, and it's been done smartly, more mono-repo than monolithic, such that you can do alternatives. But many of these pieces are utterly without peer. And the consistency of operation you get, the predictability of use, from being under the same umbrella, is an ergonomic wonder that is unmatched.
Is there a good overview of these pieces and what they're responsible for and how they fit together?
I had a quick skim of the docs on systemd.io and there are quite a few documents but they don't seem particularly organized and I couldn't see a good architecture overview.
It's not difficult to use some parts of systemd without using others. Is it any different than something like coreutils, another package of linux utilities that are synergistic but usable seperately? Nobody complains about coreutils being bundled. Even the rewrite in rust crowd bundle them.
It really is not. This "sudo replacement" (which under the hood is systemd-run) will apparently require quite a bit of the systemd stack to operate. That is not something you have access to in many lightweight container distros or on other non-systemd distros.
And coreutils is expected because most people use GNU/Linux. Coreutils is that GNU userland part. And either way, the majority of coreutils are GNU implementations of standard unix components. You can use most software that depends on coreutils on other *nix platforms like BSD for this reason.
Oh you do: this work completely differently from sudo, there is no suid binary involved, instead it does IPC to the systemd pid 1 and asks it to spawn you process, attached to your current terminal. So if you don't have systeme as pid 1, it'll have noone to talk to.
Whether you like that or not is for you to decide.
There are. Gentoo has actually always used the FreeDesktop reference implementation instead of sd-bus (the systemd implementation).
FreeDesktop also maintains a list of implementations (also including bindings so you have to read to find which ones are full impls vs libdbus bindings).
Its refreshing to see that just hate systemd because "its not UNIX way" is not anymore. I see better discussions under this post and in other places around the web.
You forget, the reason systemd was originally rationalized for its insertion into our trees was "boot times are too slow". Its chameleon-like nature and ability to solve the hastily described problem du jour seems to be its only consistently touted feature.
Bash scripts that start processes are ephemeral. If it's signal handling you want, that was your program's problem. Either that or your program didn't fork itself, which is a fish of an entirely different feather.
And now we have this sprawling mess of complexity and headache.
The resulting wayland environments are more complex because wayland itself refuses to define/include features that desktop systems are expected to have. This results in a sprawling mess of competing and incompatible interfaces for those gaps that other parts of the implementations (desktop environments) now have to compensate for by including multiple implementations of the same thing based on all these different interfaces.
Wayland by itself is simplier -- it is done by "outsourcing" everything a window system should do to the window manager.
There is where the complexity kicks in.
Wayland was a mis-fire in terms of user friendliness for the first decade or so. But it is still a big step up from the mess that was a typical xserver back in 2010.
> It reduces features adds complexity...
The irony here is Wayland is part of a huge effort to decomplex an xserver into component parts. A really commendable initiative; the path forward while maintaining the X protocol probably was impractically hairy.
The Wayland protocol design had some glaring flaws, but saying that it adds complexity is unfair. It oversimplified; it would have benefited from some flexibility in providing a standard mechanism to let people inspect the buffers graphics buffers to be composited.
> remember when you had to write bash scripts to start, stop, restart services
This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. I knew where the files were, what they were called. I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
The init system was probably peak of systemd, after that they started reinveting things in a more complicated way. Do we really need journald, systemd-boot, machinectl, systemd-networkd, sd-bus, systemd-resolved, systemd-nspawn? Do people actually use it all? Are there any metrics to show how many systems have it installed and in use?
And what would be the equivalent to, "Oh, I don´t know the name of the log for this process I can see in 'ps aux', let me cd into /var/log and see what filenames I can find ... or grep everything until I can find a couple of words that make some sense so I can keep digging further"?
The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.
Actually no, they are prefixed by the binary name and the PID number. Technically they are prefixed with: `<date> <host> <binary>[<pid>]`.
You can then use `systemctl status <pid>` to identify the unit if you need to.
I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary.
EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well.
EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\",
Well `-u` refers to the unit file, so I would start with `sudo systemctl status` which lists the status of active unit files. I bet I could find the unit name I'm looking for there. If not, then `sudo systemctl list-units` should have it. (and you can grep the output of both)
Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.
I think you can also configure the journal per unit right? Don't remember how exactly, at the very least you can stream logs to a different system.
There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.
These absolutely SUCK as answers to that question. They entirely miss the point. They provide a specific answer to a general problem.
The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work.
I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!"
Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it.
systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.
They may be correct answers, in that they accomplish a task.
They suck in that the proposed method for accomplishing the task is a suck-ass downgrade from the previous ways the new way proposes to displace.
I beg to differ about emotive, because absolutely the other side of this fence behaves every bit as butt-hurt when challenged as I just did. You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.
> You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.
Very well: I didn't say the commands were good or great. If you attempt a little objectivity you'll see it.
If you attempt a little objectivity, you will see that addressing the answers (vs the commands) was in my first response.
Before even going into the nature of the commands, I said that the answer does not match the question. The answers addressed details, while the details in the question were merely examples.
Missing the point is actually merely one of the at least two dimensions along which the answers suck. Thank you for reminding me about that.
> The answers addressed details, while the details in the question were merely examples.
No, the answers addressed the two scenarios in the question. That's why they don't "SUCK". They are good answers to the question as asked, and not the question as you dreamed it. If you spent ten words on asking the question of your dreams, rather than having a go at people for not answering it (whatever it is), you might have an answer.
To be clear -- you can just dump out all the logs to text and then use your standard grep toolkit on them. You can also dump them out to JSON, and use jq on them -- something you can't do with text logs on disk.
And as far as learning the options goes, learning the basics of systemd and journalctl is much easier than learning how to, say, use a source control system effectively.
(And it's definitely worth trying out new source control systems -- for example, I think most people should at least check out Jujutsu, and many people are going to like it more than Git. As simple as Jujutsu is to use, learning it takes a lot longer than learning journalctl.)
Agreed. I’ve tried to get used to it on Debian 12 for personal VMs. I just can’t. I don’t care. grep and awk are always going to be available, and I know how to use them.
It's way better than the previous status quo: now, more or less whatever distro I'm on, there's one command which answers that question directly, not a process of guessing how this system happens to be set up. You can guess and grep through most any kind of problem, but I don't see why anyone would want to do it as anything but a last resort, and to me that fits '37 easy steps' far more than the 1 consistent step in these replies (notice how none of them say 'oh, but on debian...'?).
> journalctl... it takes me 18 seconds just to type it.
Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.
You do have to learn a few new things, yes. But it's not too difficult. We all have to learn new things sometimes.
I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.
systemctl status <pid> will show you the unit and helpfully the last few lines of log from it. Journalctl -u <unit name> will then show you the full logs
> Oh, I don´t know the name of the log for this process I can see in 'ps aux',
With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal.
> let me cd into /var/log and see what filenames I can find
You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue.
> or grep everything until I can find a couple of words that make some sense so I can keep digging further
journalctl --grep, or more conveniently, journalctl and then / to search.
> The lack of explorability in journalctl
It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.
Same. Init system is awesome, love .ini files, networkd is as simple as it can be, journald is fine. Love systemd-nspawn for simple containers as it feels more lightweight than docker. Especially for quick one-offs where I don't need isolated networking you basically have chroot on steroids with "systemd-nspawn -D /path/to/fstree".
> I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`.
journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there.
> I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe, but finding the logs in the first place was always a challenge. Systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.
> journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there
Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.
I think my biggest gripe with journalctl is (and this may just be because I'm novice at using it) that discovering what sort of logs are on the system feels a lot more complex.
With regular logs, I can go to /var/log, ls the dir, and get a nice list of what's being logged. There will be, for example, an `apache.log` file that has all the logs relevant to apache.
It's just journalctl -u apache -f, assuming your distribution calls the unit file "apache", and the last few messages of the logfile appear with systemctl status apache
And you don't need to worry about "is this service rotating logs appropriately?"
The important thing for me at least isn't he `-u apache` but rather knowing that `apache` exists as one of the installed services on this box.
In the old ways, that was apparent because of the presence of the log file.
Granted, this isn't the problem it used to be for us, however, it would come up because we'd have our services named things like `foo-ws` and knowing that `foo-ws` existed on a given box might be tricky were it not for the log files.
I mean "systemctl status" shows all services and their hierarchy, there's also "systemctl list-unit-files" if you want to see things that aren't part of the current target (runlevel).
It's much easier in the systemd world to see the state of the system, the state of the service, and the logs of a service because it enforces this consistency.
correct me if i’m wrong, but i believe the point that is trying to be made is;
a system user/admin has an intuition about files. saying that ‘journalctl -f -u’ (fu, indeed :) and whatever else is inherently undiscoverable, and is a.. basically orthogonal mechanism for handling what should be a simple task. i.e., viewing some logs. it’s far easier to compose and extend from files (what if i only care about the mtime of the log, for instance), than this.
look, i think systemd isn’t.. terrible. i also think it’s suffered a bit of complexity fetishisation, and it seems as though that this resulting complexity may have become invisible to you.
run0 doesn’t seem like a bad idea. but i am wincing a bit at the thought of unrestricted javascript determining access control.
I think you have me confused with someone who cares about the difference between binary and text logs. I have no pony in this race; my comment was just made to help.
With `ls -l` I can see file modification dates. How do I do it with journalctl? And why do I need to that that in the first place?
Journald would be perfect if I didn't have to know a thing about it. Work in background, move stuff from stdin/stdout to /var/log/journald/service.log, move old stuff in /var/log/journald/archive/service/2024-04/15.log.gz. I'd be happy. Why do I need this cryptic CLI I don't understand. It brings nothing but pain.
honestly that's such a niche problem to have that i dont think it's worth throwing the baby out with the bathwater. anyways you shouldn't be discovering installed services by logfile presence, it should be something like querying your package manager.
I wouldn't say it's a niche problem. I also miss being able to go to /var/log and see instantly what's going on.
There should be a command like "journalctl --top" that would list all logging services with amount of logs, oldest and latest log times, and last few lines of each, to improve that experience. Maybe there is already a way to do something similar?
I appreciate that someone's invented shoes you inflate and snap. I can see some advantages.
That said, when I'm in a hurry and going to do a thing that I've been doing every day of my life, I'm really not a big fan of having to stop what I'm doing and reexamine a tool I've used all my life to figure out how to use it. I learned this stuff decades ago and I'm not anxious to relearn how to do it for a fractional improvement, and I'm not even really 100% sure that this new "inflate and snap" shoe model is actually an improvement, except for very specific use cases.
Don’t think of it as relearning a tool, think of it as learning one tool which works for everything. You no longer have to implement your own log rotation, compression, etc. or work out which configuration each program needs to do that in the manner you expect – for example, “is it safe to rotate a log file?” requires you to know implementation details on each program.
One thing to remember is that a lot of people use the testing distribution so they would have seen it before the Debian 8 release changed the default. The debate in the community took ages but one factor in concluding it was that people were using it for years without it being anything like the more hyperbolic predictions some opponents made.
I’ve been doing this for decades, too, but I don’t miss having to deal with everyone inventing multiple log hierarchies, different rotation conventions, compression strategies, etc. not to mention all of the other things which systemd removed from daily toil around process management.
'journalctl -u service -f' takes many seconds to start showing logs (even with 4GB journal size limit which not that much). 'tail -f /var/log/service' works without a noticeable delay (be it 1kb or 100Gb log file). For me it's a huge regression.
Not my memory at all. In almost all cases the logs were in /var/log and written by syslog. Unless the whole service was in a chroot directory, or the program was written by an amateur who didn't follow logging conventions.
This and many other discussions misses the point that most people that don't like systemd have with it.
It can largely do everything it replaces, but differently, in some cases with improvements. Every year it increases its scope of 'problems' its solved.
But the issue is it hasn't _ACTUALLY_ solved a problem for me in about a decade, its only introduced problems. Its replaced things that worked perfectly for me and millions of other users for years/decades with something that... also works... but differently. Now to accomplish the _SAME_ outcome without any personal benefit I have to re-learn their new and usually opinionated way.
There was a time I could log into a system, find logs, parse dates and do everything in these discussions reletave to logs, rotation. Regardless of service because the standards were close enough and the basic tooling (tail/find/tar/grep/awk etc.) worked everywhere always, and you never had to google even if the service was one you never worked on before.
For systemd its the opposite, its almost a guarantee you have to google something along the line when interacting with new systems and services. And as soon as their newly enforced opinions and way of doing things is getting comfortable they go and replace something else that was also working... perfectly fine.
If it actively solved problems for the majority of users we wouldn't hate it so much. Instead it requires more mental overhead than anything other than XORG to deal with and constantly manage than anything else in the linux ecosystem.
Ahh the vocal minority in support makes its appearance!
I wouldn't call the opposition tiny, although the vocal portion of any group tends to be the smallest. This would be my first online comment on it so I'm certainly part of the non-vocal group you ignore, that in my experience is significantly larger than the vocal side of group you dismiss.
systemd-nspawn is actually pretty great, and is mostly just a tool that exposes functionality that systemd already needs for other reasons. IIRC it was originally developed to aid in testing systemd itself, and was initially shipped as an unsupported extra in case other people found it useful.
systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar.
I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.
> I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet
I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.
I've recently become a fan of the `-x` flag to journalctl, which adds additional context to log entries and even suggests remediations at times. I've identified and fixed several issues on my Linux systems that way. Kind of hard to get that with tail -f!
> This was a really big pain, yes, but I also remember how I could `tail -f *.logs`.
Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends.
> --no-pager
To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.
I remember, and sometimes still have to deal with, programs that _didn't_ dump their logs in /var/log, forcing me to figure out where the log are in the first place
Those old init scripts were pretty boilerplate and not that hard to write. And in the case of OpenBSD (and maybe other BSDs?) the scripts for /etc/rc.d are usually just a few lines.
With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines.
I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.
Not the parent commenter. Windows has a lot of rubbish user-level interaction which belie how good an OS NT really is. Anyone who blindly disses it because 'Windoze' and 'M$ bad' is not really worthy of any attention whatsoever.
NT isn't terrible with some very well designed systems, that is quite true. It is however good for a completely different list of reasons that *nix is. Trying to conflate the two is a pottering's errand.
I'm still trying to figure out whether I like nftables more than iptables. Both are pretty complex to manage IIRC (I've only touched nftables a couple of times though, and not recently)
I don't resent systemd for existing, but I don't want to be forced to accept Lennart's shitty design decisions and it's clear he wants me to be since he successfully agitated for other critical components to hard depend on it. I want the option of opting out, which Lennart would deny me if he could.
At the end of the day, I just run Void (or MX Linux if I want something more Debian-flavored) and get on with my life.
> systemd has been a net positive for the linux ecosystem.
You're presuming to speak for an awful lot of people there, on a topic that would be difficult to measure.
> since the beginning of systemd people have moaned
> it's quite annoying that the armchair linux experts complain
Now you're overgeneralizing, and doing so in a dismissive and patronizing way.
Here are a few examples of problems I have with systemd:
System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.
Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.
At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.
There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:
Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.
I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.
Systems isn't blocking the shutdown. Systems is being blocked by a service. Since you are saying that you can't track down the exact service, you must be using a container runtime like docker or LXC which is not transmitting the kill signal properly to the container to shut it down. The container itself then keeps running for whatever reason. This entirely depends on your particular setup and has very little to do with systemd.
When it comes to the shutdown issue, I'd argue this is a side-effect of systemd bringing some consistency and correctness to what used to be the wild west, and is highlighting some issue that was before overlooked.
Granted, in a lot of cases the issue probably wasn't a big deal (the system is being shut down, the user is already logged out, do you really care that a system background process is being shut down cleanly?) but from systemd's perspective there's no difference between that and an actually business-critical process that should absolutely be allowed to terminate cleanly before unmounting the filesystem and powering off the machine.
> systemd has been a net positive for the linux ecosystem. remember when you had to write bash scripts to start, stop, restart services and handle any other signals you want to send it?
Systemd was meant to replace Ubuntu's upstart, not sysv init. And it indeed systemd was better designed than upstart.
Sysv init is a horrendous hack. People should NOT use that. Upstart had limitations but is okay, and certainly much better than sysv init. Systemd is okay too (but has much more features).
Also as an historical note, both upstart and system was inspired by Apple's launchd.
NixOS was an early adopter of upstart but it later migrated to systemd (alongside Ubuntu itself)
"There’s a new tool in systemd, called “run0”. Or actually, it’s not a new tool, it’s actually the long-existing tool “systemd-run”, but when invoked under the “run0” name (via a symlink)".
systemd-run is very useful to run tasks with specific cgroups settings, or at a specific time. It asks for password whenever needed.
Lennart's toots suggest they are replacing a complex SUID binary with an already existing component (systemd-run) with better workflow (service manager handling the elevated context), which sounds like a sane move to me.
898 comments
[ 4.0 ms ] story [ 386 ms ] threadrun0, which behaves like sudo, but works entirely differently and is not SUID. Run0 asks the services manager to create a shell or command under the target user’s ID, creating a new PTY, sending data back and forth from the originating TTY and the new PTY.
Of course. The solution to every Linux "problem" is, of fucking course, to have the PID1 spread is tentacles to yet more part of Linux.
Every single problem can be solved by giving yet more power to PID1... Except the problem of PID1 having too much power.
I don't want to learn a new network config alternative with every update (Ubuntu changed its net config tool again with 24.04). I don't want an immutable os. I don't want to learn to write new config files. I just want to do what I've been doing but with new packages. If there's a problem with something, just fix it. Don't throw out the whole thing.
I moved to FreeBSD and am happy for its reluctance to change. If there is any, it's usually offering something genuinely new to me as a feature and to boot I only need to learn about it if I need it.
Hardware support is much lower but it's worth it IMO. I had the same irritation with macOS. Every release breaking something essential that was part of my workflow and i didn't want to change. Eventually I did change but away from Apple.
I don't want to change to LennartOS either.
For example, I run a Visionfive 2 OpenBSD install with squid, everything else has to go through that.
Relayd also does a bunch of similar things and is closely integrated with pf too..
That's not simple though- humans tend to think THEY know better for all others while it's often not.
Say you don't want to have immutable distros, but many want. Android or Talos or even OS for network switches are nice examples.
Linux based solutions and products related community is wide and to keep up with evolution one needs to adapt and adopt changes.
We already see what happens for immutable/stale systems like FreeBSD - even TrueNAS is abandoning it, dying as predicted by experts since 2008-2010.
On other side - nothing stops you from keep using sudo or even choose disto vendor which doesn't use systemd, there is a choice, not thing set in stone.
Say I don't plan to switch to run0 in any near future - first it's not gonna be in any LTS disto, 2nd - it seems to be lacking LDAP based rules and very much likely other important features.
E.g., see all of web development
Also "time away from a PC" is ironic given "the current change of society". :p
Well if society goes towards fascism like the Netherlands, then yes I'm very proud of not moving along with that.
The best remaining option is to live side by side with those people and interact with them as least as possible, and hinder them as much as possible in their attempts at legislating, like using the first chamber to block things. Mind you, this is already the status quo in the US where the democrats and republicans use whatever means at their disposal to undermine the other.
We didn't really have this phenomenon in the Netherlands because we don't have this "the winner takes all" kind of political system but parties need to collaborate. But the rise of the extreme-right makes it impossible to cooperate any longer.
Same here on server. Desktop is still linux.
I love it when new and better ways are found to do things.
I love it that Linux is constantly improving and moving forward.
I’m willing to accept along the way some things seem to be mis steps (I’m looking at you snap packaging).
I love it that improved network configuration systems are being adopted because network configuration is a pain.
I love systemd and when new stuf comes out from the systemd project I think “gee I’m glad finally someone is taking a wholistic look at and fixing that messy inconsistent evolved corner of Linux and replacing it with well thought out powerful and integrated solution.”
Bring on the change, change is the best thing about computing and software. I own vintage computers but wouldn’t want to live there.
It’s bullshit and I wish it would end.
Alas, keeps the consultant bucks flooding in when we have to rewrite a load of cfgmgt to go to 24.04 I guess..
It's also easy to explicitly express "weird" stuff like "run DHCP client and use all the config options except the default route". Seriously a couple times I needed to do dumb shit and was like "there's no way they let you do this" but no, there's a way to do it.
And I'm pretty positive the config files will stay stable over the coming years and any new networking features will get appropriate config options in newer versions.
I really don't care about interface configurations that much. It's an annoyance, but one that's quite easy to fix. I think introducing yet another network configuration here isn't the answer, but my linux fleet just run k8s anyway and network config on the linux level is quite simple before we get into cillium/istio/etc :}
So maybe don't just switch out of the blue right now if you've got a working setup, but maybe keep it in mind for the next time. :)
I do, too. But I also really hate it when those new and better ways make things worse for me. Systemd does that in a couple of important ways. There is even some network-related startup stuff that I can no longer make work automatically at all. For me personally, systemd is a regression, not an advance.
But I also recognize that the Linux world is not duty-bound to make sure it remains excellent for me, and I've pretty much given up on advocating for my needs in the Linux space. There's no point, particularly with the systemd crowd.
But I've also seen the community defend terrible stuff just because.
Look at what happened with the init system. System V, fstab, etc was awful. Doing anything with a reasonable level of robustness was grotesquely obtuse and complicated. And yet it was "perfectly fine" to the greybeards. Alternative proposals were near zero.
I don't have a dog in the networkmanager/netplan fight. It could be that one is irrelevant; given history, I have a hard time trusting what I hear.
sudo has quirks for sure (which is why you see a number of alternatives).
Any serious challenge to systemd nowadays is probably going to have to at least offer compatibility with it. No one is going to rip it all out and start over again (again).
Jokes aside, I just think that life is constant change and the programming industry is a good example of that. Coding practices have improved a lot in the last few years and will continue to improve with new knowledge and new technology. Sometimes it's better to start anew from scratch than trying to adapt old code into new practices.
Btw this is not a young whippersnapper saying this. My first IT job was on FreeBSD and OpenBSD. I was a full-time FreeBSD user from 04 to ~10.
And I remember exacty this gripe back in 2011 when Debian was using one network config, RHEL another. Today I actually enjoy the progress made with systemd, and I'm that annoying co-worker who will give you crap for disabling SElinux.
I'm a fan of SELinux (and the similar mandatory access control on BSD) because it gives strong security but the user or admin keeps control. This is a much better solution than things like immutable OSes where the user can't control anything and just has to trust the developer of the OS.
Look at all the different netsevs supported by systemd-networkd. https://www.freedesktop.org/software/systemd/man/latest/syst... This is a huge list of tools that required a huge assortment of tools to do before, few of which had even part way decent management & fewer still had good init scripts. You used to have to learn from 0 each time, with each tool. Nothing was alike, nothing was as capable, nothing was integrative.
Don't listen to these complainers, for God sake. Your life is too short to get pissed off about well built work together featureful tools being built in a mono-repo fashion.
I'm so tired of the sabotage, so tired of broadscale general refusenik attitudes. This post is absurd. There was nothing to learn before, everything was 100% special snowflake & distinct. None of it was great, all of it was limited. Systemd mono-repo is built of many small pieces, but they couple together and are 100x more learnable. What you learn today will work across whatever system you run across. It's such a a better world, and these "pry it from my cold dead fingers" attitude can keep to that path for all I care, but I wish they weren't poisoning kinds with absurd incorrect negativity & being such magnets for disdain. The world today is fantastic & you rarely see these folks with even an iota of appreciation for how good we have it, never a drop of balance. But hate sells, & unites, powerfully.
But I do. So I use one, and contribute to a project that tries to create one. Am I a part of some sub-community that wants constant change? Or do I just have an unusual use-case and want to support it?
The beauty of OS is that anyone can decide which tool to use, contribute to it, and even fork it.
That's really just Ubuntu's fault. Between upstart, Unity, netplan, and snapd, Ubuntu likes to go off and do its own thing for a few years before coming back to what everyone else picked in the first place
[0]: https://www.sciencedirect.com/science/article/pii/S016740482...
Poettering's hypocrisy is painful.
Because that's what he's complaining about
https://marc.info/?l=openbsd-misc&m=171227941117852&w=2
"Liblzma ends up dynamically linked to sshd because of a systemd-related extension added by many Linux packagers that pulls in liblzma as an unrelated dependency."
https://news.ycombinator.com/item?id=39866076
"openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma."
- Linux packagers decide to patch sshd to use libsystemd for a notification, that could have been trivially done without this library.
- libsystemd depends on libzlma
- libzlma depends on xz
And therefore, systemd is insecure?
And what does this have to do with the fact that SUID is a terrible idea that needs to go?
Why was that? Would that "trivial" approach have broken the next time systemd made one of their incompatible interface changes, perhaps? Was using libsystemd the kind of thing the systemd maintainers recommended?
> And therefore, systemd is insecure?
Systems with systemd had a vulnerability that systems without systemd did not. So it certainly seems like systemd-the-system (not necessarily systemd-the-unix-process) is bad for security.
The reason why "only" sshd on debian/ubuntu was affected is that the attacker chose to tailor their exploit to those systems. Systemd was the vehicle, debian patching opensshd was what made this specific incarnation of the attack possible, but essentially, both trusted a widely used library.
Sure. But security-critical software like SSH would certainly think twice before bringing in such a huge and complex dependency as an XML parser.
> I'm absolutely certain there's other juicy targets that depend on liblzma.
You could probably make a system package manager (which has obvious reasons to depend on a compression algorithm) do something nefarious. But that would be a more complex chain of exploitation with more chance for things to go wrong. Most security teams put more attention on security-critical parts like SSH, and I think most people would agree they're right to do so.
Second, when even the package maintainers can make such "trivial" mistakes, something is wrong. You'd expect a component such as systemd to be much more trustworthy than some random library.
I'm not arguing against systemd, just that it seems to grow and grow, and is not the correct place for security. It security is obviously broken.
It absolutely is. sudo allows you to execute code as another user. If you want to do that without giving sudo itself administrative privileges, this has to be done through the service manager, which creates a completely new, elevated process and handles communication with that. This is how it should be done (and BTW, this is pretty much how also the new sudo for Windows works). Now Lennart for some reason prefers systemd as this service manager - you might disagree with that choice, but then come up with a better one.
> then come up with a better one.
Really?
Well said. What makes you think systemd does not do this? Have you ever even looked at systemd in any amount of detail? Do you think it is one big binary running as PID1 doing everything?
Also, even before the backdoor was discovered, the systemd team were making libxz be dynamically loaded only in the cases where it was needed which would have killed the backdoor dead. There's some evidence that this might have actually caused the backdoor to be sped up and hence led to its discovery. Claims that systemd has bad security have to explain why it was already implementing practices that would have blocked the xz backdoor without it even being discovered. That seems pretty decent to me.
> Claims that systemd has bad security have to explain why it was already implementing practices ...
No, they don't. It doesn't take away the fact that they did not check xz, and probably only few of their other dependencies.
Xz is also used by Debian's package manager (both dpkg and apt). Or tar. Or the Linux kernel. Or... It already was a system library on Linux systems before systemd started using it as well.
Your post reads like you have no real idea what Xz is and how it is used.
Btw, do you think the Linux kernel developers are also clueless for using Xz?
> - libzlma depends on xz
> And therefore, systemd is insecure?
Yes. You have literally just described the way it is insecure. It bundles a large amount of functionality under a single system, and therefore anything using that functionality is at risk. You seem to be suggesting that Systemd would be secure if you didn't use it, which is obviously fallacious. Anything is secure if you don't use it. Systemd offers this functionality, and did it in an insecure way. You cannot blame users for that. Saying that people shouldn't be using a certain part of Systemd is really the same as saying that part shouldn't exist to begin with. The conclusion is obviously that Systemd should be smaller to decrease the chances of things like this happening.
It was purely the attackers choice to leverage the exploit via systemd instead of injecting code in the kernel at build time.
Linux kernel, gcc, glibc - all bundle "a large amount of functionality under a single system" - does this make their design fundamentally flawed as well?
On the whole, I do not like monolithic software projects, but I can accept that they are necessary or beneficial in some cases. Systemd is simply a much bigger target than these other things because it is an especially bad example. It has many components which are only tentatively connected. It is also more fixable. Alternate init systems are used much more widely than something like Hurd to replace Linux. The laptop I'm typing this from runs GuixSD which ships without Systemd and I can hardly notice the difference. I doubt a different kernel architecture would provide such a seamless experience.
The attacker chose to target a very specific component of a very specific system. It was their choice, not some sort of technical requirement that made it impossible to use a different attack vector. Just as they chose not to target other Linux distributions that use systemd.
You’re essentially saying “I was safe because the attacker chose to ignore me.” That worked well this time, but it’s a pretty dangerous stance.
Now I want you to imagine that every piece of software has a score, which tells you how useful it is to hackers. Systemd has a high score, and hence it was chosen for this attack. Your argument is that: there are other pieces of software with a high score so it's fine for Systemd to also have one, since without it there would be other things to attack. My argument is that we should reduce the amount of software that has a high score. Do you think my reasoning or your reasoning will lead to a more secure ecosystem?
There's lots of projects that link xz, big and small. Patching sshd to include any of them would have implemented the backdoor.
And I'm sure it'll be the same excuse next time.
This is not my point.
[1] There's lots of projects that link xz
And neither Poettering nor the systemd developers patched said, the Debian people did that. Seems weird to blame systemd for that?
The inclusion of a library to send notifications shouldn't have external dependencies, it shouldn't need them. The library is included in the customer's codebase at execution time, so it is a hole in the customer's security model. This immediately opens a supply chain attack vector (which is what we saw).
This is being taken as evidence that they shouldn't have responsibility for truly security sensitive code - the replacement of sudo.
Some of this is a long-term dislike for systemd and some representative bias. However, Systemd has missed the opportunity to make their client libraries safer.
Personally? I wouldn't have thought to limit the dependencies of my client libraries. It's a growth and project age thing. One moment you're on one side of a line, the next you have to skill up and do things differently.
Using systemd as intended shouldn't result in security holes for their customers.
And again, Poettering's complaint with sudo is specifically about it being a SUID binary, so this discussion isn't even related to the thing you're accusing them to be hypocritical about... SUID is more than just "code running as root", it's "code running as root in an attacker-controlled environment". That last part is the important one.
This attack demonstrates that they should be tightly focused and have minimal downstream dependencies.
I don't have any experience with systemd, but typically, people will bundle _all_ of their client libraries into one .so and say "use that".
However, what needs to happen is there should be multiple .so's, one for each sub-API. At least there should be libraries for frequently used shim code (like notifications seems to be).
Then systems that need to push information don't need to pull in the dependencies for other parts of the overall systemd interface.
I can't think of a reason why pushing local notifications would require a compression library. The notifications should be information heavy already, so not very compressible.
As I said, it's a growth and project age thing. One moment you're on one side of a line, the next you have to skill up and do things differently.
How the team responds is what is important. That is why people are objecting to the SUID work. Not because sudo isn't a hole, but that the systemd team isn't considered responsive enough to take it on.
I know I'm taking lessons from this to my work. It's an unpleasant mirror for me to look in.
The hypocrisy is in calling out a different project for being an overengineered tool running with too high privileges.
Unit files are a neat concept I don't want to miss again, but everything else done by Lennart seems to be an inceasingly stupid mistake born from hubris.
Those sort of problems sound like the sort that get a lot of attempts which run into the complexity wall and halt. I think Amazon has one of the best implementations of a privilege system I've used and it is horrible.
If all you are using sudo on is a personal (i.e. single user) laptop/desktop to install packages, this (along with other things like pkexec or doas) would seem to present no issues (and personally, from what I can see, I'd be happy to run `run0` on my personal systems!), but sudo does significantly more than that, as is called out by the systemd devs in the linked post https://mastodon.social/@pid_eins/112353324518585654
sudo supports not just LDAP (for multi-user systems), but include various levels of logging (including logging stdin and stdout of commands), apparmor and selinux profiles, the BSD and linux audit subsystem and more in a simple, easy to read and edit config format (this is just me reading from the `sudoers(5)` man page).
Whereas it seems `run0` won't have a `sudoers` file, but will instead be configurable (implicitly) via polkit, which uses JS to write policies (which I'd view as a much harder and error-prone system than the current `sudoers` format). It's not clear to me how much of sudo is tied to SUID vs. having a separate daemon (i.e. how much would have to be ditched vs. how much could be mapped over).
I do feel this is systemd moving away from traditional multi-user unix systems to being a single-user system (targeting the laptop/desktop case, or where sys-admins are the only users of the system, and it's basically a container host).
Of course, once distros start to say 'wait, why are we shipping 3 different privilege escalation systems again? Systemd is needed for starting units anyway, so lets just drop sudo and su'
In many ways systemd has actually become the operating system. It’s so pervasive that it certainly is more deserving of naming rights than gnu. “Systemd/Linux” makes more sense than “gnu/Linux”
Seriously though, this seems to be a decent replacement for sudo if it works as seamlessly as it's described in the article. I still prefer the doas line of approach that simplifies the tool as much as possible, but I see the value of having such an important tool integrated into the existing system tooling, especially if it already includes everything but the kitchen sink.
Linux processes inherit a lot of state from the parent which means it's absolute hell to make a secure setuid binary. And at any time the Linux kernel can add a new feature which will be inherited by a child process, but that the process can't defend against because it wasn't even a thing when the code was written.
Running a binary at all also goes through a complex set of initialization steps a lot of programmers barely know exist, let alone are able to understand fully.
Both have problems, but if you're going to switch users to root you have to do one of them.
Edit: never mind, I see you are a systemd crusader.
Even if systemd did not exist, the hacker would have just picked something else to infiltrate.
This is a great argument against all computer security. If you believe in securing your computer, you're supporting hackers. Because if you ever believe that a lock has failed, you're saying the thief is innocent; that's how logic works.
And secondly, I would say even if nobody is going to blame me, I would still secure my systems. Why? To protect my data of course.
But re. expanding the attack surface - unlikely. Systemd's primary purpose is to start processes with the right environment / permissions. systemd-run/run0 basically give you the tool to invoke that functionality with a terminal attached to it. That's smaller scope of extra code than sudo/doas deal with.
But either way, you could always do notification in a few lines yourself (probably as many as you needed to link that library in the first place). I've done multiple 3-line "implementations" in Python and Ruby in the past and never linked it for example.
If I’m running on my workstation or desktop just let me run the damn thing. I don’t need an unprivileged user. On TempleOS you can modify the running system in ways you can’t on Linux.
My issue isn't even that someone tries to replace sudo. That may or may not be a completely fine thing to do, depending on the state of sudo and what improvements can be made. But what makes me really upset is this completely unexplainable need to make everything part of one particular init system. There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
This is a completely insane way to develop software, instead of creating a new piece of software in a separate project they will force all their projects simultaneously onto all their users for absolutely no reason.
I am very glad to have jumped ship from systemd. It is particularly bad software created by a team of people who engage in very bad practices and a totally unhealthy view of software in general.
That's just not true. Just because a system uses systemd the init system doesn't mean the it is forced to use the other components.
You can not try to create a large coherent system and then tell people they shouldn't use that particular part. That is totally disingenuous. Systemd is DESIGNED to be an all or nothing deal.
^[Citation needed]
We've all got confirmation bias, the trick is to be aware of it.
This is such an absurd hill to die on.
Gentoo folks have eudev or seatd but thats an uphill battle.
Why don’t you propose a better solution? How would your non-systemd solution actually work?
The systemd developers are tying it to systemd because they are systemd developers. If somebody else made something like this, it wouldn't be tied to systemd. But somebody else hasn't made something like this.
Sudo isn't going to just go away (unfortunately). You can keep using your CVE-ridden setuid binary as much as you want.
What? Literally every single other group of software developers has managed to create two projects. Even Microsoft can do it.
Of course you can. Unless they're depending on non-standard parts of libc, you can use any GNU project with other libc implementations (musl, dietlibc, ulibc etc).
https://www.chiark.greenend.org.uk/~ian/userv/
See 6.6 Error handling and input streams (eg stdin) on https://www.chiark.greenend.org.uk/~ian/userv/spec.html/ch-n...
systemd's run0 on the other hand doesn't require code to be written specifically to work with it.
Well, there's also a reason they are doing it, and many aspects of it (seatd, timesyncd, resolved, run), and more neat than cobbling together bash scripts. I like it
It is not unexplainable at all. In fact, the article explains it very well.
sudo allows you to execute code as another user. If you think about it, you could also replace sudo with ssh to localhost - just set a root password and allow root logins. Now, security-wise, this would obviously be a bad idea. Our current solution is to give the sudo binary itself administrative privileges, which is a slightly less bad idea, but still pretty bad.
Systemd already handles logins, so it is quite obvious that it could also handle this problem very well, and in fact, it already does: there is a tool systemd-run which you can already use. It will create a completely new process and will handle communication to it, just like ssh, but without the above downsides.
Exactly that was my objection.
Reasons:
- systemd is a large beast -- no need to make it larger with unrelated things
- the Unix philosophy seems to be applicable here
- the result should be portable to non-systemd systems
I think I sufficiently explained why it is very much related.
> the Unix philosophy seems to be applicable here
Systemd very much follows the unix philosophy. It is not one big binary, but actually consists of dozens and dozens of tools communicating with each other through protocols.
> the result should be portable to non-systemd systems
Portable on what basis? POSIX?
I think what people mean to say but don't have the words for is that systemd is an East coast school of thought project and folks prefer "worse is better" style tools.
This:
| Systemd already handles logins
?
But I don't see why a bring-up/shutdown system should handle logins.
> Systemd very much follows the unix philosophy. It is not one big binary, but actually consists of dozens and dozens of tools communicating with each other through protocols.
That is fine. As long as this service (sudo replacement) is a standalone, separate daemon started by a systemd unit, I'm happy. If it's a core part of systemd itself then I'm not happy.
> Portable on what basis? POSIX?
Or Linux / glibc / musl, sure, why not. Some people (no, not me) want to run Linux w/o systemd. The point is that a sudo replacement service should be fairly portable to the universe of UNIX/Unix/BSD/Linux.
This fetish of "everything should just stick with libc and POSIX" needs to go. These standards have not evolved at all, they are decades behind and don't even remotely cover the necessary requirements for implementing a "sudo replacement service". Just stick with sudo then.
> But with one key difference: it’s not in fact SUID. Instead it just asks the service manager to invoke a command or shell under the target user’s UID.
I understand your frustration, but systemd isn't the first attempt to build an integrated system. It just happens to run on Linux. It isn't insane to develop software this way, from that perspective.
https://skarnet.org/software/s6/s6-sudo.html
I was very surprised when I learned about it, but it does make sense (for s6 at least).
> There is absolutely no reason to tie your new sudo replacement to systemd. Absolutely none.
With s6, the idea is to replace a SUID binary with an IPC mechanism. That does make sense, since (parts of) the init system need to be running as root.
> I am very glad to have jumped ship from systemd.
All that aside, so am I.
Same idea with systemd-run.
Because I'm pretty sure that most of the components are optional.
You did not even discuss the reasoning given for not using sudo to instead hop on your soapbox to say it's bad software with bad practices and that they are stupid.
It's annoying how in the the more surface level Linux communities there's 0 value in discussing systemd.
"1 million lines of code for PID0!"
The new thing is blaming systemd for that recent exploit even though distros were patching in the bug themselves.
People analysing the exploit determined that a new version of systemd was going to prevent the exploit vector so the exploit seemed to have been rushed out.
Isn't this just textbook FUD?
What I've noticed is over the years is systemd would have identified a gap in functionality.
Like systemd-homed having a solution for automatically encrypting home directory when the machine is suspended.
Is that a functionality that OSX has had for years? Yes.
But anti-systemd people will dislike it automatically.
No, I am a dedicated systemd hater ever since I spend over a month full time writing and debugging systemd services for work. Systemd (the init system) is just all around badly designed and executed, I have very little confidence in the developers and their technical abilities and their tendencies to expand into completely unrelated areas for seemingly no reason makes me quite concerned.
I wouldn't blame the xz exploit on them, it is very hard to call it their fault in any way. But I do think it is a symptom of a system which has grown far too thin and wide.
You're essentially saying that the month you spent is enough for you to call it bad and the creators incompetent.
What qualifies you to make a determination like that?
There is never any actual technical reasons it's always about vague things like not adhering to UNIX philosophy, lines of code or it being badly designed (without any real architectural criticism)
This is an article about why they believe sudo isn't a good system. Where's your criticism of that from a technical / security perspective?
It's been about 10 years since systemd was adopted by Debian/Ubuntu/Redhat/Fedora etc.
Millions of deployments over the years. The companies that build and are paid to support for years with SLAs the operating systems are using it without issue.
I did not mention the first two, so please do not pretend I argued that. For bad design look at transactions. That is really dumb and makes the system near incomprehensible. The documentation is bad, dbus is literally so bad they tell you not to use it without a wrapper. The terminology is very questionable and makes it hard to explain what a unit actually does.
But I don't even see that as the worst part. The worst part is that they fundamentally can't do basic software engineering, in the sense that they do not have a defined project scope. Everything is potentially a systemd issue and not once does anyone take a step back and say "maybe systemd" isn't the right place to fix that problem.
>This is an article about why they believe sudo isn't a good system. Where's your criticism of that from a technical / security perspective?
If you don't read my posts please do not respond to me. Look at the first post I made and carefully read it.
I read your OP. It does not contain a technical / security criticism of run0. It's an angry, hand-wavey, vague rant against a project that took a design decision you apparently disagree with, but lacking any actual analytical evaluation of the thing up for discussion.
This sort of top-level post shows up on every single article that mentions "systemd", so you'll maybe understand why people tend to be dismissive.
Yes, I literally say there is nothing wrong with the idea, so you going ahead and demanding I criticize the idea, is just absurd.
Really, this is completely bizarre. I even say that the thinking behind replacing sudo is fine, yet you are here complaining that I don't deliver technical arguments against something which I even told you might be completely valid to do from a technical perspective. Baffling.
I'm not sure there's much point engaging further, I hope you have a good rest of the day.
Do you understand the difference?
I think that portability is a deliberate anti-goal of systemd.
> In contrast, if you want to use, say, run0, you must run systemd as PID 1,
No, you must run something on pid 1 that implements the spec, similar to how musl can be used instead of glibc - they both implement the same spec.
Run0 expects pid 1 to behave a certain way, much like my web browser expects web servers to behave a certain way.
Yes, and that is one of the things I dislike about it. (In fairness, the list of things I like about it and the list of things I dislike about it are both fairly long.)
> No, you must run something on pid 1 that implements the spec, similar to how musl can be used instead of glibc - they both implement the same spec.
> Run0 expects pid 1 to behave a certain way, much like my web browser expects web servers to behave a certain way.
If there's only one implementation, then it's not portable. If a webapp uses a web API that only Chrome implements, it's not portable regardless of whether Google published a spec for their non-standard behavior. There are dozens of web servers and web clients that all speak HTTP, there is one systemd.
You should look at it differently, and then it'll make perfect sense.
systemd has long stopped being just an init system. It's a system tooling suite. When thinking about systemd don't think "PID 1", think "Linux New System Software Suite". It's a big umbrella project in the style of Gnome and KDE.
For example, systemd-boot is a perfectly normal bootloader that's just been systemd-themed. It has a "ctl" tool, has the same command-line aesthetics as other systemd-group tooling, and so on. It's not in any way dependent or even interested in the init system.
I think that this has always been the core criticism of anyone who objects to them. Besides systemd, the init system, just being very poorly thought out.
I don't think having an unit that generates units at runtime is an officially supported use case, since generators exist.
>I don't think having an unit that generates units at runtime is an officially supported use case
Are you sure? Can you tell me how I would find out?
That's the point. Any situation in which you have a system modify itself at runtime is a recipe for a headache. So you do your auto-generation first, then work from a stable state.
> Are you sure? Can you tell me how I would find out?
Aside from that this kind of recursion seems like a great way to get weird problems, and that generators exist for this exact thing, the whole design of systemd discourages this kind of trickery. Units are just supposed to start a command and little else.
Maybe somebody made an official pronouncement on this somewhere, but my personal take on this kind of thing is that it's a bad idea, anywhere, not just systemd.
You can't do auto generation if that generation depends on the output of some units. Generators can not solve that problem.
>Units are just supposed to start a command and little else.
What a bizarre thing to say. No, that is not just what units are for. I think you are severely misinformed about what systemd is. Units are supposed to take care of ordering accept IPC, define how to handle failures and manage devices.
That comment alone makes me believe that you just do not know what you are talking about at all. It completely misses why systemd is designed the way it is and what it tries to accomplish.
True
> What a bizarre thing to say. No, that is not just what units are for. I think you are severely misinformed about what systemd is. Units are supposed to take care of ordering accept IPC, define how to handle failures and manage devices.
I mean that part of the point of systemd is that units mostly work out to ExecStart=/usr/bin/binary, and an unit isn't supposed to contain the arbitrary jank one can put into a SysV script.
IPC and the like is an explicit systemd feature, not something you improvise behind the scenes and then expect to work anyway.
Units are not just for services.
From systemd.unit(5) man page:
> A unit file is a plain text ini-style file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a watched file system path, a timer controlled and supervised by systemd(1), a resource management slice or a group of externally created processes.
> But what makes me really upset is this completely unexplainable need to make everything part of one particular init system
systemd is not init system, its and umbrella project for various core system components, which includes an init system.
> There is absolutely no reason to tie your new sudo replacement to systemd
well, in this case there is good reason to have it interact with service manager in general
> But with one key difference: it’s not in fact SUID. Instead it just asks the service manager to invoke a command or shell under the target user’s UID [...] Or in other words: the target command is invoked in an isolated exec context, freshly forked off PID 1, without inheriting any context from the client
strictly speaking it's not 100% coupled to systemd specifically, it most likely uses systemds d-bus API, which is part of their "portable and stable" APIs and as such could be implemented by other service managers https://systemd.io/PORTABILITY_AND_STABILITY/
> This is a completely insane way to develop software, instead of creating a new piece of software in a separate project they will force all their projects simultaneously onto all their users for absolutely no reason
The couplings between systemd projects are not that tight, you can pick and choose which parts you want, they explicitly are not "forcing all their projects" to consumers. The minimal systemd build has only init, journald, and udev, which is not exactly sprawling huge. All the rest of the projects are fully optional, and I believe quite many of them can actually work without systemd-init
Obviously one of these groups is lying.
This is literally just basic systems engineering. I don't even know what you are arguing here. The more tightly integrated a system is the harder it is to replace individual components.
This means components have to tightly conform to the components they are replacing, which obviously decrease modularity as these components are harder to maintain for available system configurations.
This isn't about whether someone can rewrite a part of systemd, but whether you can freely mix and match, which tight integration works against. Obviously
Well… yes it is. Public interfaces and contracts, anyone? Decoupling?
A component that depends on one simple interface is far easier to replace than one which depends on twenty complex ones. This seems like the most basic stuff.
The point being made is that it directly depends on the init system part.
> well, in this case there is good reason to have it interact with service manager in general
I disagree! There's no reason to not have it be its own daemon with its own configuration and looser bindings to the rest of the systemd ecosystem (e.g. through dbus and other protocols). KDE applications do this a lot, where they take advantage of other KDE components if present. This is strictly a philosophy thing and not a requirement for achieving what he proposes.
> their "portable and stable" APIs
"we're portable if you reimplement our APIs", idk how this is an argument. They consistently make very little effort to be compatible with (or provide fallbacks for) what's already there. Having stable APIs is nice but there's a reason most of the interfaces on that page don't have alternative implementations: They solve questionable problems and provide no tangible benefits over the methods there were before. Yet projects feel compelled to hard-depend on them...
> The couplings between systemd projects are not that tight, you can pick and choose which parts you want, they explicitly are not "forcing all their projects" to consumers.
In practice they are. The primary reason why everything is shipped as a single project is because that makes it easier to make available in different distros (they just enable everything), allowing it to become the "de facto" standard since it's available everywhere. There's a lot of projects that solve some of the systemd tools' in sometimes better ways, that never see the light of day because they don't have a trojan horse to ride in with.
There's a reason why projects like elogind exist, because there's only a very select few systemd tools that work without systemd at all
It's less "a new thing has been created with systemd" and more "a user interface was exposed for an existing functionality".
I appreciate and respect the KISS sentiment of doing one thing and doing it well, but oftentimes I also think open source programs' lack of integration with each other at a fundamental level is also one of its biggest downsides.
Being a Windows wizard unlike most folks around these parts, seeing and using the tight integrations between all the Windows subsystems is frankly marvelous and I wish Linux could have something like that.
It is said right there: the reason is to avoid the awkward SUID issues, and to have a privileged process create the process.
> If I chroot and then run0 will that process also run in the chroot?
That's an interesting question and it's not quite clear to me what should be the answer. For example, one legit answer might be that it would not work at all (in which case you would need to use some other tool that fits your needs), or another would be that the policy can decide what to do in that case.
But wouldn't you agree that it is better security hygiene to explicitly opt-in to inheriting properties when going to an elevated security level, instead of opting out from them?
But using sudo, or even root, in a chroot environment isn't good security practice in the first place. For example, the environment's /etc/sudoers, /etc/shadow, /etc/passwd can all be different (but actual user ids are shared), while the environment could mknod the root filesystem device, mount parts of the host under /usr and /home and quite possibly elevate its access to the host filesystem, in which case sudo in a chroot environment would interact with the host in perhaps ways that are not directly constrainable in sudoers in the first place.
I imagine it's possible to carefully construct the chroot environment and craft /etc/sudoers in a way to avoid this, but I wouldn't even try.
In a better contained system implemented with namespaces (as in docker) sudo should work as expected. I imagine e.g. with docker you could have a service that responds to the system dbus messages, either as a separate daemon in the container or something even provided by docker itself. Or, maybe the requests from the container could be forwarded to the main systemd, with information about the context the request comes from, and let it choose how to move forward.
In the end I don't think I agree that a sudo replacement exchanging messages directly with the systemd is the best way to go. There could be another daemon, run by systemd (or not), that handles these messages. I suppose in the case of dbus there is no "dbus firewall" to limit the access to dbus as one can limit access to Unix domain sockets with plain old permissions, though, so perhaps the security benefits would be minimal in the dbus architecture.
However, sudo needs to be user friendly and fail safe with decent information as to why its failed. Something that service files historically didn't do.
But, the way it's supported also needs to change, it almost certainly needs to be decoupled from systemd's release cycle.
I hope that we have all learnt from early systemd, and that we all won't take a "lets piss on each other's chips" approach. I'm too old you you lot to start flame warring over stuff you'll never actually fucking use.
since the beginning of systemd people have moaned about how complex it is and how we're reinventing the wheel. yet time and time again the people actually working on the project show that the solution they've come up with is the result of the problem they're facing on a daily basis. it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back", maybe think about whether or not you have a legitimate issue not being addressed by the solution proposed or if you are just getting rage baited by a headline.
Attack surface-wise, I don't think there's much difference between "sudo" being a part of the systemd package and not. Either way there's "sudo" code to be targeted, which package it's part of is just a technicality.
If the lesson you take from xz is "systemd bad" then you've really missed the wood for the trees.
The xz linkage was indirect through a systemd library that some systemd systems link into sshd.
They attacked the weakest link, and systemd was just a small pawn in that game. Sure, a smaller attack surface is better, but it's not like OpenSSHd has a small attack surface even without libsystemd. Not even in projects with a similar possibility of obscure "test data."
Except for libz, they are only linked indirectly through libsystemd.
> I don't see why linking to libxz would be so outrageous
The XZ Utils library is called liblzma, not libxz.
Ah, that invalidates my point re. obscure test data. Sloppy use of ldd. (I'm guessing it would be much harder making such an attack on a crypto library.)
Thanks.
The free-standing implementation: https://github.com/openssh/openssh-portable/commit/08f579231...
This is a serious problem and it makes it way harder to make things cross platform.
There is very little benefit for most userspace software to tie itself to systemd and by extension linux when otherwise it could be portable to any unix or unix like platform. Especially when an alternative, portable solution already exists and is well established.
systemd is scary for 3 reasons.
1. it is inscrutable. Debugging it is nearly impossible, so you had better hope you don’t get a buggy release, especially with how hostile the devs can be.
2. it is large, and growing. Lots of things it claims are optional are in reality: not really. This is fine until uou get something that really doesnt work well (systemd-resolved is consistently the largest reason I have connectivity issues, wether it be because it interferes with docker inter-container networking or because it needs to time-out when trying DNSSec to continue- or if it fights with my vpn provider for power over my resolvers etc). Due to my distro being very tied to it: I gotta keep using it and working around it.
3. The interface is irreplaceable. Why are their 13 init systems? Because init is a closed scope. To be a sucessful init you need to spawn processes and do it cheaply. Supervising processes after start? Noble, and there were implementations that could do that (CDDL licensed SMF from Solaris for example). However we have already reached the state where it will be literal man-decades of work to replace systemd as we will need to make any replacement bug compatible with systemd itself. Its the ultimate show stopper.
The implementation is the reference. Which is a large departure now from what came before.
In theory, yes. In practice, I had a lot of trouble ordering things correctly in non-trivial cases.
> it's quite annoying that the armchair linux experts complain about how "lol systemd is so stupid for reinventing the wheel, give me my shell scripts back"
I can only speak for myself, but I don't want the abysmal sysvinit scripts back. I just want a simple process supervision suite which is true to the UNIX way of doing things. The sysvinit/systemd dichotomy is false.
Runit and s6 are both very real alternatives to systemd. They lack a ton of features, but they are a way to reliably run services. They do use shell scripts, but not for the reasons sysvinit did. They are extremely simple and have a very small attack surface as a result. Runit itself is a spiritual descendant of daemontools which predate systemd by great many years.
The problem with systemd is that it's a mediocre solution to many problems. UNIX deserved better.
[Edit: whitespace]
"sysvinit" is an ill-defined concept, anyway, as every distribution had their own scripts and tooling around actual services, and sysvinit was generally only responsible for starting getty and launching the distribution's actual service system. How initscripts were created and how you managed them depended significantly on the distribution.
[1]: https://gitweb.gentoo.org/repo/gentoo.git/tree/net-vpn/tails...
The full system also cares about dependencies and parallel running. And this is not recent, but the black legend will never die.
Bonus: by having the shipped scripts under /etc, tools like etckeeper can show your changes as well as updates, all in one place.
All of these are taken em or leave em. But they're good expectations to have. I love the bazaar model, but Linux used to have so few common expectations, used to manage various bits so poorly. And now there's a much better base of capabilities, which have universal patterns of management that tend to apply to them (how etc files are laid how, being dbus and maybe varlink accessible). Linux hadn't been growing; when you came to a random systems you expected enormously little and typically got it, and what extras were available were scattered/random and often not particularly high quality or capable software. Systems has extended a much bigger base of competency & capability. That we can plug in a USB drive and systemd-home can create an isolated dynamic user out of that & let that user securely run their environment there is a neat as heck expectation. That our network manager supports setting up such a wide range of bridges and tunnels and taps and tuns is fantastic, is excellent. I don't have words for people turning their noses up at this better world; this is so much more competent & capable a world than where we were, gives us so much we all can now take for granted, and it's been done smartly, more mono-repo than monolithic, such that you can do alternatives. But many of these pieces are utterly without peer. And the consistency of operation you get, the predictability of use, from being under the same umbrella, is an ergonomic wonder that is unmatched.
I had a quick skim of the docs on systemd.io and there are quite a few documents but they don't seem particularly organized and I couldn't see a good architecture overview.
Edit; and as far as architecture goes they are all _separate_ programs. It's not a single large "systemd"
[citation needed]
And coreutils is expected because most people use GNU/Linux. Coreutils is that GNU userland part. And either way, the majority of coreutils are GNU implementations of standard unix components. You can use most software that depends on coreutils on other *nix platforms like BSD for this reason.
Whether you like that or not is for you to decide.
FreeDesktop also maintains a list of implementations (also including bindings so you have to read to find which ones are full impls vs libdbus bindings).
https://www.freedesktop.org/wiki/Software/DBusBindings/
I guess that depends on whether or not you consider the great number of CVEs caused by it don't matter enough.
You forget, the reason systemd was originally rationalized for its insertion into our trees was "boot times are too slow". Its chameleon-like nature and ability to solve the hastily described problem du jour seems to be its only consistently touted feature.
Bash scripts that start processes are ephemeral. If it's signal handling you want, that was your program's problem. Either that or your program didn't fork itself, which is a fish of an entirely different feather.
And now we have this sprawling mess of complexity and headache.
> It reduces features adds complexity...
The irony here is Wayland is part of a huge effort to decomplex an xserver into component parts. A really commendable initiative; the path forward while maintaining the X protocol probably was impractically hairy.
The Wayland protocol design had some glaring flaws, but saying that it adds complexity is unfair. It oversimplified; it would have benefited from some flexibility in providing a standard mechanism to let people inspect the buffers graphics buffers to be composited.
This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. I knew where the files were, what they were called. I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
The init system was probably peak of systemd, after that they started reinveting things in a more complicated way. Do we really need journald, systemd-boot, machinectl, systemd-networkd, sd-bus, systemd-resolved, systemd-nspawn? Do people actually use it all? Are there any metrics to show how many systems have it installed and in use?
The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.
You can then use `systemctl status <pid>` to identify the unit if you need to.
I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary.
EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well.
EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\",
Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.
Can you filter on search? How do you prevent applications from writing to Systemd at all?
If you don’t want a service to log to systemd, then don’t run it in systemd?
There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.
[2]: https://www.freedesktop.org/software/systemd/man/latest/syst...
The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work.
I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!"
Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it.
systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.
No they don't. They are good answers. If you want answers to a more general problem, then ask. This is not an emotive topic.
They suck in that the proposed method for accomplishing the task is a suck-ass downgrade from the previous ways the new way proposes to displace.
I beg to differ about emotive, because absolutely the other side of this fence behaves every bit as butt-hurt when challenged as I just did. You yourself just said these crappy commands were good, as a purely unfounded assertion. They are great because you just say they are great. That is even less objective than my rant. I at least explained what exactly I find so bad. Tell me more about not emotive.
Very well: I didn't say the commands were good or great. If you attempt a little objectivity you'll see it.
Before even going into the nature of the commands, I said that the answer does not match the question. The answers addressed details, while the details in the question were merely examples.
Missing the point is actually merely one of the at least two dimensions along which the answers suck. Thank you for reminding me about that.
No, the answers addressed the two scenarios in the question. That's why they don't "SUCK". They are good answers to the question as asked, and not the question as you dreamed it. If you spent ten words on asking the question of your dreams, rather than having a go at people for not answering it (whatever it is), you might have an answer.
And as far as learning the options goes, learning the basics of systemd and journalctl is much easier than learning how to, say, use a source control system effectively.
(And it's definitely worth trying out new source control systems -- for example, I think most people should at least check out Jujutsu, and many people are going to like it more than Git. As simple as Jujutsu is to use, learning it takes a lot longer than learning journalctl.)
I like systemd and have found it extremely useful. If it doesn’t fill a need for you then use something else.
Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.
I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.
With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal.
> let me cd into /var/log and see what filenames I can find
You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue.
> or grep everything until I can find a couple of words that make some sense so I can keep digging further
journalctl --grep, or more conveniently, journalctl and then / to search.
> The lack of explorability in journalctl
It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.
journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there.
> I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename.
My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe, but finding the logs in the first place was always a challenge. Systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.
Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.
Which involves glob’ing. I assume it allows to monitor several services at the same time.
Can systemd do that?
You sure would have forgotten about it when you tried to get onto another machine without the magic set of aliases
I am also prepared for someone to chime in that "log egress tools exist" to move up the Maslow's Hierarchy of Log Needs
With regular logs, I can go to /var/log, ls the dir, and get a nice list of what's being logged. There will be, for example, an `apache.log` file that has all the logs relevant to apache.
And you don't need to worry about "is this service rotating logs appropriately?"
In the old ways, that was apparent because of the presence of the log file.
Granted, this isn't the problem it used to be for us, however, it would come up because we'd have our services named things like `foo-ws` and knowing that `foo-ws` existed on a given box might be tricky were it not for the log files.
It's much easier in the systemd world to see the state of the system, the state of the service, and the logs of a service because it enforces this consistency.
a system user/admin has an intuition about files. saying that ‘journalctl -f -u’ (fu, indeed :) and whatever else is inherently undiscoverable, and is a.. basically orthogonal mechanism for handling what should be a simple task. i.e., viewing some logs. it’s far easier to compose and extend from files (what if i only care about the mtime of the log, for instance), than this.
look, i think systemd isn’t.. terrible. i also think it’s suffered a bit of complexity fetishisation, and it seems as though that this resulting complexity may have become invisible to you.
run0 doesn’t seem like a bad idea. but i am wincing a bit at the thought of unrestricted javascript determining access control.
Journald would be perfect if I didn't have to know a thing about it. Work in background, move stuff from stdin/stdout to /var/log/journald/service.log, move old stuff in /var/log/journald/archive/service/2024-04/15.log.gz. I'd be happy. Why do I need this cryptic CLI I don't understand. It brings nothing but pain.
There should be a command like "journalctl --top" that would list all logging services with amount of logs, oldest and latest log times, and last few lines of each, to improve that experience. Maybe there is already a way to do something similar?
That said, when I'm in a hurry and going to do a thing that I've been doing every day of my life, I'm really not a big fan of having to stop what I'm doing and reexamine a tool I've used all my life to figure out how to use it. I learned this stuff decades ago and I'm not anxious to relearn how to do it for a fractional improvement, and I'm not even really 100% sure that this new "inflate and snap" shoe model is actually an improvement, except for very specific use cases.
For how long?
I've been around for decades at this point with people telling me how the newest and greatest thing will obsolete everything that came before it.
The best way to look through logs is still to materialize them in as text in a files hierarchy and use find with grep to look for issues.
Debian switched to systemd about 12 years ago, so at least that long.
Perhaps it was available prior to then, but so were daemontools, upstart, and probably a variety of other alternatives.
Once you deal with that you’ll realize journald is just in the way and needs to be modified to output a human readable log by default.
It can largely do everything it replaces, but differently, in some cases with improvements. Every year it increases its scope of 'problems' its solved.
But the issue is it hasn't _ACTUALLY_ solved a problem for me in about a decade, its only introduced problems. Its replaced things that worked perfectly for me and millions of other users for years/decades with something that... also works... but differently. Now to accomplish the _SAME_ outcome without any personal benefit I have to re-learn their new and usually opinionated way.
There was a time I could log into a system, find logs, parse dates and do everything in these discussions reletave to logs, rotation. Regardless of service because the standards were close enough and the basic tooling (tail/find/tar/grep/awk etc.) worked everywhere always, and you never had to google even if the service was one you never worked on before.
For systemd its the opposite, its almost a guarantee you have to google something along the line when interacting with new systems and services. And as soon as their newly enforced opinions and way of doing things is getting comfortable they go and replace something else that was also working... perfectly fine.
If it actively solved problems for the majority of users we wouldn't hate it so much. Instead it requires more mental overhead than anything other than XORG to deal with and constantly manage than anything else in the linux ecosystem.
It does and we don't. The only "we" who are "hating it so much" is a tiny vocal minority.
I wouldn't call the opposition tiny, although the vocal portion of any group tends to be the smallest. This would be my first online comment on it so I'm certainly part of the non-vocal group you ignore, that in my experience is significantly larger than the vocal side of group you dismiss.
systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar.
I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.
I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.
Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends.
> --no-pager
To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.
With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines.
I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.
Systemd unit files make Linux feel like a rational OS.
At the end of the day, I just run Void (or MX Linux if I want something more Debian-flavored) and get on with my life.
You're presuming to speak for an awful lot of people there, on a topic that would be difficult to measure.
> since the beginning of systemd people have moaned
> it's quite annoying that the armchair linux experts complain
Now you're overgeneralizing, and doing so in a dismissive and patronizing way.
Here are a few examples of problems I have with systemd:
System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.
Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.
At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.
There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:
Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.
I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.
That is not what I said.
ctrl-alt-del skips iirc?
Granted, in a lot of cases the issue probably wasn't a big deal (the system is being shut down, the user is already logged out, do you really care that a system background process is being shut down cleanly?) but from systemd's perspective there's no difference between that and an actually business-critical process that should absolutely be allowed to terminate cleanly before unmounting the filesystem and powering off the machine.
Systemd was meant to replace Ubuntu's upstart, not sysv init. And it indeed systemd was better designed than upstart.
Sysv init is a horrendous hack. People should NOT use that. Upstart had limitations but is okay, and certainly much better than sysv init. Systemd is okay too (but has much more features).
Also as an historical note, both upstart and system was inspired by Apple's launchd.
NixOS was an early adopter of upstart but it later migrated to systemd (alongside Ubuntu itself)
"There’s a new tool in systemd, called “run0”. Or actually, it’s not a new tool, it’s actually the long-existing tool “systemd-run”, but when invoked under the “run0” name (via a symlink)".
systemd-run is very useful to run tasks with specific cgroups settings, or at a specific time. It asks for password whenever needed.