i honestly think he believes his stance is the only correct one. you can see it in the way he answered the questions.
the bit where he picks apart upstart, gentoo, etc., are nothing more than callous, myopic opinions yet he answers them with cheerful conviction.
technical arguments can be made for or against any piece of code. but, he dismisses other projects with a wave of his hand firmly stating that systemd is the "only" correct evolution.
i'm am neutral on systemd. do i think the old init system(s) are the right way? not really. do i think systemd is the right way? not really. would i be fine with systemd in the future? perhaps. would i be fine with another bit of code? perhaps.
but to dismiss the concerns of a large number of community members, to neglect or downright deny bugs and issues, to shoehorn a project into _everything_ while closing your ears to everyone else, these are my issues with the project.
if it works, great. but at least open your eyes and ears and work with everyone.
i migrated my home kit to freebsd from debian because i was having errors. failed boots. crashes. config problems. and i didn't have a decade of old crusty init scripts to begin with.
what used to be rock solid on my hardware became annoying to deal with.
i'm all for change and evolution in software/systems and i don't personally have a problem with poettering or what he is trying to do, but he's doing it in a piss poor way.
"...to make it clear, I think Upstart actually has its benefits. The source code is very, very nice, and it’s very simple, but I think it’s too simple. It doesn’t have this engine that can figure out what the computer is supposed to be doing."
I find myself wondering why the init is supposed to figure out what the computer is to be doing. That kind of "automagical" behavior is what drove me bonkers when trying to use Windows for more than a games console...
I dunno. If it's able to do what you want automatically that's amazing. If I can just plug the printer in and have it work that's amazing. If I can turn the wireless on my laptop on and it works that's amazing. And so on.
The problems arise when it doesn't automatically do what you want it to do and you don't have a good way of fixing it yourself. I'm not in a position to understand init systems so I will be in this camp if there's ever any problem with any init system. What matters to me, as the end user, is how likely it is to work automatically.
If systemd 'just works', then what's the problem? If systemd is broken... well, that's a problem, isn't it. But from what I've seen a lot (though not all, of course) of the arguments against it are highly political/philosophical and aren't based on whether it's actually working or not.
Hotplug isn't the job of systemd to begin with, though. It's the job of the device manager, which on Linux is usually udev (but alternatives like eudev, mdev, smdev and vdev exist) - udev being a part of the systemd repository and to an extent coupled with it (which will become complete when kdbus is merged into mainline Linux), however it's still a rather distinct piece of software that maintains the hardware database and listens to kernel uevents.
So if your init system is handling hotplug events, then that's some bad design. systemd doesn't stoop that low, though it still has plenty of mistakes - like putting the INI parser in PID1. As an example, even Apple put launchd's XML parser as a separate process!
There is tight interaction between udev and systemd. Heck, udev was folded into the systemd code (you find it now as a sub-dir of the systemd source tree over at freedesktop.org) because they didn't want code duplication...
This after having udev exist as a independent project for a decade.
These days the only way to download and install udev manually is to download the whole of systemd and then extract udev from that. Something that caused Gentoo to fork udev into eudev.
I find myself wondering why the init is supposed to figure out what the computer is to be doing.
I've never messed with systemd configuration too much, but is it much more complicated than a make-style "how do I get there from here? OK lets do that"?
That's actually a pretty good summary of the difference between Upstart and systemd, with systemd your services declare what they depend on and your runlevel is now a "target" that declares that it depends on certain services. It does dependency resolution just like a package manager does.
If that were true you could never start those services in sysvinit either. It works great because there aren't cyclic references and practically no conflicting services. The dependency tree will always be pretty simple.
I believe it's not about between magical in the general case, but about how you specify dependencies between services, an area where systemd and upstart work differently.
Reminds me why slackware packages do not have dependency resolution, because it invariably breaks. This most often in the sense of X wants Y while Z wants Y+-1. And so you are at an impasse because the dependency resolver can't handle having two versions of Y installed at the same time.
A recent failure of systemd i have seen was related to NFS. For some reason systemd would yank the network down before dismounting NFS.
If you don't unmount NFS before taking down the network, it will sit there forever waiting for that final server response.
Another example was where while mount would happily ignore a flawed fstab entry but mount the rest, systemd would bail on the whole mount task and so halt the boot process.
It's not really "automagical"; it's all quite well defined.
Honestly, I'll take a dozen 5-line unit files over a single "portable" init script that implements its own utility library, or yet another "simple" init script that's 90% standard boilerplate and still doesn't work because the 10% doing the real work doesn't check PID files correctly ("I can't just kill all processes called foo?") ...
Anyway, the point is that systemd models system states declaratively, solves the dependencies and executes the necessary steps required to reach that state, all without explicitly telling it what to do. If your dependencies are declared correctly, there is little danger of misbehaviour. The advantage to this is that parallelization is free and you never need explicit waits. In practice of course there will be misbehaving units and incomplete dependency declarations, but I think as admins gain experience with systemd those kinds of issues will drastically decrease. You'll still be able to run plain old scripts from unit files. I do that on my home server to start containers. since the contained process doesn't background itself and just prints to stdout and stderr, when started via systemd all its output will go into the journal and the tail shows up when I do "service container@foo status".
Analogous to SQL, I suppose something like "systemctl explain" might be useful. There's list-dependencies, which gets halfway there.
There have been hundreds of thousands of words written on how simple utilities can form the basis of supportable, complex systems and still retain the ability to be comprehensible. He has dismissed all that as beneath consideration, and he has the One True Vision™ on how Unix should be architected.
Don't get me wrong, hubris and self-righteousness have given us great things: GNU, OpenBSD, qmail, etc. But I don't think anyone would accuse rms or theo or djb of not being callous and myopic.
He has dismissed all that as beneath consideration, and he has the One True Vision™ on how Unix should be architected.
I don't see him passing a value judgement on how it should be done, just dismissing a particular criticism as poorly founded. If you look at the difference between the Unix and Linux development models, there is a point. There are a lot of properties to Unix so it's a bad argument to say "not Unixy enough" when you really mean something more along the lines of "components take on too many responsibilities at once and are too tightly coupled."
the whole of systemd sits in a single source tree in freedesktop.org. You don't find system-networkd in one tree, udev in another, systemd-init in a third.
In contrast, each part of XFCE, the window manager, the panel, etc, sit as their own source trees within the XFCE repository.
The interview wasn't as bad as I expected from the OP's opinion but this one seems mind-bogglingly wrong:
"What’s typical for Unix, for example, is that all the tools, the C library, the kernel, are all maintained in the same repository, right? And they’re released in sync, have the same coding style, the same build infrastructure, the same release cycles – everything’s the same. So you get the entire central part of the operating system like that. If people claim that, because we stick a lot of things into the Systemd repository, then it’s un-Unixish, then it’s absolutely the opposite. It’s more Unix-ish than Linux ever was!"
I don't even know how you could get this misconception. This really does sound clueless or trollish.
The rest of the interview is of a better quality and you follow his reasoning much better even if in the end you would still disagree with him.
He mentioned kernel, c libraries, and "all the tools". Well, kernel is in one repository. The (gnu) c library is in one repository. The tools, well the category of stuff like "ls" and "cat" is afaik in one repository. In similar manner, he seems to think that systemd should manage the set of most crucial processes of the base system.
I am not saying that he is correct, I am just interpreting!
Thank you. That's well said. I wish I could read all the piece with the pay per view part. But with this part he is absolutely sure , he is the righteous on every aspect of computing and it's history that he just tries to "create un realistic lines" to make his point.
He's talking about UNIX, not Linux, and he's 100% right. Look at the BSDs some time, or the Solaris forks.
The disconnect between the Linux kernel and basic userland has been a criticism of UNIX traditionalists for years. It's part of the reason why BSD has a few forks and Linux has hundreds of distributions.
I'm very well aware of the difference of Unix and Linux. I even smiled slighty as he also could just have claimed Linux to not be Unix and that would have been undisputable.
I actually don't know what you mean with "criticism of UNIX traditionalists for years" in regards to the Linux kernel and the userland.
I know of Linus' disregard of POSIX where it's crappy or doesn't make any sense anymore in his opinion but that can't extend to the userland as Linus doesn't do userland.
Some clarification or links to said criticism would be welcome.
> I actually don't know what you mean with "criticism of UNIX traditionalists for years" in regards to the Linux kernel and the userland.
There's a joke from the 90's that goes something like "FreeBSD is what you get when Unix hackers write a Unix clone for the PC. Linux is what you get when PC hackers write a Unix clone for the PC." From the point of view of BSD or proprietary Unix fans, the early Linux distros were just throwing together a haphazard collection of poorly tested components.
Yeah, and from what I've heard most Solaris (and proprietary Unix) installs wound up with a copy of the GNU utilities installed somewhere and added to the sysadmin's PATH because the bundled ones were shit. Before Linux came along, this was the main use of stuff like GNU coreutils. I'm not sure this is really an aspect of traditional Unix we want to be imitating.
I think he is referring to traditional Unix's where you would get all of the tools and such as part of the system. For OSS stuff it would be FreeBSD vs the Linux kernel.
I see now what he meant with "disconnect between the Linux kernel and basic userland". As others have already pointed out, I also don't think that this is an integral part of what being like Unix is and I stand by my first assessment of him being wrong, trollish, or as an additional option, throwing in this argument to detract from the real criticism.
The article doesn't go into much detail why having the kernel and the userland being maintained by the same group is better. Neither on theoretical grounds nor with actual examples.
On theoretical grounds, I would agree that it should be benefitial although in my personal experience (Solaris, FreeBSD, OSX, Windows with Cygwin, Linux (normal userland and busybox for low-spec devices), it's not a given that this means necessarily a better user experience.
Solaris used to have (some years ago, haven't used it lately) and OSX still has a really bad default set and the sooner you replace them with the GNU tools, the better for your peace of mind. I'd even go so far and claim that Windows with Cygwin is a better user experience than those two.
Lennart seems to harbor this very strange belief that developing your software in one repository makes you Unix-like, and that this alone is sufficient entirely.
So by this logic, a standalone Unix kernel is not Unix-like, but a fully integrated DOS clone, is.
In addition:
Then we convinced the Fedora Technical Committee to adopt it, and then Red Hat internal management accepted it for RHEL, and we managed to convince every committee that mattered, bit by bit.
So there was, in fact, a PR campaign.
So Journald is a requirement, and Udev is a requirement. But pretty much all other components are completely optional.
Though for some reason the build system doesn't let you disable a ton of these components by default.
The committers group is quite diverse, and for us it’s quite an exercise in making the diversity of the community be reflected in the diversity of the people who work on it.
Er, how do you think the Systemd developers convinced these technical teams to use Systemd? With bribes? Sexual favours? Or perhaps, you know, using the software itself?
It's funny. The lead developers of Arch Linux, OpenSUSE, Fedora, Mageia, Debian etc. etc. etc. have all chosen to use Systemd. Yet for some reason, many commentators assume these people -- distro technical leads -- simply don't have a clue, and have switched their projects to Systemd because Lennart is somehow forcing it on them.
All these distro developers, people a LOT smarter than me, and putting in more work than me, have chosen Systemd. That says a lot to me, and I'm not going to judge them. I'm still on the fence about a lot of Systemd, but to say that major distros have switched because of "PR campaigns" is ridiculous.
Hold up. Please don't invoke the argument from popularity, it simply doesn't work. The lead developers of major distros, as smart as they may be, do make mistakes (Debian with cdrkit, sticking with libav for too long, the PRNG bug, etc.) and they may not be fully in line with the intricacies and conceptual differences between process management, init, supervision, hotplug and so on. There's lots of misinformation from both sides and a lot of knowledgeable people (like those maintaining powerful daemontools derivatives) have never been ones to proselytize, but stick to their inner circles.
Furthermore, how are you sure the decision to adopt was entirely technical in nature? Perhaps they realized that the incoming systemd-integrated Linux desktop userland wasn't worth the maintenance hassle of trying to avoid it?
The way Lennart worded those sentences sounded funny. He is known to distort and exaggerate when promoting, for which I don't blame him, since his goal is to get his software adopted, but nonetheless. systemd proponents have been all too eager to completely dismiss any political meddling on part of systemd's adoption as totally unsubstantiated, insisting it was purely "technical merits". Lennart's words paint a subtly different picture, at the least.
Most people don't fully grasp the state of Unix process management, so it's no surprise they're susceptible. For example, the only alternatives that have been debated primarily were Upstart and OpenRC. That's a red flag.
"Please don't invoke the argument from popularity, it simply doesn't work."
Even when almost every distro has switched? If the situation were 50-50, I'd agree with you. But given that Systemd has been overwhelmingly adopted by distros, I think that says alot.
"The lead developers of major distros, as smart as they may be, do make mistakes"
Yes, but again, they surely haven't all made a giant mistake, right? I don't buy the "they've switched because of upcoming desktop userland" argument either. Distro devs -- like Linux users in general -- aren't ones to roll over easily.
With overwhelming support from distro developers, the people who know their stuff, Systemd must be doing something right. And if you look at the Benefits section here, you can see a lot of things that are appealing to distro devs: https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530
The distro developers likely all 'chose' systemd because they were forced to to continue using GNOME, which is the experience people expect as end-users on Linux.
Arch was one of the first to switch and while a decent number use gnome I sincerely doubt they made the choice because of gnome. Arch has a long history of alternative WM/DE It's not likely any individual one could force their hand on an architecture change like systemd.
Actually, the Arch devs said that if anyone stepped up and maintained the sysv scripts it would continue to be officially supported, no one did, and that was that.
I believe there's a lot of worry about systemd because what Pottering is doing is relatively new. To grossly oversimplify; the ecosystem of inits among Linux distros looked something like "sysvinit -or- spin your own." So when systemd comes along and starts getting integrated into damn near everything, there's naturally some alarm. There have been endless flame wars about init systems for almost 25 years now. It's almost unfathomable that every major distro has managed accepted systemd with relatively little drama.
I feel like "the marketing is too aggressive" or "they're being astroturfed" or "they're being forced" are all expected responses to what's going on.
Well Fedora/RHEL and the systemd people have something of a relationship.
I don't know the politics that went on with opensuse for it to be accepted there.
Debian has had a big dustup going about it for some years.
Ubuntu was pretty much waiting to see what Debian went with.
Arch was pretty much a dictatorial decree (and one "respin" is not adopting systemd iirc).
Gentoo is trying to stay init agnostic, and so has ended up forking udev to eudev (and Poettering has taken to making verbal pokes in their direction in systemd release announcements).
Slackware is making the choice if it ever comes to a head, but then it has been using BSD init rather than sysv or upstart.
1 Redhat pushed systemd into Rhel7
2 Crucial system parts were pr'd into hard dependencies on systemd
3 It's base idea and vision which is currently replaced with broader ones was good evolution. (better init system with woos)
4 technically most of these woos are only implemented with it
I am not trying to say systemd does not have it's merits. It has. It is trying hard to bring what it promises on the table. Unfortunately these promises mostly comes with traps like vendor lock-in and forced technicalities on other software like kernel.
It tries to be everything in a single pack with no space for other contenders.
These may be thought as well refuted ideas But I believe systemd cabal has the ability to refute with half assed realities. Like "It is not monolitic. it is modular." Reality it has compile time modularity like it is 70's.Also people report that modularity is oversold.
RHEL actually was one of the later distros to adopt systemd, so that line of reasoning doesn't make much sense.
> Crucial system parts were pr'd into hard dependencies on systemd
Can you tell which crucial system parts you mean? The thing with udev was that it's hard/impossible to build without systemd, however there is no dependency on it. The other thing that is brought up in this context is gnome 3, but that was already covered in the interview.
Gnome and logind is a crucial one. Also it brings it's pam dependency. It is a show stopper for slackware. Kdbus and udev is and will be other parts in this category.
>> Then we convinced the Fedora Technical Committee to adopt it, and then Red Hat internal management accepted it for RHEL, and we managed to convince every committee that mattered, bit by bit.
> So there was, in fact, a PR campaign.
So, how would you spread a piece of software without convincing people of it?
It should be interesting to investigate about how other technology shifts happened before. Were they an obvious revelation or the result of lobby/PR/... ?
> Lennart seems to harbor this very strange belief that developing your software in one repository makes you Unix-like, and that this alone is sufficient entirely.
He is saying that a trait of the Unix systems was the development in a single repository. In that particular aspect, a fully integrated DOS clone might be more Unix-like than, say, Linux. This is a response to the people who mistake the "Linux way" for the "Unix way".
The thing is, the aforementioned trait is far too generic - it's a characteristic of how development is structured, and has little or nothing to do with the more technical aspects of the Unix design, which is what is being contested.
It's not just Unix. Development in a single repository has been how almost all operating systems were done.
In fact, standalone kernels have usually been research projects throughout the years. Linux is a notable exception.
OpenBSD solved the Year 2038 problem by incompatibly changing the kernel and libc ABI in lockstep, which broke every single existing binary application using time_t - how's that for tight coupling...
Maintained in a single repository and maintained as a single blob of code is two different things. Can i get the systemd repository to hand over just udev as a tarball, and then compile it? Or do i have to grab the whole of systemd, extract udev from that, and then attempt to compile?
for example XFCE is maintained in a single repository, but each part is released on their own.
Who gives a shit about Unix like or not. Unix is dead, Linux is the new standard. Lennart is solving a very important problem. If you don't like it write your own or don't use it. But just STFU.
There's so much emotion wrapped up in this issue that there really isn't anything that this guy could say that would change the perceptions of a lot of people.
"if you do an init system but still invoke all the shell scripts and all the other things needed to bring up the system, you’ve only solved part of the problem."
That's exactly what I think is wrong with systemd. If a script bring a speed issue on start up, it should fixed by optimizing the script, use/create another script language with parallelism and higher pref, or even turn it into a compiled program. This is not supposed to be the problem of the init system developper.
In fact the only purpose of an init system is to start executables in a defined order, not more.
This is going to be like X, a huge pile of unmaintainable code, and that's definitely not the UNIX philosophy.
> In fact the only purpose of an init system is to start executables in a defined order, not more.
That's where the disagreement lies. I see the purpose of an init system as being to transition the machine from one state to another. Sometimes this means starting executables. Sometimes it means stopping them. Sometimes it involves more complicated tasks.
> This is going to be like X, a huge pile of unmaintainable code
That is exactly my opinion of the shell scripts that previously handled networking.
Yes, "init" is a bit of a misnomer, given that it also handles shutdown/runlevel transitions. I see the big difference here as being between the people who think that the job of the OS is to provide a simple base for executing user-defined scripts, and well, everyone else. I think that an appropriate analogy would be handcrafted goods versus mass production. Linux has been transformed into a wide variety of products at various times, and while I don't think the init scripts played a very large role in any of that, the idea that the OS is what you choose to turn it into does have strong credence in this community.
The other side of the debate, I think, are the people who feel like a collection of init scripts should be refactored somehow, and mostly this seems to involve rewriting things in C, adding dependency annotations to init files, and implementing support for cgroups. At least that's what OpenRC has been doing. Systemd unit files go a step further, but it's a good step in my opinion. Anything that any two scripts do should be refactored into a common method or library, and hopefully that process leaves very little for the init script or unit file to do.
It doesn't matter to me personally whether this stuff gets done in Bash or C; I can't really maintain either. I could probably write a unit file if I had to. I understand there are lots of people who have put lots of time into writing Bash scripts. Some sysadmins may even think that their job is to write scripts. These people should continue to write all the scripts that they want, because Linux is all about that, but maybe they should also let other people write a decent plumbing layer for Linux, because modern OSes are all about that.
> In fact the only purpose of an init system is to start executables in a defined order, not more.
A good init system should also do process supervision and integrate with the OS's resource-management subsystem (on Linux, cgroups). You could do that with an even bigger pile of shell scripts, of course, and some Linux distros towards the end of the sysvinit era have been trying to. But at some point a giant tangle of shell scripts which are full of copy/paste boilerplate and frequent bugs starts to look like not the best solution.
It's not like systemd is the first system to come to that conclusion, either. Solaris dropped sysvinit 10 years ago, and OSX dropped its BSD-ish init in the same year. Earlier than that, djb also wrote an init replacement called daemontools, which was a bit of a step in that direction, though a smaller one. That got some uptake but was hampered by some of the oddities of being djbware (e.g. for years it was "license-free software", and it was only intermittently maintained).
Are there any examples of a process supervisor that isn't also intended to replace the traditional init system? Even systems less ambitious than SMF/launchd/systemd, like daemontools and runit, aim to replace sysvinit, because otherwise you'd have to specify twice, in different ways, how each service should start and stop (though it's possible to set up configurations where that works).
And even though it can handle both init and process supervision, runit has a clear separation between the two tasks. The "runit" program handles init tasks. The "runsvdir" program handle process supervision. And each one is completely functional without the other.
I wonder how much reduced the grief would be if systemd could run on top of another init process as just a supervisor (similar to how daemontools work, and runit can work).
but then the argument is that systemd needs to be init to properly manage the cgroups etc.
At the moment I'm using OpenRC which does exactly that on top of sysvinit, as are most Gentoo users. You don't have to specify everything twice because init never actually knew how to start and stop services in the first place - with a traditional sysvinit configuration that was all handled by scripts that it started, and OpenRC replaces those scripts.
> unfortunately it is so primitive that in practice it is only used to spawn getty(8).
The AIX System Resource Controller is spawned by a record in /etc/inittab. (AIX actually comes with utilities for maintaining /etc/inittab such as mkitab.) So too is svscanboot in daemontools 0.75 and later. Gerrit Pape is currently having problems with the Debian package for running runit, precisely because runit has for some years been run from /etc/inittab in some configurations and in Debian 8 it is suddenly no longer a file whose existence is always guaranteed because it is part of an "essential" package. There are some other packages in Debian 7 that have undocumented dependencies from the existence of an /etc/inittab file, because they too read/write it for their own purposes. I have yet to verify whether they've been fixed for Debian 8.
Don't think for one second that in the second decade of the 21st century people only ever use /etc/inittab for getty. (-:
This is one of the quieter on-going problems for Debian 8. It switches one to using systemd, because that's the Debian 8 default; but there's no upgrade path or compatibility mechanism for /etc/inittab. Every few months, another "I was (unknowingly) using /etc/inittab; I upgraded; it broke." person seems to pop up.
Gerrit Pape's runit has "runit" (exec()ed from "runit-init") as a system manager running as process #1 doing the system management tasks that only process #1 can do, and "runsvdir" as a service manager that runs as some other process.
Daniel J. Bernstein's daemontools, Adam Sampson's freedt, and Bruce Guenter's daemontools-encore do not, strictly speaking (and pace Paul Jarc's patches), have a program for running as process #1. They all have "svscanboot", which is to be spawned by process #1 and which runs "svscan" as a service manager.
Laurent Bercot's s6 does not have a program for running as process #1. Its "s6-svscan" is a service manager, usually run as some other process. And its "s6-svscanboot" sample program is analogous to "svscanboot". It comes with examples for writing one's own program to run as process #1, but "s6 cannot provide live examples without becoming system-specific" (http://skarnet.org./software/s6/s6-svscan-1.html).
Wayne Marshall's perp has "perpd" as the service manager. This can be started by "perpboot", which (like "svscanboot") is spawned by process #1 rather than run as process #1. There's no program in the package for doing process #1 tasks.
nosh has a program for process #1, "system-manager", and a program that runs in another process to do service management, "service-manager".
All of these aim at replacing System 5 rc. Only two in fact offer in-the-box replacements for System 5 init. Do not conflate rc and init. The person that you were replying to was wrong; hence the basis for your question was wrong. Learn from "system-manager"'s manual page, "runit"'s manual page, what Paul Jarc found when he experimented with running svscan as process #1, and indeed "systemd"'s manual page to an extent. "init's job" is more than "starting a process supervisor".
Okay. So, init starts the process supervisor. Then, the process supervisor starts everything else.
Then all of a sudden, something goes wrong and the process supervisor crashes. Init then inherits all the children, and has no clue what's going on with them, so your system is hosed.
What benefit is there to having the init be separate from the process supervisor?
EDIT: let me just expand on this a bit... Monit and other process supervisors do more than just manage the lifecycle of a process; they can run various checks to ensure that a service actually works. Systemd's process supervision is limited to knowing the current state of the process (running, stopped by admin, failed to start, constantly crashing, or optionally: not responding to watchdog), so systemd in no way makes monitoring daemons redundant.
When the traditional sysvinit starts and stops processes, it actually has no clue what it's doing and thus init scripts need to rely on PID files and other hacks to provide basic functionality.
I would like to know why you think that the process that starts and stops processes should not be interested in whether the processes are actually running or not.
Your second statement misses the point. Your process supervisor can be ultimately stable and never crash, but if it does crash, it's just as fatal as init crashing.
systemd PID 1 won't be rendered prone to crashing just because it contains more than trivial amounts of code. If that were the case, then surely the Linux kernel would be crashing every fifteen minutes, considering how much code it contains.
As far as I'm aware, the only component that can screw up systemd is dbus, and since the relevant parts are moving into the kernel, you won't just be able to hose your system by killing the dbus daemon accidentally.
I have yet to see an argument for process supervision functionality not existing in PID 1, besides simply stating that it must be so. Meanwhile, an init which is guaranteed to know whether the processes it starts (or stops!) are actually running is able to behave much more intelligently than scripts sending signals to PIDs that hopefully correspond to the correct process.
No, your process supervisor crashing won't be as fatal as init crashing, because init crashing is an instant kernel panic under Linux. Also, very few projects are as robust as the Linux kernel or have development practices that are as good, and systemd is unlikely to be one of them.
When the supervisor crashes, which should occur extremely rarely in any case, the state of which services are running is lost, and its child processes are re-parented to PID1, so when a new instance of the supervisor starts it cannot tell which services are running, and which of the running processes belong to which service. During the time the supervisor needs to re-start (presumably init would respawn it?), some of the running processes could exit without notice.
What is gained by restarting then? You'll likely want to reboot to get the system into a consistent state anyway.
Here is a war story of an embedded developer who actually created his own init system with separate supervisor process, and found that it doesn't actually make the system as a whole more robust:
Edited to add: Also, is your comment about bad development practices in the systemd project purely based on statistical conjecture (which would mean it applies to every single project, except of course the Linux kernel where presumably you have personally observed the absence of bad practices), or do you have anything to back that up?
Lovely exchange in those LWN comments about monolithic vs modular. In particularly how the proponents are (perhaps willfully) confusing public and private APIs...
> When the traditional sysvinit starts and stops processes, it actually has no clue what it's doing and thus init scripts need to rely on PID files and other hacks to provide basic functionality.
Tooting my own horn here, but I wrote a filesystem called runfs that specifically addresses this. A service writes a PID file to runfs, and runfs automatically removes it once the process dies.
Then the process supervisor can then launch your processes. But then all init is doing is launching your process supervisor. We can just drop a step by providing the process supervisor executable to the kernel. There is no need for init anymore.
> But then all init is doing is launching your process supervisor.
This "all init does/needs to do" notion is oft-repeated, but it is oft-repeated rubbish. It is a notion commonly held by people who have never actually written a process #1 program that functions in production systems; because experience (as I can attest) teaches otherwise. There are, in fact, several things that various operating system kernels and other programs demand of process #1 that one simply cannot escape. People think that fork()ing things and acting as the parent of orphaned processes is the prime function. Ironically, dealing with orphaned processes is (with recent Linux kernels) a part the system that one can largely factor out of process #1 into other processes, whilst the things that people usually forget in their off-the-top-of-the-head designs for init (such as handling SIGINT, SIGPWR, SIGWINCH, and so forth sent from the kernel and enacting the various system state change requests) are the parts that one cannot. To see what one actually has no choice but to do in process #1 programs, look at the overlaps in the operation of Gerrit Pape's runit, Felix von Leitner's minit, and the system-manager program from the nosh package. There are several things for a /system manager/ to do when there's a separate /service manager/.
> It's not like systemd is the first system to come to that conclusion, either. [...] Earlier than that, djb also wrote an init replacement called daemontools, [...]
daemontools gained svscan in 1998, and svscanboot in 2001. Years earlier, in AIX version 3.2 released in January 1992, IBM introduced its (then) new System Resource Controller. This was, and still is, a service manager that is spawned by the initial user process, that has the task of supervising daemon processes, bringing them up and down in response to administrative command, logging control activity, and notifying about failures to run.
You'll find things like the srcmstr process; the startsrc, stopsrc, and lssrc commands for starting, stopping, and listing services; the whole notion of signals to control daemons (SIGTERM, SIGNORM, SIGFORCE, et al.); entry 5.2.1 in the Usenet comp.mail.sendmail FAQ which talks about how fork-and-exit-the-main-process by Sendmail breaks service management; all of the "How do I get my too-clever-for-its-own-good daemon to work with this new service management system?" discussions in the mid-1990s; and IBM's documented advice from at least as early as 1995 on how to write daemons properly, achingly familiar.
One can look to mainframe and minicomputer operating systems before that, and also to Windows NT 3.1 which had the initial user process spawning the Service Control Manager, released in 1993. But in the narrow Unix+Linux worldview, credit should go to AIX for being the first operating system released where people had come to that conclusion, approaching a quarter of a century ago.
> So we slowly started doing stuff that all the other Linux distros did, and implemented that in simple C code that was fast and parallelised.
That's part of my issue with systemd: replacing safe (-r) shell scripts with C. Sure, it's fast, but I trust C about as far as I can throw it (and yes, I realise that the Linux kernel is written in C: it constantly has local root exploits, which proves my point). I really wish that projects like systemd were written in something of like Ocaml, Go, Rust, Common Lisp—anything fast enough but safer.
Also, it probably doesn't help Lennart that he looks like the Rolfe Gruber in the Sound of Music…
While bugs happen in any language, shell scripts are far easier to debug than C. You tend to not get nasty interactions spanning multiple source files in shell, where the bugs tend to on a single line (like in that Valve example, where rm should have never been used in such a careless/negligent manner). On the other hand, are you sure you've checked for all of the potential undefined behavior problems in your C programs? These are far more numerous[1] than many people realize.
At its core, they are the very same commands you would give manually. but wrapped in basic programming logic (if clauses etc).
thus you can walk the shell script and get a good feel for why the script is doing what it is doing.
If you want to do that with a binary, you pretty much have to either hit it with a debugger (i think a previous thread here on Hacker News actually recommended using strace to inspect systemd boot behavior) or have the source code sit beside it for reference.
It was not an attack, just a statement of fact. He looks just like a movie villain; it's not his fault, and no-one should hold it against him, but I wonder if it hurts him anyway.
The part about Gnome and logind vs consolekit could have been more detailed.
Poettering was sitting as co-maintainer of consolekit for some time. Then as logind got underway, the maintenance was to pass to Ubuntu under a new project name. But i can't find any trace of this actually happening.
After that the state of consolekit as a project seems to have been in limbo, until recently when a XFCE dev forked the existing code to github as consolekit2.
Apparently said dev would have loved to continue consolekit in place rather than fork, but the final email list traffic between Poettering and Ubuntu gave him the impression that Poettering wanted to see consolekit dead.
So it may well be that people didn't maintain the consolekit code in Gnome because they had the impression that consolekit was dead and buried.
but then Gnome and systemd devs seems to share a "one true way" attitude.
Things like this makes me wonder if RH changed once Oracle bought Sun and then forked RHEL into Oracle Linux. Thus Oracle went from being something of a partner to being a competitor, and RH rolled back on various behaviors (package patch sets etc).
"But we believed that an init system should work the other way around, where you say: this is where I want to go to, and you figure out the rest. Because of that design, Upstart was very simple, but it put a lot of complexity on admins and developers, because you actually had to write down all these rules"
...and this is the problem. Pushing this kind of complexity into PID 1 (or however systemd achieves this) is exactly the wrong solution. The init system should not be clever, or contain an "engine that can figure out what the computer is supposed to be doing" as he says later. This really smells of second system syndrome.
In reality even with a simple init that pushes complexity to the admins, end users rarely incur the cost. The distribution authors set up init as needed and end users don't need to worry unless they makes changes. Then you get the best of both worlds: easy maintainability and a simple, reliable init system.
90 comments
[ 7.1 ms ] story [ 212 ms ] threadthe bit where he picks apart upstart, gentoo, etc., are nothing more than callous, myopic opinions yet he answers them with cheerful conviction.
technical arguments can be made for or against any piece of code. but, he dismisses other projects with a wave of his hand firmly stating that systemd is the "only" correct evolution.
i'm am neutral on systemd. do i think the old init system(s) are the right way? not really. do i think systemd is the right way? not really. would i be fine with systemd in the future? perhaps. would i be fine with another bit of code? perhaps.
but to dismiss the concerns of a large number of community members, to neglect or downright deny bugs and issues, to shoehorn a project into _everything_ while closing your ears to everyone else, these are my issues with the project.
if it works, great. but at least open your eyes and ears and work with everyone.
i migrated my home kit to freebsd from debian because i was having errors. failed boots. crashes. config problems. and i didn't have a decade of old crusty init scripts to begin with.
what used to be rock solid on my hardware became annoying to deal with.
i'm all for change and evolution in software/systems and i don't personally have a problem with poettering or what he is trying to do, but he's doing it in a piss poor way.
"...to make it clear, I think Upstart actually has its benefits. The source code is very, very nice, and it’s very simple, but I think it’s too simple. It doesn’t have this engine that can figure out what the computer is supposed to be doing."
is callous and myopic?
The problems arise when it doesn't automatically do what you want it to do and you don't have a good way of fixing it yourself. I'm not in a position to understand init systems so I will be in this camp if there's ever any problem with any init system. What matters to me, as the end user, is how likely it is to work automatically.
If systemd 'just works', then what's the problem? If systemd is broken... well, that's a problem, isn't it. But from what I've seen a lot (though not all, of course) of the arguments against it are highly political/philosophical and aren't based on whether it's actually working or not.
So if your init system is handling hotplug events, then that's some bad design. systemd doesn't stoop that low, though it still has plenty of mistakes - like putting the INI parser in PID1. As an example, even Apple put launchd's XML parser as a separate process!
This after having udev exist as a independent project for a decade.
These days the only way to download and install udev manually is to download the whole of systemd and then extract udev from that. Something that caused Gentoo to fork udev into eudev.
I've never messed with systemd configuration too much, but is it much more complicated than a make-style "how do I get there from here? OK lets do that"?
See the 'On Upstart' section here [1].
1: http://0pointer.de/blog/projects/systemd.html
A recent failure of systemd i have seen was related to NFS. For some reason systemd would yank the network down before dismounting NFS.
If you don't unmount NFS before taking down the network, it will sit there forever waiting for that final server response.
Another example was where while mount would happily ignore a flawed fstab entry but mount the rest, systemd would bail on the whole mount task and so halt the boot process.
This isn't specific to systemd – I had to deploy workarounds to the same bug in RHEL, Debian, Ubuntu, FreeBSD, etc.
Honestly, I'll take a dozen 5-line unit files over a single "portable" init script that implements its own utility library, or yet another "simple" init script that's 90% standard boilerplate and still doesn't work because the 10% doing the real work doesn't check PID files correctly ("I can't just kill all processes called foo?") ...
Anyway, the point is that systemd models system states declaratively, solves the dependencies and executes the necessary steps required to reach that state, all without explicitly telling it what to do. If your dependencies are declared correctly, there is little danger of misbehaviour. The advantage to this is that parallelization is free and you never need explicit waits. In practice of course there will be misbehaving units and incomplete dependency declarations, but I think as admins gain experience with systemd those kinds of issues will drastically decrease. You'll still be able to run plain old scripts from unit files. I do that on my home server to start containers. since the contained process doesn't background itself and just prints to stdout and stderr, when started via systemd all its output will go into the journal and the tail shows up when I do "service container@foo status".
Analogous to SQL, I suppose something like "systemctl explain" might be useful. There's list-dependencies, which gets halfway there.
There have been hundreds of thousands of words written on how simple utilities can form the basis of supportable, complex systems and still retain the ability to be comprehensible. He has dismissed all that as beneath consideration, and he has the One True Vision™ on how Unix should be architected.
Don't get me wrong, hubris and self-righteousness have given us great things: GNU, OpenBSD, qmail, etc. But I don't think anyone would accuse rms or theo or djb of not being callous and myopic.
I don't see him passing a value judgement on how it should be done, just dismissing a particular criticism as poorly founded. If you look at the difference between the Unix and Linux development models, there is a point. There are a lot of properties to Unix so it's a bad argument to say "not Unixy enough" when you really mean something more along the lines of "components take on too many responsibilities at once and are too tightly coupled."
the whole of systemd sits in a single source tree in freedesktop.org. You don't find system-networkd in one tree, udev in another, systemd-init in a third.
In contrast, each part of XFCE, the window manager, the panel, etc, sit as their own source trees within the XFCE repository.
"What’s typical for Unix, for example, is that all the tools, the C library, the kernel, are all maintained in the same repository, right? And they’re released in sync, have the same coding style, the same build infrastructure, the same release cycles – everything’s the same. So you get the entire central part of the operating system like that. If people claim that, because we stick a lot of things into the Systemd repository, then it’s un-Unixish, then it’s absolutely the opposite. It’s more Unix-ish than Linux ever was!"
I don't even know how you could get this misconception. This really does sound clueless or trollish.
The rest of the interview is of a better quality and you follow his reasoning much better even if in the end you would still disagree with him.
I am not saying that he is correct, I am just interpreting!
The disconnect between the Linux kernel and basic userland has been a criticism of UNIX traditionalists for years. It's part of the reason why BSD has a few forks and Linux has hundreds of distributions.
I actually don't know what you mean with "criticism of UNIX traditionalists for years" in regards to the Linux kernel and the userland.
I know of Linus' disregard of POSIX where it's crappy or doesn't make any sense anymore in his opinion but that can't extend to the userland as Linus doesn't do userland.
Some clarification or links to said criticism would be welcome.
There's a joke from the 90's that goes something like "FreeBSD is what you get when Unix hackers write a Unix clone for the PC. Linux is what you get when PC hackers write a Unix clone for the PC." From the point of view of BSD or proprietary Unix fans, the early Linux distros were just throwing together a haphazard collection of poorly tested components.
"of" meaning "belonging to" not "directed at." Confused me for a bit too.
It's a bit trollish though.
https://www.over-yonder.net/~fullermd/rants/bsd4linux/03
I see now what he meant with "disconnect between the Linux kernel and basic userland". As others have already pointed out, I also don't think that this is an integral part of what being like Unix is and I stand by my first assessment of him being wrong, trollish, or as an additional option, throwing in this argument to detract from the real criticism.
The article doesn't go into much detail why having the kernel and the userland being maintained by the same group is better. Neither on theoretical grounds nor with actual examples.
On theoretical grounds, I would agree that it should be benefitial although in my personal experience (Solaris, FreeBSD, OSX, Windows with Cygwin, Linux (normal userland and busybox for low-spec devices), it's not a given that this means necessarily a better user experience.
Solaris used to have (some years ago, haven't used it lately) and OSX still has a really bad default set and the sooner you replace them with the GNU tools, the better for your peace of mind. I'd even go so far and claim that Windows with Cygwin is a better user experience than those two.
So by this logic, a standalone Unix kernel is not Unix-like, but a fully integrated DOS clone, is.
In addition:
Then we convinced the Fedora Technical Committee to adopt it, and then Red Hat internal management accepted it for RHEL, and we managed to convince every committee that mattered, bit by bit.
So there was, in fact, a PR campaign.
So Journald is a requirement, and Udev is a requirement. But pretty much all other components are completely optional.
Though for some reason the build system doesn't let you disable a ton of these components by default.
The committers group is quite diverse, and for us it’s quite an exercise in making the diversity of the community be reflected in the diversity of the people who work on it.
This is not as idealistic as Lennart portrays it. See Fallacy #13 here: http://judecnelson.blogspot.com/2014/09/systemd-biggest-fall...
Er, how do you think the Systemd developers convinced these technical teams to use Systemd? With bribes? Sexual favours? Or perhaps, you know, using the software itself?
It's funny. The lead developers of Arch Linux, OpenSUSE, Fedora, Mageia, Debian etc. etc. etc. have all chosen to use Systemd. Yet for some reason, many commentators assume these people -- distro technical leads -- simply don't have a clue, and have switched their projects to Systemd because Lennart is somehow forcing it on them.
All these distro developers, people a LOT smarter than me, and putting in more work than me, have chosen Systemd. That says a lot to me, and I'm not going to judge them. I'm still on the fence about a lot of Systemd, but to say that major distros have switched because of "PR campaigns" is ridiculous.
Furthermore, how are you sure the decision to adopt was entirely technical in nature? Perhaps they realized that the incoming systemd-integrated Linux desktop userland wasn't worth the maintenance hassle of trying to avoid it?
The way Lennart worded those sentences sounded funny. He is known to distort and exaggerate when promoting, for which I don't blame him, since his goal is to get his software adopted, but nonetheless. systemd proponents have been all too eager to completely dismiss any political meddling on part of systemd's adoption as totally unsubstantiated, insisting it was purely "technical merits". Lennart's words paint a subtly different picture, at the least.
Most people don't fully grasp the state of Unix process management, so it's no surprise they're susceptible. For example, the only alternatives that have been debated primarily were Upstart and OpenRC. That's a red flag.
Even when almost every distro has switched? If the situation were 50-50, I'd agree with you. But given that Systemd has been overwhelmingly adopted by distros, I think that says alot.
"The lead developers of major distros, as smart as they may be, do make mistakes"
Yes, but again, they surely haven't all made a giant mistake, right? I don't buy the "they've switched because of upcoming desktop userland" argument either. Distro devs -- like Linux users in general -- aren't ones to roll over easily.
With overwhelming support from distro developers, the people who know their stuff, Systemd must be doing something right. And if you look at the Benefits section here, you can see a lot of things that are appealing to distro devs: https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530
And i think at least one Arch offshot is not adopting systemd at this time.
I feel like "the marketing is too aggressive" or "they're being astroturfed" or "they're being forced" are all expected responses to what's going on.
I don't know the politics that went on with opensuse for it to be accepted there.
Debian has had a big dustup going about it for some years.
Ubuntu was pretty much waiting to see what Debian went with.
Arch was pretty much a dictatorial decree (and one "respin" is not adopting systemd iirc).
Gentoo is trying to stay init agnostic, and so has ended up forking udev to eudev (and Poettering has taken to making verbal pokes in their direction in systemd release announcements).
Slackware is making the choice if it ever comes to a head, but then it has been using BSD init rather than sysv or upstart.
1 Redhat pushed systemd into Rhel7 2 Crucial system parts were pr'd into hard dependencies on systemd 3 It's base idea and vision which is currently replaced with broader ones was good evolution. (better init system with woos) 4 technically most of these woos are only implemented with it
I am not trying to say systemd does not have it's merits. It has. It is trying hard to bring what it promises on the table. Unfortunately these promises mostly comes with traps like vendor lock-in and forced technicalities on other software like kernel.
It tries to be everything in a single pack with no space for other contenders.
These may be thought as well refuted ideas But I believe systemd cabal has the ability to refute with half assed realities. Like "It is not monolitic. it is modular." Reality it has compile time modularity like it is 70's.Also people report that modularity is oversold.
> Crucial system parts were pr'd into hard dependencies on systemd
Can you tell which crucial system parts you mean? The thing with udev was that it's hard/impossible to build without systemd, however there is no dependency on it. The other thing that is brought up in this context is gnome 3, but that was already covered in the interview.
> So there was, in fact, a PR campaign.
So, how would you spread a piece of software without convincing people of it?
I'm not sure what's bad about it?
He is saying that a trait of the Unix systems was the development in a single repository. In that particular aspect, a fully integrated DOS clone might be more Unix-like than, say, Linux. This is a response to the people who mistake the "Linux way" for the "Unix way".
It's not just Unix. Development in a single repository has been how almost all operating systems were done.
In fact, standalone kernels have usually been research projects throughout the years. Linux is a notable exception.
https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10
OpenBSD solved the Year 2038 problem by incompatibly changing the kernel and libc ABI in lockstep, which broke every single existing binary application using time_t - how's that for tight coupling...
http://www.openbsd.org/papers/eurobsdcon_2013_time_t/
for example XFCE is maintained in a single repository, but each part is released on their own.
Seemed like quite a clear interview to me, and interesting to hear a bit of the history behind such a fundamentally useful piece of software.
That's exactly what I think is wrong with systemd. If a script bring a speed issue on start up, it should fixed by optimizing the script, use/create another script language with parallelism and higher pref, or even turn it into a compiled program. This is not supposed to be the problem of the init system developper.
In fact the only purpose of an init system is to start executables in a defined order, not more.
This is going to be like X, a huge pile of unmaintainable code, and that's definitely not the UNIX philosophy.
That's where the disagreement lies. I see the purpose of an init system as being to transition the machine from one state to another. Sometimes this means starting executables. Sometimes it means stopping them. Sometimes it involves more complicated tasks.
> This is going to be like X, a huge pile of unmaintainable code
That is exactly my opinion of the shell scripts that previously handled networking.
The other side of the debate, I think, are the people who feel like a collection of init scripts should be refactored somehow, and mostly this seems to involve rewriting things in C, adding dependency annotations to init files, and implementing support for cgroups. At least that's what OpenRC has been doing. Systemd unit files go a step further, but it's a good step in my opinion. Anything that any two scripts do should be refactored into a common method or library, and hopefully that process leaves very little for the init script or unit file to do.
It doesn't matter to me personally whether this stuff gets done in Bash or C; I can't really maintain either. I could probably write a unit file if I had to. I understand there are lots of people who have put lots of time into writing Bash scripts. Some sysadmins may even think that their job is to write scripts. These people should continue to write all the scripts that they want, because Linux is all about that, but maybe they should also let other people write a decent plumbing layer for Linux, because modern OSes are all about that.
A good init system should also do process supervision and integrate with the OS's resource-management subsystem (on Linux, cgroups). You could do that with an even bigger pile of shell scripts, of course, and some Linux distros towards the end of the sysvinit era have been trying to. But at some point a giant tangle of shell scripts which are full of copy/paste boilerplate and frequent bugs starts to look like not the best solution.
It's not like systemd is the first system to come to that conclusion, either. Solaris dropped sysvinit 10 years ago, and OSX dropped its BSD-ish init in the same year. Earlier than that, djb also wrote an init replacement called daemontools, which was a bit of a step in that direction, though a smaller one. That got some uptake but was hampered by some of the oddities of being djbware (e.g. for years it was "license-free software", and it was only intermittently maintained).
And even though it can handle both init and process supervision, runit has a clear separation between the two tasks. The "runit" program handles init tasks. The "runsvdir" program handle process supervision. And each one is completely functional without the other.
but then the argument is that systemd needs to be init to properly manage the cgroups etc.
The AIX System Resource Controller is spawned by a record in /etc/inittab. (AIX actually comes with utilities for maintaining /etc/inittab such as mkitab.) So too is svscanboot in daemontools 0.75 and later. Gerrit Pape is currently having problems with the Debian package for running runit, precisely because runit has for some years been run from /etc/inittab in some configurations and in Debian 8 it is suddenly no longer a file whose existence is always guaranteed because it is part of an "essential" package. There are some other packages in Debian 7 that have undocumented dependencies from the existence of an /etc/inittab file, because they too read/write it for their own purposes. I have yet to verify whether they've been fixed for Debian 8.
Don't think for one second that in the second decade of the 21st century people only ever use /etc/inittab for getty. (-:
This is one of the quieter on-going problems for Debian 8. It switches one to using systemd, because that's the Debian 8 default; but there's no upgrade path or compatibility mechanism for /etc/inittab. Every few months, another "I was (unknowingly) using /etc/inittab; I upgraded; it broke." person seems to pop up.
Yes. You are conflating System 5 init and System 5 rc.
The IBM AIX srcmstr program is intended to be run by init, not to replace it. ("The SRC is operationally independent of the init command." - http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com... "You can also start the SRC from the command line, a profile, or a shell script, but there are several reasons for starting it during initialization" - http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com...)
Gerrit Pape's runit has "runit" (exec()ed from "runit-init") as a system manager running as process #1 doing the system management tasks that only process #1 can do, and "runsvdir" as a service manager that runs as some other process.
Daniel J. Bernstein's daemontools, Adam Sampson's freedt, and Bruce Guenter's daemontools-encore do not, strictly speaking (and pace Paul Jarc's patches), have a program for running as process #1. They all have "svscanboot", which is to be spawned by process #1 and which runs "svscan" as a service manager.
Laurent Bercot's s6 does not have a program for running as process #1. Its "s6-svscan" is a service manager, usually run as some other process. And its "s6-svscanboot" sample program is analogous to "svscanboot". It comes with examples for writing one's own program to run as process #1, but "s6 cannot provide live examples without becoming system-specific" (http://skarnet.org./software/s6/s6-svscan-1.html).
Wayne Marshall's perp has "perpd" as the service manager. This can be started by "perpboot", which (like "svscanboot") is spawned by process #1 rather than run as process #1. There's no program in the package for doing process #1 tasks.
nosh has a program for process #1, "system-manager", and a program that runs in another process to do service management, "service-manager".
All of these aim at replacing System 5 rc. Only two in fact offer in-the-box replacements for System 5 init. Do not conflate rc and init. The person that you were replying to was wrong; hence the basis for your question was wrong. Learn from "system-manager"'s manual page, "runit"'s manual page, what Paul Jarc found when he experimented with running svscan as process #1, and indeed "systemd"'s manual page to an extent. "init's job" is more than "starting a process supervisor".
Then all of a sudden, something goes wrong and the process supervisor crashes. Init then inherits all the children, and has no clue what's going on with them, so your system is hosed.
What benefit is there to having the init be separate from the process supervisor?
EDIT: let me just expand on this a bit... Monit and other process supervisors do more than just manage the lifecycle of a process; they can run various checks to ensure that a service actually works. Systemd's process supervision is limited to knowing the current state of the process (running, stopped by admin, failed to start, constantly crashing, or optionally: not responding to watchdog), so systemd in no way makes monitoring daemons redundant.
When the traditional sysvinit starts and stops processes, it actually has no clue what it's doing and thus init scripts need to rely on PID files and other hacks to provide basic functionality.
I would like to know why you think that the process that starts and stops processes should not be interested in whether the processes are actually running or not.
So obviously that's not a good idea.
Additionaly it bears the problem that you can't upgrade your process supervisor without rebooting.
So what benefits does it actually provide?
systemd PID 1 won't be rendered prone to crashing just because it contains more than trivial amounts of code. If that were the case, then surely the Linux kernel would be crashing every fifteen minutes, considering how much code it contains.
As far as I'm aware, the only component that can screw up systemd is dbus, and since the relevant parts are moving into the kernel, you won't just be able to hose your system by killing the dbus daemon accidentally.
I have yet to see an argument for process supervision functionality not existing in PID 1, besides simply stating that it must be so. Meanwhile, an init which is guaranteed to know whether the processes it starts (or stops!) are actually running is able to behave much more intelligently than scripts sending signals to PIDs that hopefully correspond to the correct process.
What is gained by restarting then? You'll likely want to reboot to get the system into a consistent state anyway.
Here is a war story of an embedded developer who actually created his own init system with separate supervisor process, and found that it doesn't actually make the system as a whole more robust:
https://lwn.net/Articles/623527/
For a good disucssion of the trade-offs involved see this comment by JdeBP:
https://news.ycombinator.com/item?id=8384251
Edited to add: Also, is your comment about bad development practices in the systemd project purely based on statistical conjecture (which would mean it applies to every single project, except of course the Linux kernel where presumably you have personally observed the absence of bad practices), or do you have anything to back that up?
Tooting my own horn here, but I wrote a filesystem called runfs that specifically addresses this. A service writes a PID file to runfs, and runfs automatically removes it once the process dies.
Code: https://github.com/jcnelson/runfs
This "all init does/needs to do" notion is oft-repeated, but it is oft-repeated rubbish. It is a notion commonly held by people who have never actually written a process #1 program that functions in production systems; because experience (as I can attest) teaches otherwise. There are, in fact, several things that various operating system kernels and other programs demand of process #1 that one simply cannot escape. People think that fork()ing things and acting as the parent of orphaned processes is the prime function. Ironically, dealing with orphaned processes is (with recent Linux kernels) a part the system that one can largely factor out of process #1 into other processes, whilst the things that people usually forget in their off-the-top-of-the-head designs for init (such as handling SIGINT, SIGPWR, SIGWINCH, and so forth sent from the kernel and enacting the various system state change requests) are the parts that one cannot. To see what one actually has no choice but to do in process #1 programs, look at the overlaps in the operation of Gerrit Pape's runit, Felix von Leitner's minit, and the system-manager program from the nosh package. There are several things for a /system manager/ to do when there's a separate /service manager/.
daemontools gained svscan in 1998, and svscanboot in 2001. Years earlier, in AIX version 3.2 released in January 1992, IBM introduced its (then) new System Resource Controller. This was, and still is, a service manager that is spawned by the initial user process, that has the task of supervising daemon processes, bringing them up and down in response to administrative command, logging control activity, and notifying about failures to run.
You'll find things like the srcmstr process; the startsrc, stopsrc, and lssrc commands for starting, stopping, and listing services; the whole notion of signals to control daemons (SIGTERM, SIGNORM, SIGFORCE, et al.); entry 5.2.1 in the Usenet comp.mail.sendmail FAQ which talks about how fork-and-exit-the-main-process by Sendmail breaks service management; all of the "How do I get my too-clever-for-its-own-good daemon to work with this new service management system?" discussions in the mid-1990s; and IBM's documented advice from at least as early as 1995 on how to write daemons properly, achingly familiar.
One can look to mainframe and minicomputer operating systems before that, and also to Windows NT 3.1 which had the initial user process spawning the Service Control Manager, released in 1993. But in the narrow Unix+Linux worldview, credit should go to AIX for being the first operating system released where people had come to that conclusion, approaching a quarter of a century ago.
That's part of my issue with systemd: replacing safe (-r) shell scripts with C. Sure, it's fast, but I trust C about as far as I can throw it (and yes, I realise that the Linux kernel is written in C: it constantly has local root exploits, which proves my point). I really wish that projects like systemd were written in something of like Ocaml, Go, Rust, Common Lisp—anything fast enough but safer.
Also, it probably doesn't help Lennart that he looks like the Rolfe Gruber in the Sound of Music…
Because shell scripts are always safe, right? [1][2][3] We used to get root on Android devices by exploiting poorly-written shell scripts.
> Also, it probably doesn't help Lennart that he looks like the Rolfe Gruber in the Sound of Music…
Personal attacks won't help your case.
[1] https://github.com/ValveSoftware/steam-for-linux/issues/3671
[2] https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi...
[3] http://www.exploit-db.com/papers/13199/
[1] https://www.securecoding.cert.org/confluence/display/seccode...
At its core, they are the very same commands you would give manually. but wrapped in basic programming logic (if clauses etc).
thus you can walk the shell script and get a good feel for why the script is doing what it is doing.
If you want to do that with a binary, you pretty much have to either hit it with a debugger (i think a previous thread here on Hacker News actually recommended using strace to inspect systemd boot behavior) or have the source code sit beside it for reference.
Hence 'safe (-r),' not 'safe.'
> Personal attacks won't help your case.
It was not an attack, just a statement of fact. He looks just like a movie villain; it's not his fault, and no-one should hold it against him, but I wonder if it hurts him anyway.
Poettering was sitting as co-maintainer of consolekit for some time. Then as logind got underway, the maintenance was to pass to Ubuntu under a new project name. But i can't find any trace of this actually happening.
After that the state of consolekit as a project seems to have been in limbo, until recently when a XFCE dev forked the existing code to github as consolekit2.
Apparently said dev would have loved to continue consolekit in place rather than fork, but the final email list traffic between Poettering and Ubuntu gave him the impression that Poettering wanted to see consolekit dead.
So it may well be that people didn't maintain the consolekit code in Gnome because they had the impression that consolekit was dead and buried.
but then Gnome and systemd devs seems to share a "one true way" attitude.
However, they did this in complete silence, so very few people actually know about this.
[1] https://hg.java.net/hg/solaris-desktop~spec-files/file/79574... (15 patches)
...and this is the problem. Pushing this kind of complexity into PID 1 (or however systemd achieves this) is exactly the wrong solution. The init system should not be clever, or contain an "engine that can figure out what the computer is supposed to be doing" as he says later. This really smells of second system syndrome.
In reality even with a simple init that pushes complexity to the admins, end users rarely incur the cost. The distribution authors set up init as needed and end users don't need to worry unless they makes changes. Then you get the best of both worlds: easy maintainability and a simple, reliable init system.