I think most systemd criticism misses the point and revolves too much around usage semantics.
Systemd allows actual resource management on a level that simply didn't exist with traditional init [1]. Yes we can argue about implementation, but the necessity of better resource management in more dense and shared environments can't just be ignored, especially when building a shared VM hosting platform like the author is talking about.
Some examples of poor quality of code? This has not been my experience. I've gone through the code base a couple of times for my own education and to do some small patches, and I found it clean and easy to read.
The problem isn't so much "typical programming errors", but the appoach to solving problems. So, it's not that there are tons of buffer overflows in the code (well, not that I know of, though I didn't explicitly look for them either), but stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades, or the bug that caused .* to match .. and thus systems to be destroyed by the temp cleanup code, which is equally a potential pitfall that is well-documented and explicitly even standardized against in POSIX, but they still failed to implement this in a way that is both evidently sane and also otherwise generally accepted to be the correct way to do things ... and then they even pretended that it's somehow not actually a bug.
Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude.
They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).
> stuff such as the vulnerabilities in the DNS resolver because they reimplemented things from scratch without the necessary domain knowledge on how to do that securely that has been collected and implemented by the dozens of existing implementations over the decades
It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.
I've had way more headache with badly written and plain faulty init scripts than with systemd, including systemd-networkd, systemd-timesyncd, systemd-resolved and systemd-nspawn, which I used to replace a lot of LXC, LXD and openvz setups.
Well it just works and is extremely simple to use.
I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it.
For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts:
[Network]
Bridge=br-int
Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file:
[Exec]
PrivateUsers=no
Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.
One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...)
But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)...
Haven't dealt with live migration, but mounting filesystems should be easy using systemd's unit dependencies.
Just drop a .mount file in /etc/systemd/system and set RequiredBy=systemd-nspawn@something.service and StopWhenUnneeded=true and the filesystem should be mounted before the machine starts and unmounted when the machine is shut down. See the manpages systemd.unit and systemd.mount for details.
The problem is that badly written init scripts are reasonably easy to debug and make work, even if it's often ugly, while systemd is not a good enough replacement to just make all those problems a thing of the past, so you still have to debug stuff occasionally and try to make it work somehow, which just is so much harder due to the way systemd works.
It's a bit like ISA vs. ISA PnP vs. PCI. Jumpering ISA cards to make sure resources didn't conflict was a bit of a chore and sometimes difficult to get right, but essentially there always was a way. ISA PnP tried to automate this, which was great if it worked, but more often than not just failed, and then you had no jumpers to fall back on to just fix things up manually (though sometimes you had special config utilities that with some luck you could use to fix things up with some cards ... maybe). PCI, though, was an actual reliable abstraction that actually worked essentially all the time, so there actually was no use for jumpers, so it is fine that PCI cards don't have IRQ/IO jumpers.
Systemd seems to me like the ISA PnP of init systems.
Wow, you unlocked some painful memories of buying an expansion card and spending a frustratingly long time wondering whether I’d ever get it working. There are many things I get nostalgic for about old computers, but this isn’t one of them. Thank you for the reminder—this is an awesome metaphor.
> the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis.
They lament
> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration
Using systemd was costing them too much. Moving back to the previous init system looks like a rational choice for them.
My experience is different but I only manage a handful of virtual servers and not on a daily basis, plus my laptop. Systemd configuration files are not difficult to write and they restart the daemons if they crash. For complex stuff I make systemd run a bash script that eventually executes a daemon, kind of cheating. My laptop still runs well. Booting time is definitely not an issue, it went from fast once per month or so (kernel upgrades), to fast still once per month so. I didn't notice any difference after the change of the init system. A good thing but maybe it means that the return on investment was dubious for this use case.
Binary log files are objectively worse than text ones. cat, less and tail were good enough and shorter to type than journalctl. Ok, I could alias it to a four characters word but it was still a lot of work that could have been invested on some other goal. Instead I've got servers with possibly compact binary logs made of very few lines and large text logs from web applications.
I'm also puzzled by the philosophy of bundling more things together and tighten dependencies. It's somewhat disconcerting and I'd like a system where we could swap components out more freely, but this is an opinion and not facts.
Actually, I think that you are missing the point. It’s true that systemd implements some good ideas that did not exist in other init systems at the time, but most of those feature are pretty niche, and more importantly, you can not evaluate software purely on how good it’s ideas are. You have to consider how good it actually is. Systemd has real flaws that cause real problems for real people. The sooner everyone understands this the better.
I spprt Devuan because I support choice. And I hope one day - be it 5 or 10 years - we arrive to the point where Devuan is no longer needed - you will be able to replace one init system with another in Debian. Currently it's not possible and the system I've been using since 1998 became something else that I can't control in a way that was possible until now.
When I ask my fellow colleagues (mostly sysadmins), some of them like Systemd, some don't, but we all agree that in this particular case the freedom of choice is much needed.
* https://packages.qa.debian.org/s/systemd.html
* https://packages.qa.debian.org/u/upstart.html (it's gone from stretch that's true but still supported in jessie)
* https://packages.qa.debian.org/o/openrc.html
* https://packages.qa.debian.org/s/sysvinit.html
I tried to run sysvinit on Debian for a long time. What happened was that often when I tried to install something, I got systemd back, and sysvinit thrown out, via some funny dependencies. Maybe I was doing something wrong. Maybe you just didn't want to install the packages I wanted to install. I don't know.
In any case, the pain was real. So finally I switched to Devuan, even though I wanted to stay with Debian, and now those problems and the time I had to spend fighting them are gone.
I run Debian with sysv too and had the same issue early on. Fortunately, that one is easily fixed - if you drop the following into a file in /etc/apt/preferences.d, it will stop that from happening:
The i386 parts are only needed if you have it setup with multi-architecture.
Unfortunately, I have also had to recompile some packages to remove systemd dependencies as well. Fortunately that hasn't been too much effort so far but eventually I may have to switch to Devuan. The most ridiculous one was policykit and neatly illustrates the creeping "infection" of systemd into Debian - policykit-1 hard-depends on libpam-systemd which hard-depends on systemd. Why neither was marked as "Recommends" instead of "Depends" I can't understand (policykit-1 -> libpam-systemd or libpam-systemd -> systemd) - either would have been fine and allowed both options to work perfectly well as far as I can tell.
> Unfortunately, I have also had to recompile some packages to remove systemd dependencies as well.
And I think that's exactly what Devuan does - they recompiled/rebuild the packages with funny dependencies that caused systemd to creep back in. The rest are just the normal Debian packages. They even forward you to the Debian download servers for those.
The advantage of using Devuan is that they have spent tome to do this work, once, so now not everyone who wants to use Debian without systemd has to spent all this work again.
Yes, the Debian devs could have easily done this themselves. For some reason they decided not to.
With the correct pin (as described in my linked comment), you can prevent systemd-as-PID1 from coming back.
If you install certain desktop packages, then a package named "systemd" might come back, but systemd won't actually run as PID 1. Since the pain from systemd comes from it running as PID 1, I don't see a problem with this.
If you want a system that's 100% free of all systemd code, even if it's not doing anything, then you need Devuan, but personally I don't see that as a compelling reason to fork Debian.
> With the correct pin (as described in my linked comment), you can prevent systemd-as-PID1 from coming back.
Frankly, it's an ugly hack, not a proper solution. I hope to run my servers for years to come and the upgrade process should be as smooth as it can be.
So, the whole thing with systemd is not that "it's bad" but that "I don't like it and I am stuck with it". Moreover, it leaves those of us who don't like it disenfranchised by distros that insist on using only systemd
Opensource is about alternatives. Forks are a good thing,I wish devuan great success so people like myself won't have to whine about systemd all the time.
If you like systemd and it's philosophy,I wish you the best of luck with it. Just keep in mind that others don't have to like it and different people or organizations have different needs.
My thoughts exactly. I personally have nothing against systemd and haven't been directly affected since I rarely need to touch init scripts, but I still appreciate Devuan and loosely follow its progress for this reason. (Kind of like ReactOS — 99% I'll never want or need it, but the fact that it exists is still really cool.)
I was a systemd proponent until I had a hosed box. At that point I learned that alternatives are important. It made my life very difficult due to how opaque it is.
Whike problems are surmountable, rehearsing for failure is advised.
Unfortunately I've run into bugs where services provided by the kernel (encryption key management) work on distributions without systemd but break on distributions with systemd.
> Forks are a good thing,I wish devuan great success so people like myself won't have to whine about systemd all the time.
The real issue with systemd is that there simply aren't enough volunteers (both individuals and companies) out there willing to contribute. In short, since gnome is pretty much developed by RedHat and systemd is developed by RedHat, gnome is tied into systemd.
If there was enough of a community around gnome that RedHat wouldn't be able to pull it wherever they want, gnome wouldn't have been able to be that coupled with systemd. The issue is that debian simply doesn't have the manpower needed to de-systemd gnome.
That's not quite correct. GNOME only really needs systemd's logind component. For Guix that component has been carved out of systemd as elogind. GNOME is virtually unchanged.
You can just uninstall it. It looks like apt remove systemd would work on my system, at the cost of GNOME and NetworkManager. APT automatically fills the gap with sysvinit and consolekit. OpenRC is also available.
Debian even has ports to kFreeBSD and HURD that don't support systemd at all.
There were some rough edges last time I ran Debian without systemd, and it might have gotten worse since then, but it surprises me how Debian ended up at the center of the debate when it doesn't seem that bad as systemd-defaulting distros go.
> We tried to build Data Center Light on Debian and Ubuntu, but servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration made it too expensive to run Debian or Ubuntu.
Can you expand on that? I work at a cloud provider, and I never heard of such issues before. Our default images are Ubuntu 16.04, RHEL 7 and SLES 12, so systemd all around.
This could be an honest mistake. On my system, systemd-fsck-root.service and systemd-fsck@.service have `TimeoutSec = 0`, which I guess to mean "no timeout", but the systemd-fsck executable could have its own timeout. In any case, filing a bugreport sounds much easier than switching distributions.
Regarding not rebooting: sometimes systemd hangs and does not reboot in our setups, but we were never able to sufficiently debug this, as all terminals are already gone and it seems that systemds is waiting in a loop for dependencies that don't solve.
Have you tried passing 'debug' instead of 'quiet splash' to the kernel cmdline and doing halt instead of poweroff/reboot? That way you should see what's going on during shutdown.
In terms of booting: we are aware that systemd just kills fsck after a timeout. And that you can adjust the timeout. It is just something additional that we need to take care of, when running a system with systemd that is not the expected behaviour of a Linux system (like: why would you ever want to abort fsck?)
If you have a large (multi TB) filesystem, sometimes those can take a long time to fsck.
And if the fsck is just a scheduled one (after x reboots) instead of being triggered due to filesystem weirdness, then getting the server back up and serving can be the higher priority.
That's just an example because you asked though. Wouldn't want that same fsck to be skipped if it was actually started due to filesystem weirdness being detected. :)
And re halt/poweroff/reboot: We tried to figure it out for some systems, where a user session was blocking reboot, but eventually stopped investing resources into it. Similar to the booting problem, we expect a computer to "just reboot" and the default behaviour should be that kill -9 is issued after some timeout.
And regarding networking: one very simple, obvious non-feature is that search paths from radvd or dhcpd are not correctly passed on to the OS processes, which breaks resolving server names.
Soooooo, I'm part of a CubeSat team and we use systemd. Extensively. We us their dbus library for our daemons to communicate, some of the ops is basically done in service files.
I have to get our systems people to write a paper on it. My experiences with systemd are pretty much perpendicular to what some people claim.
It's not about claiming here, it's about experiences.
I guess we should open up a youtube channel for filming our most catastrophic experiences with systemd, because otherwise somebody will claim again that its brokeness is just a claim ...
Let me tell you about my most catastrophic experiences with sysv... I've had quite a few.
Anecdotal evidence is useless.
What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd.
I run large scale VM infrastructure on a systemd distro (RHEL, in particular) and I've yet to encounter a single issue that was caused by systemd or, for that matter, any issue with systemd that wasn't easily resolved.
> What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd.
That doesn't mean that there aren't any. Also, FUD (as I understand it) is people spreading unfounded rumours, not people documenting their own experiences and problems with a piece of software. The intent is incomparable.
If you are actually interested in tangible structural flaws in systemd, look no further:
This formulation may be misunderstood. The bug did not grant users root rights, it ignored the user= setting on a service if the service was intended to run as a user stating with a numeral. So you already needed to be root to create a service. The bug allows a service to run with more permissions than intended though.
I don't like systemd, but it works pretty well nowadays in my experience and we use it for servers at work.
Truth be told, I'm fine with monoliths. They just have to be good. The Linux kernel is a monolith.
The real problem to me is how systemd makes the rest of the system more complex and integrated to systemd. Dbus is also not a great IPC layer, and yet it's the current state of the art.
We need better APIs and foundations. Systemd doesn't provide those; it just lets everyone depend on systemd. Good for Redhat, bad for innovation.
The fact that they weren't able to set up computers to boot using systemd is enough to warrant that I'd never want to buy or rent servers from this company, sorry to say.
I have. At my previous company, I built single system storage units for our large scale distributed storage systems that had between 100-500TB of physical storage in a single box. Linked together via infiniband, 40/100 GbE, and others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS).
Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well.
The greatest pain was the networking.
Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking.
Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases.
My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works.
I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over.
Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.
Last time I checked, systemd - the init system - wasn't involved with networking at all.
There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts.
You said there were "myriad of shortcomings" and "egregious settings", some examples would help.
Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases.
On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image.
This is just a small sampling of the open source build system, that I grabbed from some of the configs we used.
Yes, systemd is broken, but not irretrievably. It is fixable.
# fix some systemd timeout brokenness
sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf
sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf
sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf
# fix systemd journaling. Yeah, really
sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf
sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf
sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf
sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf
# fix the INSANE logind.conf per user directory size ... hard code it to 256M
sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf
#
# fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ...
sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf
# mask off systemd-udev-settle ... yes it is broken
> Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to bring up interfaces. Controlling it is very hard for any less "traditional" use cases.
Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen.
I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd.
> fix some systemd timeout brokenness
Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s).
ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea.
> fix systemd journaling
Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative.
> RuntimeDirectorySize
What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there.
> KillUserProcesses
Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server)
> mask off systemd-udev-settle ... yes it is broken
Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events.
Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.
> Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen.
The last job developed/supported Debian based high performance storage systems in production, starting with Debian 7, then Debian 8, and Debian 9.
> Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s).
This comes from long debugging/tracing sessions where we were trying to understand why we wound up with effectively repeating timeouts.
[...]
On systemd-udev-settle, allowing it to run was actually the cause of a race condition on start. The only way to fix this that I found was to disable this. Once we did this, no more race condition. The system booted normally.
It sounds like your experience is significantly different than mine on this. We built very large high performance computing and storage systems for many users, where the things that people used simply had to work, without surprise. Every line I showed came from a set of long and often painful debugging sessions where we had to figure out what new change had broken a key bit of software.
The RuntimeDirectorySize issue actually tickled another bug, and caused some incredible paging problems when used with a specific software package. Enough end users on a system (we had many) could launch an effective DoS against the system by using these tmps.
Generally, my observation, and I've seen others make a similar set elsewhere, is that the default and general distribution settings for systemd parameters appear to be generally set for desktop users (single or very few users per machine). Not for large shared resources.
> But first, let me put some warning out here: Dear Devuan friends, while I honor your work, I also have to be very honest with you: in theory, you should not have done this. Looking at creating Devuan, which means splitting of Debian, economically, you caused approximately infinite cost.
I thought the same at first but after trying to maintain a Debian installation without systemd, I was convinced otherwise. The "infinite cost" was not caused by Devuan developers but by Debian developers.
I felt the Debian transition to systemd was handled very badly and was rushed in order to get Jessie out of the door. Debian still provides the sysvinit-core package as if it were a supported init system however if you use it you'll find the operating system has many quirks and many things are broken. Before Jessie was released, I commented on one bug report related to a hard dependency on systemd for NetworkManager. Instead of fixing the regression, the package maintainer gave the excuse that they don't have the resources and would need more maintainers. I sympathise with the package maintainer but this is the sort of thing that led to this mess. I remember there being a lot of hostility in that period from Debian developers towards those who did not wish to use systemd.
Based on my experience, I think the right decision was made to "fork" Debian to create Devuan. It's a shame that they had to do this. I still hold hope that maybe one day all of the work put into Devuan can be reintegrated into Debian and it can return to be the "Universal Operating System" that they still claim to be.
I'm pretty agnostic towards systemd, but I don't understand this complaint that Debian now doesn't support other init systems. Did they support other systems besides sysv before?
They also supported upstart but that was dropped in Stretch. We are left with systemd and System V init to provide /sbin/init. Debian still provides System V init which itself works fine but other packages, many of which previously worked, have come to depend systemd only. This type of regression would not be acceptable in other projects but Debian has done nothing about it. I think they should stop pretending that Debian is still usable System V init and just drop the package altogether.
That really depends on your definition of "supported". It does not form part of the base distribution, nor was there ever any paid support for it.
A Debian developer built a deb package for it and added it to the "main" repository. Everything under "main" is normally considered to be part of the distribution. As you can see from the link I provided, the package was maintained for several years and received several updates until it was dropped for Stretch. That is exactly what I would expect from a supported package in Debian.
Yes. In Debian 7 one could switch among van Smoorenburg init+rc, systemd, upstart, and Felix von Leitner's minit.
runit-init had been removed in 2010. Felix von Leitner's minit was later removed by Debian member Iain R. Learmonth in November 2015. upstart was removed as a consequence of a bug report (#789524) filed by Kyle Amon in June 2015.
One of the contentious events during the Debian Hoo-Hah was one of the package maintainers leaping to remove support for an init system from one package almost as soon as the Technical Committee had made its decision favouring systemd over upstart, OpenRC, and van Smoorenburg rc. People opined then that Debian package maintainers should not blithely remove support for init systems like that.
More recently, however, this has begun to happen. In August 2017, a group of Debian people got together to organize the removal of all upstart support from all packages; and that is progressing.
Right, but sysv is still around, so if "support" means having the package, then Debian still supports other init systems. The complaint here is that stuff doesn't work well with sysv, so my question is: if back in Debian 7 something didn't work well with Upstart, was that considered a blocking bug? If not, I fail to see what changed. As you pointed out, they already removed obsolete init systems before systemd came around, so removing Upstart (which has been abandoned by upstream) is not a change in regular procedure.
That concept of "obsolete as a Debian package" does not really exist, so you are asserting something meaningless. And you brought up the timing of systemd yourself, only to then find that your point was wrong.
You have a lot to learn about the rather sad history of runit and daemontools in Debian, as well as about the ways that Debian people decide what is packaged. This is only some of it.
I understand that the ways that Debian developers make their decisions is not to your liking, but I disagree with your presentation of the facts, nor do I care for your passive-aggressive accusations.
Runit-init-the software was not removed from Debian. Only runit-init-the-package was removed and that was long before systemd-the-package (which, yes, was what I was referring to) appeared, and it was by a ROM from the author himself, after a bug report classified as RC (justifiably, in my opinion).
So I fail to see any changes in behavior by the Debian team, since I don't believe in this conspiracy theory that Debian Developers were finding bugs in other init systems just to push for systemd, despite not even caring enough to package it.
Could they be more open to other init systems? Probably. But that was never my question.
The presentation of the facts is simple hyperlinking to the actual discussions, to disagree with which is bizarre, to say the least.
After propounding a non-existent concept of "obsolete as a Debian package" you are proceeding to propound an equally nonsensical concept of removing runit-run whilst not removing runit-run. https://tracker.debian.org/pkg/runit-run is really quite clear and unambiguous.
And that conspiracy theory is a straw man of your own making, too. As is indeed the commentary on "the ways that Debian developers make their decisions" which are your opinions not anyone else's. You have a lot to learn and you're getting a lot of things quite wrong.
Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water.
Mind you, there are plenty of other complaints (stability, back-compatibility, etc.) that are useful/valid regarding this software, but the above litmus serves to eliminate rather a lot of FUD and hand-wringing in my experience.
It filters out: people who just want to complain about centralization; people who are annoyed about change and trying to rationalize it without trying or learning the new tool; and people who irrationally feel like they are no longer in control.
Paradoxically, this improves the signal of discussions about systemd's flaws, and there are plenty. With the obviously irrational/haven't-used-it/axe-to-grind stuff filtered out, it's much easier to learn about real features, problems, fixes, benefits, and drawbacks of any project, systemd included.
>Every time I read something complaining about systemd from a centralization/reducing-choices POV, I mentally replace the word "systemd" with "glibc" and check to see if the argument still holds water.
Actually, I replace systemd with *BSD. You know, the OS which has blessed core packages which are not replaceable?
The difference is that in the BSDs (especially OpenBSD) a lot of care is taken to be configurable. Consistent. Clear and transparent. OpenBSD has the most extensive suite of tools built specifically for systems administrators that exists in a base system.
SystemD has opaque behaviour and binary interfaces. It's the antithesis of the BSD model. Despite BSDs being quite coupled groupings of software.
Anecdotally we're attempting to move to BSDs instead of upgrading to RHEL7 due to SystemD.
One gotcha (for me) with BSD is that I'm developing on Golang. It doesn't really debug well on anything other than Linux... so had to move to that for development.
... and for deployment, the Golang debugger doesn't even run on BSD at this point. So, going to have to setup new Linux (likely CentOS 6 - no systemd - or 7) boxes soon now that we're getting closer to first initial real world deployment for the project I'm on.
I'm not comparing code quality. I never looked at either and therefore can't comment on that.
It's just the "UNIX is supposed to be decoupled" line that gets repeated every time systemd comes up that BSD (which, ironically, those people want to switch to) disproves.
As the author of a toolset that can replace parts of Base such as init, van Mewburn rc, and service with its own system, service, terminal, and logging management, I point out that "not replaceable" is not exactly true.
glibc's actually pretty bad, and many people prefer alternatives such as musl. Ideally, it would be easier to use an alternative libc. Wherever possible we should encourage modularization and standards-compliance, not lock-in, so that the best implementation can rise to the top.
So yes, the same centralization/reducing-choices arguments apply to glibc.
Those are good points. Systemd is somewhat modular/swappable for other init (et cetera) systems, but the practice of actually swapping it out as a dependency of a given package is roughly as tricky as switching out glibc for something else. You have to crack open the package and make it not depend on/automatically install the init system (or libc) you don't want, tell it to install one that you do want, and then thoroughly test to make sure that the old version wasn't depending on a piece of behavior specific to the init (or libc) that you switched out. Or you could build and package the software yourself, with similar hassles to address.
The problem with standards compliance is that, once something becomes ubiquitous enough, its compliance begins to "drift", and replacement parts get harder to find. Is it tricky to swap out the init component of systemd with something else? Not terribly. Is it tricky to swap out its dbus messaging setup with something equivalent that can still be used by unaltered software that was previously talking via the systemd setup? Yes, a bit. And so on. For equivalent hassles regarding libcs, check out some of the FOSDEM talks from the musl folks, or the alpine folks, where they talked about the difficulties in porting software that had come to depend on the quirks of glibc.
Or just, you know, google "quirks mode" and get simultaneously nostalgic and sad :)
Oddly enough, systemd depends on glibc, which is why Void Linux is the only Linux distro that I know of to switch away from systemd (to runit): they wanted to provide a musl option.
I am running Void with musl on a Raspberry Pi 2. I love it.
I thoroughly support Devuan. I switched my home servers to it around the systemd introduction and kept their complete stability, and now I've been able to influence my company to run it instead of pure Debian. You have to make a few allowances for packages that assume they're running on systemd Debian, but overall the system works exactly as intended. And as commented in the article, it's stable and easy to understand.
At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux.
In my opinion, the Debian developers were wrong to fully embrace systemd; it goes quite against the Debian principle of being able to adapt your machine to do anything. Yes, Red Hat held a lot of sway in convincing the community to adopt it, but Debian is a big distro as well - just look at the install base of Ubuntu. They could have held out and pushed back, but instead caved and now we're in the systemd mess. It intrudes so massively into userspace that it's impossible to get away from, and we wind up with the Windows model as this article points out - a massively complex spaghetti model of processes, all interlinked, all hinging on everything playing nicely. Just like Windows, it's a neat house of cards that could topple if any one of those crashed. This isn't Linux.
I'm not defending sysvinit, there's no denying it's showing its age and harks back to a much simpler time, but systemd is not the answer.
> At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux.
That's far from the only advantage systemd has.
Even then, boot times are extremely important for modern VM and container infrastructures.
I still think systemd was a NSA/Redhat backdoor in the midnight for Linux. I've forced myself to deal with it because its everywhere, but my Spidey-senses tingle everytime I find some new way its trying to takeover command of a core function.
What irks me more about this blog post than the systemd part
- which has been discussed in-depth countless times in the past, with valid arguments for and against - is the decision making and risk assessment process.
I've run large-scale infrastructure on both Debian and RHEL, with and without systemd. I've been involved in many distro decisions in the past. The init system was never a major consideration, and I've always been able to work around whatever issues I've had with both sysv and systemd.
Here's some considerations which were important for my team the last time we had to make a choice:
- Vendor stability and long term support
If you're running a business, you'll want long term stability guarantees - both from a technical, and a business point of view. Running a small community distro with few - if any - commercial users is a huge liability. Yes, you always have the option to fork it or take it over, but unless you're in the business of building a Linux distro, it's almost certain to be a bad business decision since your competitors are spending their time on their products instead.
Anyone who doesn't appreciate long term vendor stability hasn't yet been burned by the lack of it. Your Gentoo wizard left the company? Too bad, good luck maintaining your infra now (I've actually seen that exact scenario play out not once, but twice!). A few core maintainers left for another project and you're left building your own packages? D'oh.
Having a mature and well-funded organization or a large community of commercial users (in the case of Debian) supporting your distro is extremely valuable, even if you aren't the ones paying for it.
- Ecosystem
The ecosystem is also really important. It's often the main motivation for using a particular distro. Development tools, third party package repos, 3rd party enterprise software, troubleshooting resources, documentation and much more depend on a healthy ecosystem.
With Devuan, this isn't as much of an issue, but it's still sufficiently different from Debian in sometimes subtle ways that it will nullify some of the advantages of being in a common ecosystem.
This also includes hiring - you'll have a harder time staffing your company if you're using exotic stuff and you'll unnecessarily spend your time training them. Ever wondered why companies like Google publish so many papers, talk about their infrastructure and even publish books about it[1]?
It's because it means they can - by advancing the state of the industry - hire people who are already familiar with their architecture and concepts. This is a real problem for them - Google, for instance, is in some cases so far ahead of others that they have to spend a significant amount of resources to bring their new hires up to speed. I don't know about Amazon and Facebook, but I'm sure they have similar problems.
Security follows the supply chain. Your security is only as strong as your distro's - no matter how good your security controls and processes are, if your distribution vendor is compromised, you won't stand a chance unless you have a world-class security team.
Likewise, your customers fully rely on you - if you're compromised, they will be, too.
Security is much more than signing your packages and publishing security advisories. In fact, those are merely the results of a properly implemented security management framework - risk assessment and mitigation needs to be an integral part of your vendor's (and your) processes. Where do they keep their PGP keys? Who has access to it? Is is stored on a HSM or just sitting on a server somewhere? Is there a process for revoking it? Is there a change review process? Does the build system verify source code integrity? Is there two f...
In your "security" section you completely skipped over the distro's process for deciding when to patch and when to push/outsource to upstream. Especially given that you're discussing Debian that's a glaring omission.
More to the point-- what's Debian policy on applying local patches in their packages?
For example-- if you do it today on a core app/lib without consulting upstream is that enough to get your credentials revoked?
I ask because Debian famously patched a library to quiet a valgrind memory error which resulted in their userbase generating predictable key material for a few years[1].
I trust them not to do that again (at least with libs that are obviously security related). But is there explicit policy in this regard?
From a business perspective, the most important thing -- indeed, the only thing that truly matters -- isn't the software itself. It's support. When it comes to open source, you either use a supported configuration or you assume ALL the responsibility for supporting your systems yourself. Who would you rather handle the support on the off chance your machine goes tits up? You? Or Red Hat?
No wait, scratch that. Who would YOUR BOSS rather handle the support?
The supported init system for most flavors of Linux (that aren't in LTS) is systemd. Systemd makes the most sense from a business perspective, and "but muh Unix philosophy" doesn't hold when there's money on the line. May as well get used to it.
123 comments
[ 9.1 ms ] story [ 206 ms ] threadSystemd allows actual resource management on a level that simply didn't exist with traditional init [1]. Yes we can argue about implementation, but the necessity of better resource management in more dense and shared environments can't just be ignored, especially when building a shared VM hosting platform like the author is talking about.
[1] http://0pointer.de/blog/projects/resources.html
Those are two somewhat recent examples I can think of, but there have been many more showing the same kind of attitude.
They are ignoring hard-earned knowledge on how to do things securely, safely and reliably, they ignore the intentions of abstractions, and they pretend that obvious bugs aren't bugs because there is some weird way to reinterpret the bug into being correct behaviour (that noone expects and that causes harm).
By domain knowledge how to do this securely in implementations tested oer time, do you mean something like https://access.redhat.com/articles/2161461?
It doesn't look like systemd's resolver fares worse in comparison... In addition it can be sandboxed (and is), an advantage over having a resolver part of libc.
Would you mind explaining how exactly sandboxing prevents cache poisoning?
I have used LXD (with ZFS storage) in production but have not played with systemd-nspawn...
Thanks!
I usually debootstrap into /var/lib/machines/something and do "machinectl enable something; machinectl start something", that's it. Then I attach to the machine using "machienctl shell something" and configure networking (host0 interface) inside the domain, that's it.
For drop in configuration systemd-nspawn parses a config file /etc/systemd/nspawn/something.nspawn which usually just contains network configuration on my hosts:
[Network] Bridge=br-int
Systemd-nspawn enables and user namespacing by default and chowns the machines's root filesystem on first start. If that's not desired (Things like Samba fileservers don't work well with user namespacing) just disable it in the .nspawn file:
[Exec] PrivateUsers=no
Everything you need to know is in the manpages systemd-nspawn and systemd.nspawn. I usually install systemd from stretch-backports because running a fairly recent systemd version helps as it still gets new features, but I never had problems with stability.
One thing I somewhat miss from what you are explaining is all the aditional things that LXD gets you (snapshots using ZFS, image publishing/sharing, migrating containers between LXD hosts...)
But maybe some of those things are still doable (e.g. mounting a ZFS dataset as storage for /var/lib/machines/containerX)...
Thanks for your answer!
Just drop a .mount file in /etc/systemd/system and set RequiredBy=systemd-nspawn@something.service and StopWhenUnneeded=true and the filesystem should be mounted before the machine starts and unmounted when the machine is shut down. See the manpages systemd.unit and systemd.mount for details.
It's a bit like ISA vs. ISA PnP vs. PCI. Jumpering ISA cards to make sure resources didn't conflict was a bit of a chore and sometimes difficult to get right, but essentially there always was a way. ISA PnP tried to automate this, which was great if it worked, but more often than not just failed, and then you had no jumpers to fall back on to just fix things up manually (though sometimes you had special config utilities that with some luck you could use to fix things up with some cards ... maybe). PCI, though, was an actual reliable abstraction that actually worked essentially all the time, so there actually was no use for jumpers, so it is fine that PCI cards don't have IRQ/IO jumpers.
Systemd seems to me like the ISA PnP of init systems.
> the reason to use Devuan is hard calculated costs. We are a small team at ungleich and we simply don't have the time to fix problems caused by systemd on a daily basis.
They lament
> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration
Using systemd was costing them too much. Moving back to the previous init system looks like a rational choice for them.
My experience is different but I only manage a handful of virtual servers and not on a daily basis, plus my laptop. Systemd configuration files are not difficult to write and they restart the daemons if they crash. For complex stuff I make systemd run a bash script that eventually executes a daemon, kind of cheating. My laptop still runs well. Booting time is definitely not an issue, it went from fast once per month or so (kernel upgrades), to fast still once per month so. I didn't notice any difference after the change of the init system. A good thing but maybe it means that the return on investment was dubious for this use case.
Binary log files are objectively worse than text ones. cat, less and tail were good enough and shorter to type than journalctl. Ok, I could alias it to a four characters word but it was still a lot of work that could have been invested on some other goal. Instead I've got servers with possibly compact binary logs made of very few lines and large text logs from web applications.
I'm also puzzled by the philosophy of bundling more things together and tighten dependencies. It's somewhat disconcerting and I'd like a system where we could swap components out more freely, but this is an opinion and not facts.
>> servers that don't boot, that don't reboot or systemd-resolved that constantly interferes with our core network configuration
Which is interesting given Debian doesn't use systemd-resolved (unless manually configured). So they made up problems that don't exist by default?
This I have a problem with, if it crashes, I'd prefer it stay crashed, instead of crashing over and over.
At least when it crashes, a human can come and see why it's crashing, rather than doing the caveman thing and just restarting it over and over.
https://www.freedesktop.org/software/systemd/man/systemd.ser...
When I ask my fellow colleagues (mostly sysadmins), some of them like Systemd, some don't, but we all agree that in this particular case the freedom of choice is much needed.
Sincerely asking, I never tried running a non-default init system.
Right, but did they support using any other init system (besides sysv) before?
All my Debian systems run sysvinit, and they work great - certainly better than the one system I co-administer which runs systemd.
I frankly do not understand why Devuan exists.
In any case, the pain was real. So finally I switched to Devuan, even though I wanted to stay with Debian, and now those problems and the time I had to spend fighting them are gone.
That's why Devuan exists. YMMV, of course.
Unfortunately, I have also had to recompile some packages to remove systemd dependencies as well. Fortunately that hasn't been too much effort so far but eventually I may have to switch to Devuan. The most ridiculous one was policykit and neatly illustrates the creeping "infection" of systemd into Debian - policykit-1 hard-depends on libpam-systemd which hard-depends on systemd. Why neither was marked as "Recommends" instead of "Depends" I can't understand (policykit-1 -> libpam-systemd or libpam-systemd -> systemd) - either would have been fine and allowed both options to work perfectly well as far as I can tell.
Sounds like a bug report is needed, so that gets fixed. :)
"Fill a bug report" automatic reply gets old quickly and it's especially absurd when it's said by a person totally uninformed on the issue.
And I think that's exactly what Devuan does - they recompiled/rebuild the packages with funny dependencies that caused systemd to creep back in. The rest are just the normal Debian packages. They even forward you to the Debian download servers for those.
The advantage of using Devuan is that they have spent tome to do this work, once, so now not everyone who wants to use Debian without systemd has to spent all this work again.
Yes, the Debian devs could have easily done this themselves. For some reason they decided not to.
If you install certain desktop packages, then a package named "systemd" might come back, but systemd won't actually run as PID 1. Since the pain from systemd comes from it running as PID 1, I don't see a problem with this.
If you want a system that's 100% free of all systemd code, even if it's not doing anything, then you need Devuan, but personally I don't see that as a compelling reason to fork Debian.
Frankly, it's an ugly hack, not a proper solution. I hope to run my servers for years to come and the upgrade process should be as smooth as it can be.
Opensource is about alternatives. Forks are a good thing,I wish devuan great success so people like myself won't have to whine about systemd all the time. If you like systemd and it's philosophy,I wish you the best of luck with it. Just keep in mind that others don't have to like it and different people or organizations have different needs.
Whike problems are surmountable, rehearsing for failure is advised.
There definitely is bad too.
The real issue with systemd is that there simply aren't enough volunteers (both individuals and companies) out there willing to contribute. In short, since gnome is pretty much developed by RedHat and systemd is developed by RedHat, gnome is tied into systemd.
If there was enough of a community around gnome that RedHat wouldn't be able to pull it wherever they want, gnome wouldn't have been able to be that coupled with systemd. The issue is that debian simply doesn't have the manpower needed to de-systemd gnome.
* http://jdebp.eu./FGA/debian-systemd-packaging-hoo-hah.html
It seems dead.
For me it is not only a Sunday morning essay, but also important that people understand, why the Devuan movement is so important to all of us.
Can you expand on that? I work at a cloud provider, and I never heard of such issues before. Our default images are Ubuntu 16.04, RHEL 7 and SLES 12, so systemd all around.
This could be an honest mistake. On my system, systemd-fsck-root.service and systemd-fsck@.service have `TimeoutSec = 0`, which I guess to mean "no timeout", but the systemd-fsck executable could have its own timeout. In any case, filing a bugreport sounds much easier than switching distributions.
And if the fsck is just a scheduled one (after x reboots) instead of being triggered due to filesystem weirdness, then getting the server back up and serving can be the higher priority.
That's just an example because you asked though. Wouldn't want that same fsck to be skipped if it was actually started due to filesystem weirdness being detected. :)
I have to get our systems people to write a paper on it. My experiences with systemd are pretty much perpendicular to what some people claim.
Anecdotal evidence is useless.
What you're suggesting is basically spreading FUD. Your article didn't point out a single actual issue with systemd.
I run large scale VM infrastructure on a systemd distro (RHEL, in particular) and I've yet to encounter a single issue that was caused by systemd or, for that matter, any issue with systemd that wasn't easily resolved.
That doesn't mean that there aren't any. Also, FUD (as I understand it) is people spreading unfounded rumours, not people documenting their own experiences and problems with a piece of software. The intent is incomparable.
If you are actually interested in tangible structural flaws in systemd, look no further:
https://suckless.org/sucks/systemd
https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awa...
Also, see how Poettering handled this zero day (Which grants root to any user with a numeral): https://github.com/systemd/systemd/issues/6237 HINT: He doesn't think it's a bug...
The fact that this even fucking exists in the first damn place: https://latesthackingnews.com/2017/06/29/a-systemd-vulnerabi...
See also: http://without-systemd.org/wiki/index.php/Arguments_against_...
This formulation may be misunderstood. The bug did not grant users root rights, it ignored the user= setting on a service if the service was intended to run as a user stating with a numeral. So you already needed to be root to create a service. The bug allows a service to run with more permissions than intended though.
* https://twitter.com/Serianox_/status/922050013655650305
* https://twitter.com/gibbon_/status/935786202724208641
* https://twitter.com/ManChicken1911/status/931820892291608576
* https://twitter.com/hawko2600/status/931082871187505153
* https://twitter.com/RPaschedag/status/930860506256113666
* https://twitter.com/BoolKiRool/status/930422520074919936
* https://twitter.com/infosecabaret/status/929069203193200640
* https://twitter.com/sirzeitgeist/status/925535140947922944
* https://twitter.com/saruspete/status/925182958012715009
* https://twitter.com/iMilnb/status/928260238322651136
* https://twitter.com/patrick_mooney/status/936374590070185984
* https://twitter.com/patrick_mooney/status/930929899686182912
* https://twitter.com/patrick_mooney/status/930929899686182912
* https://twitter.com/patrick_mooney/status/925026104972165120
* https://twitter.com/patrick_mooney/status/925023617674461184
* https://twitter.com/patrick_mooney/status/923815696894607360
* https://twitter.com/warnecke/status/915010187932717056
Truth be told, I'm fine with monoliths. They just have to be good. The Linux kernel is a monolith.
The real problem to me is how systemd makes the rest of the system more complex and integrated to systemd. Dbus is also not a great IPC layer, and yet it's the current state of the art.
We need better APIs and foundations. Systemd doesn't provide those; it just lets everyone depend on systemd. Good for Redhat, bad for innovation.
Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we figured out the right ordering of units, it went pretty well.
The greatest pain was the networking.
Systemd is a highly opinionated system. And it is opinionated in ways it really shouldn't be. Happily some aspects are configurable, and in much of our system setup, I reconfigured some of the more egregious settings. Some aspects were simply painful, such as networking.
Years ago, I dealt with other highly opinionated, and similarly broken systems. These systems insisted on doing things in their order to bring up their services, even if I didn't need them, because they maximized dependence radii. Which, for the life of me, I did not understand for my use case. But I could see it for other use cases.
My approach to dealing with this was to provide the absolute minimal basis for that system to operate, and then exit its configuration as rapidly as possible, given our experience with its bugs. That is, have it execute for the bare minimum possible time, before we transfer control to something we've developed, that actually works.
I adapted that to systemd. While I had to put up with all sorts of timeouts for systemd services that I could not adequately control, and could not remove due to these insane dependence radii, I could tune those timeouts way down. Which enable me to escape the systemd startup within reasonable timeframes. And then allow my code to take over.
Customers didn't notice unless they looked at bootlogs. They simply saw a reliable service. Which was made reliable after working around systemd's myriad of shortcomings. I could not get systemd to do what I needed, my opinions were different than its, and its control plane couldn't fathom what I needed to do. So the idea was simply push it out of the way as rapidly as possible.
Last time I checked, systemd - the init system - wasn't involved with networking at all.
There's systemd-networkd, but that's an optional component and no major distro is actually using it as a default. RHEL 7 - which runs systemd - supports both NetworkManager and their legacy networking scripts.
You said there were "myriad of shortcomings" and "egregious settings", some examples would help.
On egregious settings, google is your friend. However, here are just a tiny smattering of what I had to deal with. These were dealt with over a few years of delivering and supporting systems that had to work reliably and predictably, in a supportable manner. Each line has often a significant amount of debugging time invested behind it before we came up with the line you see. $TARGET is the target install directory for the image.
This is just a small sampling of the open source build system, that I grabbed from some of the configs we used.
Yes, systemd is broken, but not irretrievably. It is fixable.
# fix some systemd timeout brokenness
# fix systemd journaling. Yeah, really # fix the INSANE logind.conf per user directory size ... hard code it to 256M # # fix the INSANE logind.conf KillUserProcesses problem, which nukes nohup/tmux/screen ... # mask off systemd-udev-settle ... yes it is broken [edited for formatting]Are you sure about that? To my knowledge, Debian 9 is still using the good old network scripts. I'm 100% about Debian 8 since that's why I use in production, with systemd-networkd nowhere to be seen.
I agree about it being less flexible. Either way, it's a separate daemon not tied to systemd.
> fix some systemd timeout brokenness
Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s).
ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware watchdog reboot your system may not always be a good idea.
> fix systemd journaling
Debian runs a non-persistent runtime journal by default and leaves the long-term storage responsibility with the syslog daemon. What's wrong with this? If anything, they're being too conservative.
> RuntimeDirectorySize
What's wrong with the 10% of physical RAM default? Hard coding this to a small value is potentially messing with things like flatpak's portals which store large-ish data there.
> KillUserProcesses
Off by default in Debian. I specifically turn it on, since it kills any rogue user processes when they log out and makes sure nobody leaves anything running in a screen instead of doing it properly (I've even seen a crashed vim instance max out CPU on a server)
> mask off systemd-udev-settle ... yes it is broken
Masking this introduces race conditions into your boot process. The systemd-udev-settle unit calls "udevadm settle", which waits for all pending udev actions to complete before any services are started which may depend on it. If anything is broken, it's the particular subsystem which takes too long to initialize/does not support async events.
Important examples are LVM and iSCSI. If you mask the settle unit, systemd won't wait for all devices to be ready. This may or may not work and the safe default is to wait.
The last job developed/supported Debian based high performance storage systems in production, starting with Debian 7, then Debian 8, and Debian 9.
root@localdns:~# cat /etc/debian_version
8.10
root@localdns:~# locate systemd-networkd
/lib/systemd/systemd-networkd
/lib/systemd/systemd-networkd-wait-online
/lib/systemd/system/systemd-networkd-wait-online.service
/lib/systemd/system/systemd-networkd.service
/usr/share/man/man8/systemd-networkd-wait-online.8.gz
/usr/share/man/man8/systemd-networkd-wait-online.service.8.gz
/usr/share/man/man8/systemd-networkd.8.gz
/usr/share/man/man8/systemd-networkd.service.8.gz
Its there, but I disabled it in this case.
> Why are you decreasing the timeouts? I actually increased them in my case to give services more time to exit (upstream default is 90s).
This comes from long debugging/tracing sessions where we were trying to understand why we wound up with effectively repeating timeouts.
[...]
On systemd-udev-settle, allowing it to run was actually the cause of a race condition on start. The only way to fix this that I found was to disable this. Once we did this, no more race condition. The system booted normally.
It sounds like your experience is significantly different than mine on this. We built very large high performance computing and storage systems for many users, where the things that people used simply had to work, without surprise. Every line I showed came from a set of long and often painful debugging sessions where we had to figure out what new change had broken a key bit of software.
The RuntimeDirectorySize issue actually tickled another bug, and caused some incredible paging problems when used with a specific software package. Enough end users on a system (we had many) could launch an effective DoS against the system by using these tmps.
Generally, my observation, and I've seen others make a similar set elsewhere, is that the default and general distribution settings for systemd parameters appear to be generally set for desktop users (single or very few users per machine). Not for large shared resources.
I thought the same at first but after trying to maintain a Debian installation without systemd, I was convinced otherwise. The "infinite cost" was not caused by Devuan developers but by Debian developers.
I felt the Debian transition to systemd was handled very badly and was rushed in order to get Jessie out of the door. Debian still provides the sysvinit-core package as if it were a supported init system however if you use it you'll find the operating system has many quirks and many things are broken. Before Jessie was released, I commented on one bug report related to a hard dependency on systemd for NetworkManager. Instead of fixing the regression, the package maintainer gave the excuse that they don't have the resources and would need more maintainers. I sympathise with the package maintainer but this is the sort of thing that led to this mess. I remember there being a lot of hostility in that period from Debian developers towards those who did not wish to use systemd.
Based on my experience, I think the right decision was made to "fork" Debian to create Devuan. It's a shame that they had to do this. I still hold hope that maybe one day all of the work put into Devuan can be reintegrated into Debian and it can return to be the "Universal Operating System" that they still claim to be.
I don't know whether that means it's "supported" or not.
A Debian developer built a deb package for it and added it to the "main" repository. Everything under "main" is normally considered to be part of the distribution. As you can see from the link I provided, the package was maintained for several years and received several updates until it was dropped for Stretch. That is exactly what I would expect from a supported package in Debian.
runit-init had been removed in 2010. Felix von Leitner's minit was later removed by Debian member Iain R. Learmonth in November 2015. upstart was removed as a consequence of a bug report (#789524) filed by Kyle Amon in June 2015.
One of the contentious events during the Debian Hoo-Hah was one of the package maintainers leaping to remove support for an init system from one package almost as soon as the Technical Committee had made its decision favouring systemd over upstart, OpenRC, and van Smoorenburg rc. People opined then that Debian package maintainers should not blithely remove support for init systems like that.
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746715
More recently, however, this has begun to happen. In August 2017, a group of Debian people got together to organize the removal of all upstart support from all packages; and that is progressing.
* https://lintian.debian.org/tags/package-installs-deprecated-...
As for systemd, it wasn't as a Debian package, so it's hardly relevant for this discussion. It was only even packaged in 2012.
You have a lot to learn about the rather sad history of runit and daemontools in Debian, as well as about the ways that Debian people decide what is packaged. This is only some of it.
* https://tracker.debian.org/pkg/runit-run
* http://smarden.org/pape/
* https://lists.debian.org/debian-vote/2014/11/msg00059.html
* https://unix.stackexchange.com/a/284453/5132
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766187#78
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861536#44
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=752075#35
Runit-init-the software was not removed from Debian. Only runit-init-the-package was removed and that was long before systemd-the-package (which, yes, was what I was referring to) appeared, and it was by a ROM from the author himself, after a bug report classified as RC (justifiably, in my opinion).
So I fail to see any changes in behavior by the Debian team, since I don't believe in this conspiracy theory that Debian Developers were finding bugs in other init systems just to push for systemd, despite not even caring enough to package it.
Could they be more open to other init systems? Probably. But that was never my question.
After propounding a non-existent concept of "obsolete as a Debian package" you are proceeding to propound an equally nonsensical concept of removing runit-run whilst not removing runit-run. https://tracker.debian.org/pkg/runit-run is really quite clear and unambiguous.
And that conspiracy theory is a straw man of your own making, too. As is indeed the commentary on "the ways that Debian developers make their decisions" which are your opinions not anyone else's. You have a lot to learn and you're getting a lot of things quite wrong.
Mind you, there are plenty of other complaints (stability, back-compatibility, etc.) that are useful/valid regarding this software, but the above litmus serves to eliminate rather a lot of FUD and hand-wringing in my experience.
Paradoxically, this improves the signal of discussions about systemd's flaws, and there are plenty. With the obviously irrational/haven't-used-it/axe-to-grind stuff filtered out, it's much easier to learn about real features, problems, fixes, benefits, and drawbacks of any project, systemd included.
Actually, I replace systemd with *BSD. You know, the OS which has blessed core packages which are not replaceable?
SystemD has opaque behaviour and binary interfaces. It's the antithesis of the BSD model. Despite BSDs being quite coupled groupings of software.
Anecdotally we're attempting to move to BSDs instead of upgrading to RHEL7 due to SystemD.
... and for deployment, the Golang debugger doesn't even run on BSD at this point. So, going to have to setup new Linux (likely CentOS 6 - no systemd - or 7) boxes soon now that we're getting closer to first initial real world deployment for the project I'm on.
It's just the "UNIX is supposed to be decoupled" line that gets repeated every time systemd comes up that BSD (which, ironically, those people want to switch to) disproves.
* http://jdebp.eu./Softwares/nosh/
So yes, the same centralization/reducing-choices arguments apply to glibc.
The problem with standards compliance is that, once something becomes ubiquitous enough, its compliance begins to "drift", and replacement parts get harder to find. Is it tricky to swap out the init component of systemd with something else? Not terribly. Is it tricky to swap out its dbus messaging setup with something equivalent that can still be used by unaltered software that was previously talking via the systemd setup? Yes, a bit. And so on. For equivalent hassles regarding libcs, check out some of the FOSDEM talks from the musl folks, or the alpine folks, where they talked about the difficulties in porting software that had come to depend on the quirks of glibc.
Or just, you know, google "quirks mode" and get simultaneously nostalgic and sad :)
I am running Void with musl on a Raspberry Pi 2. I love it.
At the risk of igniting something, the only serious recommendation I hear about systemd is improved boot times. My computers are rebooted only for kernel updates - this is the entire point of running Linux. I want uptime. I can stand a boot time of a minute or so if it's once a month or less. If I have to reboot regularly enough that boot times become a serious advantage, someone has done something very wrong with Linux.
In my opinion, the Debian developers were wrong to fully embrace systemd; it goes quite against the Debian principle of being able to adapt your machine to do anything. Yes, Red Hat held a lot of sway in convincing the community to adopt it, but Debian is a big distro as well - just look at the install base of Ubuntu. They could have held out and pushed back, but instead caved and now we're in the systemd mess. It intrudes so massively into userspace that it's impossible to get away from, and we wind up with the Windows model as this article points out - a massively complex spaghetti model of processes, all interlinked, all hinging on everything playing nicely. Just like Windows, it's a neat house of cards that could topple if any one of those crashed. This isn't Linux.
I'm not defending sysvinit, there's no denying it's showing its age and harks back to a much simpler time, but systemd is not the answer.
That's far from the only advantage systemd has.
Even then, boot times are extremely important for modern VM and container infrastructures.
I've run large-scale infrastructure on both Debian and RHEL, with and without systemd. I've been involved in many distro decisions in the past. The init system was never a major consideration, and I've always been able to work around whatever issues I've had with both sysv and systemd.
Here's some considerations which were important for my team the last time we had to make a choice:
- Vendor stability and long term support
If you're running a business, you'll want long term stability guarantees - both from a technical, and a business point of view. Running a small community distro with few - if any - commercial users is a huge liability. Yes, you always have the option to fork it or take it over, but unless you're in the business of building a Linux distro, it's almost certain to be a bad business decision since your competitors are spending their time on their products instead.
Anyone who doesn't appreciate long term vendor stability hasn't yet been burned by the lack of it. Your Gentoo wizard left the company? Too bad, good luck maintaining your infra now (I've actually seen that exact scenario play out not once, but twice!). A few core maintainers left for another project and you're left building your own packages? D'oh.
Having a mature and well-funded organization or a large community of commercial users (in the case of Debian) supporting your distro is extremely valuable, even if you aren't the ones paying for it.
- Ecosystem
The ecosystem is also really important. It's often the main motivation for using a particular distro. Development tools, third party package repos, 3rd party enterprise software, troubleshooting resources, documentation and much more depend on a healthy ecosystem.
With Devuan, this isn't as much of an issue, but it's still sufficiently different from Debian in sometimes subtle ways that it will nullify some of the advantages of being in a common ecosystem.
This also includes hiring - you'll have a harder time staffing your company if you're using exotic stuff and you'll unnecessarily spend your time training them. Ever wondered why companies like Google publish so many papers, talk about their infrastructure and even publish books about it[1]?
It's because it means they can - by advancing the state of the industry - hire people who are already familiar with their architecture and concepts. This is a real problem for them - Google, for instance, is in some cases so far ahead of others that they have to spend a significant amount of resources to bring their new hires up to speed. I don't know about Amazon and Facebook, but I'm sure they have similar problems.
[1]: https://landing.google.com/sre/book.html
- Security
Security follows the supply chain. Your security is only as strong as your distro's - no matter how good your security controls and processes are, if your distribution vendor is compromised, you won't stand a chance unless you have a world-class security team.
Likewise, your customers fully rely on you - if you're compromised, they will be, too.
Security is much more than signing your packages and publishing security advisories. In fact, those are merely the results of a properly implemented security management framework - risk assessment and mitigation needs to be an integral part of your vendor's (and your) processes. Where do they keep their PGP keys? Who has access to it? Is is stored on a HSM or just sitting on a server somewhere? Is there a process for revoking it? Is there a change review process? Does the build system verify source code integrity? Is there two f...
Debian does a particularly good job with security patching (better than Red Hat and Canonical, IMO).
For example-- if you do it today on a core app/lib without consulting upstream is that enough to get your credentials revoked?
I ask because Debian famously patched a library to quiet a valgrind memory error which resulted in their userbase generating predictable key material for a few years[1].
I trust them not to do that again (at least with libs that are obviously security related). But is there explicit policy in this regard?
[1] https://www.schneier.com/blog/archives/2008/05/random_number...
This is a good analogy to remind Devuan why they rejected systemd in the first place.
From a business perspective, the most important thing -- indeed, the only thing that truly matters -- isn't the software itself. It's support. When it comes to open source, you either use a supported configuration or you assume ALL the responsibility for supporting your systems yourself. Who would you rather handle the support on the off chance your machine goes tits up? You? Or Red Hat?
No wait, scratch that. Who would YOUR BOSS rather handle the support?
The supported init system for most flavors of Linux (that aren't in LTS) is systemd. Systemd makes the most sense from a business perspective, and "but muh Unix philosophy" doesn't hold when there's money on the line. May as well get used to it.
That's just it, some of us aren't interested in the 'business perspective' and have other uses for our systems.