I really like systemd. My blog has a 10 line .service file. My app has a 10 line .service file. My load balancer has a 10 line service file. They're way less brittle than bash scripts would have been. Stdout automatically becomes logs with the service name attached. It's nice.
Basically DJB did this in the 90s with daemontools, which many of us now use a revamped fork of called 'runit'.
You still write shell scripts, but you also still have control.
Again, I'm not entirely opposed to the ideas behind systemd, but I am opposed to basically every fucking Linux server distribution breaking because of things people are doing for the desktop, which began with upstart.
There is also no love left to lose between me and sysV init scripts.
I love djb's stuff - he has his own concepts of everything, though, including the FHS, which makes his stuff hard to run on regular Linux distros. If there was a djb Linux I'd definitely install it.
Sure, much in the same way you can use daemontools without strictly following its FHS requirements. Or more likely you'd be using one of its many derivatives, as vanilla daemontools is rather old by now.
Your system now has millions of more lines of code that few people understand which are vulnerable to privilege escalation, unfortunately. :/
I mean, you could just use Heroku, for free, most likely, to host what you are doing. Should all UNIX become Heroku? What happens if your blog starts doing a billion page views a day? When one of the systemd processes is taking up 95% CPU, what do you do?
I'm not saying it's bad, I'm saying these are the type of concerns its' critics have. I am a fan of a lot of the ideas. I have downgraded several Linux systems which I needed To Work(tm) because of systemd being completely half-assed, and there are some great audits of its' code which make it appear to have been written by kindergarten students.
I'd take systemd over sysv init, atd, crond, three or four watcher daemons, syslog, various other semi-maintained tools that replicate each other's functionality, and around 30x 50-line shell scripts.
Anyone who brings up sysvinit and the "30x 50-line shell scripts" is clearly one whose perception is limited to the ad-hoc LSB-based sysvinit setups that once permeated GNU/Linux, but which have always been merely one relatively inefficient approach. Even back when Debian first introduced their sysv-rc, we already had IBM SRC, daemontools, simpleinit and NetBSD rc.d which showed different solutions.
EDIT: Let me join in on the moaning about downvotes. Not that I understand why.
SysVinit is systemd's predecessor, and thus the logical comparison. Upstart would also be a relevant, if anyone had ever adopted it properly, unfortunately it never got past being a SysVinit wrapper.
I've used minimal shell based systems on BSD, Solaris SMF, and launchd, as well as Windows services, and maintain the successor to the now-unmaintained Unix Rosetta Stone. However a minimal shell-script based init is not something anyone on Linux is seriously considering, so I'm not going to debate it with you - that would be a waste of both our time.
Ehr, that's exactly how it used to be. Various sysvinit scripts that all differed across distributions. It's pretty appalling the way you respond though, just dismissing instead of trying to educate. I'm reading along and I have no clue what you're on about. Maybe you mean that there was other stuff out there? I care about what is on a distribution. E.g. RHEL had upstart, but in practice it was lots of large sysvinit scripts.
Have you taken the time to examine how a Linux system that uses a replacement for sysvrc written in the last ten years actually handles startup? Or what services people who aren't running a RedHat distro actually run on their Linux servers?
Or have you simply either taken the systemd folks at their word about the "sorry state" of sysvrc replacements, or -perhaps- glanced at a >10 line startup script and recoiled in horror?
Good. Far too many people in these conversations have poorly-informed opinions.
> Folks who aren't running 'RedHat' (sic) use systemd, sysvinit, or ... upstart [sic] ...
These are init and RC systems, not services. :)
> ...or pretend to use upstart but as a thin wrapper to sysvinit.
I dunno. The Upstart project page says:
"Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.
It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init."
If I look at the file list for the upstart package, [1] I see that upstart provides its own /sbin/init, /sbin/halt, and -it seems- just about everything that's provided by the sys-apps/sysvinit package on my Gentoo system.
Also, sysvinit and sysvrc aren't the same thing. OpenRC -the default Gentoo RC system- is a sysvrc replacement. [2] It relies on sysvinit (or similar) to get its work done, and is an obvious improvement over sysvrc.
If you were aware of the distinction, and have the time, would you be so kind as to mention the top N worst things about sysvinit? If you weren't aware, and have the time, would you care to learn the distinction, and then bounce on back and answer the query?
Yep, I assumed you meant service managers not services. People not running Red Hat use the same services as Red Hat people do. Everyone runs web servers, mail servers, NFS, whatever else. If you actually meant services, then what's your point?
Re further requests for info, I don't really want to continue the conversation, because you should have apologised by this point and haven't. Don't presume to know things about other people, it's rude and looks poor when you're wrong.
> Don't presume to know things about other people...
That would be why I started my initial comment with questions, rather than presuming that I knew your background and knowledge and launching into poorly-formed and ill-founded accusation and/or assertion. :)
I'm trying to have a serious, level-headed discussion about a topic that is -inexplicably- sensitive for a few groups of people. In order to do so, I need to discover the pertinent information about the background and knowledge of my conversation partners.
> If you actually meant services, then what's your point?
I did. That would be why I asked about services rather than RC systems. :) I know that many people don't work this way, but if you can try to keep in mind [0] that I try hard to say what I mean, and leave nothing but white noise in between the lines of my statements.
Anyway, my point is this. You said:
> I'd take systemd over sysv init, atd, crond, three or four watcher daemons, syslog, various other semi-maintained tools that replicate each other's functionality...
which I took to be a little strange. It seems like this is your understanding of the typical set of services a Linux system runs, but it doesn't jive with how either I or my handful of professional sysadmin friends run our systems. We generally don't run atd, leave any FS watching up to the software package that actually needs it, avoid using tools that duplicate each other's functionality like the plague, and tend to not use unmaintained software... unless it's clear that that software is "done".
I've known several friends and acquaintances who have gone off to work for Red Hat. They've -to a man- come back with some really strange, very hard-line positions about how Linux systems are "supposed" to work. One of them vehemently insisted that -because of the complexity of dependency management and testing- a highly-flexible, rolling-release distro like Gentoo Linux was impossible. [1] Because of my personal experience with how Red Hat seems to have enweirdened some of my friends and acquaintances, and because I knew next to nothing about your background, I wanted to ensure that you had indeed stepped out of the Red Hat bubble and seriously looked at what other companies and distros consider a typical set of services and the features provided by and operation of their sysvrc replacements.
[0] I won't fault you if you can't. I have quite a difficult time remembering what I had for breakfast, let alone the rhetorical styles of the couple-dozen one-off Internet conversation partners that I've had over the past several months. :)
[1] Note that his position was not "Only a fool would attempt to provide guaranteed enterprise support for any and all arbitrary Gentoo Linux configurations because of $COMPLEXITY."! I can't argue with that position. I asked him if perhaps he misspoke and that maybe this or something similar was closer to his actual position. He vehemently denied that it was, and doubled-down on his original statement.
> 'Did you do (basic research)' is an impolite way to suggest someone did not.
I ask basic questions because I often have conversation partners who have not done their basic research. In the past, I've wasted a lot of time having mid-to-advanced level discussions about topics with people who -as it turns out- didn't have even the most rudimentary grasp of the fundamentals of the topic. I ask brief clarifying questions of this sort out of respect for the busy schedules of myself and my conversation partners.
More than that, I've found that presuming to know something about other people often leads to hurt feelings and wildly incorrect assumptions. :)
From the start, I've been behaving in exactly the way you claim that you want me to behave. I even explained -at length- the reasoning behind my first and second comments and my conversational MO.
After your reply to my first comment, you've failed to meaningfully engage in any further discussion, citing my failure to behave in the way that you want me to behave as the reason for your non-participation.
We're getting really far afield here, but do you make it a policy to apologise to people (who aren't paying you) who continue to fail to understand something long after you've made that thing clear to everyone else in the room?
Let's pull the last one from the list, because not only is it the most important, but it's the one which best demonstrates the attitude the systemd maintainers have towards interoperability, standards, and security. Can systemd's logger send to redis stores for aggregation? Can it email if there's an issue? Can I pull windows events into the same aggregation systems? Is there an RFC for syslog's file and network format?
These are all flaws in systemd with just one of those daemons you mentioned. Were I to get into init, I'd ask you about BSD-style rc files, which are tiny and very readable, as well as being shell scripts, so you can run more pre-execution tests to ensure your job will run correctly. With BSD atd and crond, I'd discuss how atd functionality has been merged in as a cron job. There are a lot of other solutions to sysv init than systemd. Unfortunately, the Linux monoculture sees bash as the only shell, sysv init as the only pre-systemd init, and the poorly configured rsyslogd distros ship with as the only syslog.
> Unfortunately, the Linux monoculture sees bash as the only shell, sysv init as the only pre-systemd init, and the poorly configured rsyslogd distros ship with as the only syslog.
People like to make fun of Gentoo, but it does many things right. ;) OpenRC and netifrc are sane and flexible. syslog-ng is the "default" syslog and comes with a reasonable default config file. Use of logrotate is strongly recommended, etc, etc, etc.
> ... BSD-style rc files, which are tiny and very readable ...
As someone who has an on-going project to replace 157 rc.d scripts from FreeBSD 9/10, I can report that the generalizations about the Mewburn rc.d system that are often made, in order to contrast it with the egregious mess that one can find in the System V rc clone on Linux operating systems, are not in fact strictly accurate.
Yes, most of the rc.d scripts are small. Yes, the Mewburn rc.d system did almost 16 years ago what people are erroneously characterizing as novel to systemd: namely taking all of the common management code out of the individual service definitions, leaving just a few parameterizations in some cases. Yes, there is a definite common coding style that is a breath of fresh air when coming from the Linux world.
But no, the scripts are not universally tiny. One problematic example currently of interest is /etc/rc.d/bluetooth, weighing in at over 360 lines. Others include the subsystems that are split across multiple scripts, such as /etc/rc.d/atm1, /etc/rc.d/atm2, and /etc/rc.d/atm3.
Nor are they universally consistent, /etc/rc.d/serial being an example of one with quite different configuration semantics to the key=value-in-rc.conf{,.local} norm.
And whilst one might rail against the problems of having a common script for SuSE Linux, Debian Linux, and RedHat Linux, all of which have different sets of init.d script helper tools (resulting in "portable" init.d scripts ending up being a case "$distribution" in...esac statement for pretty much every step), one can equally point to the fact that OpenBSD's rc.d system (https://news.ycombinator.com/item?id=8506460) has entirely different function and parameter names to the NetBSD one (as adopted by FreeBSD/PC-BSD and DragonFly BSD) as well as different rules about rc.conf and different variable naming conventions.
Be very careful about attempting to claim some kind of moral high ground on interoperability and standards. (-:
Also be very careful about notions that the Linux world sees the Bourne Again shell as the only shell, some rather silly statements by some people who should know better notwithstanding. The adoption of the (Debian-modified) Almquist shell as /bin/sh by Debian and Ubuntu, and its significant effect on bootstrap speed, is quite famous. And my experience is that the Z Shell is quite popular, for starters.
I don't bite. You're affirming many of the claims while listing a few inconsistent counterexamples. This does not invalidate the original claims though, as, relative to the virtually complete lack of standardization in writing SysV rc scripts (LSB notwithstanding), it is in significantly stronger shape. There are indeed disparities in rc.conf and rc.subr between the BSDs, though probably not any more or even less so than disparities in LSB initscript frameworks.
Really though, the primary reason to cite rc.d is to debunk the pants-on-head claim of shell script-based rc being intrinsically spaghetti and voodoo.
It's not a few. It's somewhere between a quarter and a third of the corpus. "few" is merely the level of outright extreme cases, like "/etc/rc.d/msgs stop" not doing what one might think at all.
With two applications running under the same user, if one is compromised, it has ready access to the other.
Generally it's fairly easy to make a system user for an application if no special accesses are required - just make a custom system user for the application, and ensure it can access the various things it needs to touch (eg: logging).
... and if you want to learn more than has been already said, start with the famous design of qmail as a collection of mutually untrusting daemons that run under individual unprivileged user accounts, precisely to insulate each from the others using the normal operating system mechanisms that insulate one user's processes, files, directories, and so forth from another's.
A good question to ask yourself: "Presume that my daemon is compromised by an attacker ...
* ... what processes can it trace, debug, read/write the memory spaces and environments of, or send signals to?"
* ... what files, devices, and directories can it read and write?"
* ... what files, devices, and directories does it own, and can thus grant access to?"
Then ask: "How does the answer change if instead of all of my daemons running as the same (even if unprivileged) user they each run as different users, some (where appropriate) having no write access to anything and owning no files nor directories at all?"
It's not even the number of lines that matters -- because start-stop-daemon could probably accomplish all of that in "1" gigantic line.
It's the fact that it's incredibly readable. Someone with only a passing knowledge of UNIX and no knowledge of systemd could understand almost everything in this file at a glance.
I can't say I agree. Does Restart=always mean the service is restarted if it is killed? Or something else? Why is there a block named [Install]? Why would my service file need to be installed? What is multi-user.wants? How do I know that this sorta-kinda-but-not-really corresponds to Runlevel 5?
However, the reverse view raises similar questions. How do you know what runlevel 5 is? (Hint: Someone with "a passing knowledge of Unix" knows that there's no coherent definition of run level 5. It's "power the system off" on some Unices, for example. Someone without a passing knowledge of Unix is probably at a complete loss as to what "level 5" could be.)
In actual fact, the multi-user boot versus single-user boot notion goes back to pretty much the beginning (https://news.ycombinator.com/item?id=10207674), and the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism. It's been a Solaris SMF milestone name for over a decade. It was a concept used in the 4.1 Berkeley Distribution manual for init(8) in 1981.
> ...the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism.
Sure, but I had to do a bit of reading to understand that
[Install]
WantedBy=multi-user.target
meant: "This service should start in the runlevel called 'multi-user'".
It also seems... backwards to put runlevel membership in a service definition file. Runlevels are an administratively controlled thing. Does systemd have two or more places a given service's runlevel membership can be specified, or is the only way to change a service's runlevel in its service definition file?
Yes, I know that the notion of "runlevels" is probably "legacy". However, assume that -on my OpenRC system- I have two particular runlevels, one for regular multi-user operation, and one for operation when I connect to hostile networks. Transitioning to this second runlevel shuts down all services that listen on anything other than localhost. Transitioning back brings those services back up.
Although this situation is a little contrived, this sort of flexibility is valuable.
The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requres/RequiredBy/etc). Instead they are defaults.
What systemd actually does at startup, or when starting services individually, is look for symlinks. If I have a foo.service and a bar.service, and a symlink called foo.service.d/bar.service -> ../bar.service, then systemd will start bar.service automatically whenever I ask it to start foo.service. If it also sees a multi-user.wants.d/foo.service -> ../foo.service, then it will start foo (and also bar) whenever it enters the multi-user "runlevel". (This is almost exactly how sysv runlevels work, except that they're named rather than numbered, and they can depend on each other.)
I can make that symlink manually, if I want, or I can call "systemctl enable foo.service" in which case systemd will read the dependency information from the .service files and make the appropriate symlinks.
If I don't like how the distro or packager has set up the service file, I can override it. The service files live in /etc/systemd/system by default, but systemd also looks in /var/systemd/system for overrides. This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.
Putting the overrides in a separate directory is a pretty nice decision; it's always obvious what changes have been made, and upgrades never overwrite them. I can even put those overrides into version control and track changes to them over time.
> The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requr[i]es/RequiredBy/etc).
Is this true of the identically-named keys in the Unit section?
> If I don't like how the distro or packager has set up the service file, I can override it. ... This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.
You should look at how OpenRC handles service dependency specification. It's at least as flexible as the system you describe in systemd. :)
> Putting the overrides in a separate directory is a pretty nice decision...
Yep. At dependency graph generation time, OpenRC examines its master config file at /etc/rc.conf as well as config files in /etc/conf.d/ for -among other things- additions to the need, use, before, &etc declarations in the depends section of a given service's init script. To augment -say- the "use" list in the openvpn service, either add the line
rc_use="my_custom_service"
to /etc/conf.d/openvpn , or add the line
openvpn_rc_use="my_custom_service"
to the OpenRC master config file at /etc/rc.conf . If I -say- wanted to override OpenVPN's default dependency on the "net" virtual service, I would add
rc_need="!net"
to /etc/conf.d/openvpn (or perform a similar transformation as before and store in /etc/rc.conf).
> I can make that symlink manually, if I want, or I can call...
To add the openvpn service to the "custom_init" and "default" runlevels, I would simply do
rc-update add openvpn custom_init default
If I omit the runlevel, the service is added to the current runlevel. I could screw around with symlinks if I cared to, but why would I? :)
The OpenRC configuration processing system is substantially more powerful than I describe, handles the processing of OpenRC-standard config files [0] automatically, and lets you shift the complexity of a configurable init script that should be in a config file into a config file.
Because my overrides and augmentations to the distro-supplied init scripts are in text files stored in standard places, I -too- can easily put them in version control and track their changes over time.
[0] Which -shockingly enough- look exactly like what any Linux programmer would expect a non-XML non-JSON text config file to look like. ;)
[Service]
User=%i
Group=%i
# Note that the first argument of the command line (i.e. the program to execute)
# may not include % specifiers.
ExecStart=/srv/webapp %i
WorkingDirectory=/srv/%i/
PrivateTmp=true
Type=simple
Restart=always
RestartSec=2s
[Install]
WantedBy=multi-user.target
---- 8< ----
/srv/webapp is a simple script that will do "exec $1/webapp" which is the actual executable.
Reliable per-service logging was largely perfected by multilog.
Judging complexity by observing external configuration is nonetheless exceptionally naive. One must factor in how the key-value pairs are built into an internal representation and dispatched, which is where things become an issue. Non-deterministic boot order (man 7 bootup) is one of many undesirable trade-offs.
> Non-deterministic boot order (man 7 bootup) is one of many undesirable trade-offs.
Yeah, also something I found annoying with Upstart's induction into Ubuntu.
A predecessor at a place I worked at with a huge amount of traffic installed Ubuntu 9 on all of our memcache servers, and we found on a handful of other machines installed with Ubuntu 9 that non-deterministic boot order would randomly cause them to fail booting, and it was a coin-flip whether that would be a one-off problem or a forever problem requiring a reinstall.
When you have 4 8GB machines (this was some years ago) powering memcache for a billion requests a day you don't really want a power cycle to cause 25% cache misses for hours as you wrestle with hardware and boot config. :/
Upstart doesn't have a formal dependency system, instead it orders things based on named preconditions and postconditions called events. Though these can be used to fulfill ordering requirements, I'd wager the information wasn't granular enough for parallelization to be efficient.
Actually the main problem is that a lot of modern systems do not make their dependency graphs well observable, but instead treat them as in-memory ephemera that are constantly recalculated and cannot e.g. be written to persistent storage as known-good checkpoints.
This is one area where s6-rc gets it right. You have to compile a manifest of your service dependencies and consistently boot from that well-known state, so for any changes it must be regenerated.
serel was an earlier example of a service dependency system and parallel launcher which directly had you specify the nodes and edges of a directed graph as an RDF manifest. Clever.
Folks gush about how concise systemd unit files are.
For services that require no complicated pre-start checks or configuration, equivalent service start scripts are more concise and just as legible in either OpenRC or -AIUI- Upstart. (Look at my conversion of a simple systemd unit file [0] to an OpenRC init script. [1])
What's more, for jobs that need complicated pre-start logic of one kind or another -AIUI- systemd provides no way to bundle that logic with your service description file. You have to put that logic in another script file which is then called by systemd.
EDIT: I can't speak to Upstart, but even if your service has dependencies or needs to be started before or after some other service or class of service, [2] specifying them is similarly concise and legible. Add
depend() {
use some_optional_service
need some_requred_service
before some_service_or_service_class
after some_other_service_or_srv_class
}
somewhere near the top of your script. For most services, this adds three non-whitespace lines. :)
[2] For example: The network start scripts provide the "net" service. You might have a service that -for some reason- needs to start before or after the "net" service, but doesn't actually depend on it to start. This is what "before" and "after" are used for.
I love the systemd principles, as they were outlined by Poettering in the initial announcement. Configuration files instead of boilerplate shell script (and worse, shell script with special comments), socket-based startup, massive parallelism, cgroups, the end of PID files...
I just wish systemd itself had stayed there and that the additional functionalities it keeps developing lived in separate projects with well-defined interfaces.
I find it disputable that parallelism is all that desirable. In systemd's case it requires the introduction of the dependency system with its multifaceted job queueing modes and transaction management that easily make up its most brittle edges, at least w.r.t. to its service manager, specifically. Seemingly innocuous configurations can result in cyclical transactions and dependency loops, and the internal graph engine bookkeeping these relationships provides very little debugging context at all.
Considering parallelism is about startup optimization, a much better option is to load a process from a checkpoint image, e.g. one taken by CRIU or DMTCP.
Sadly, no. I actually might have mis-remembered that detail and made a liar of myself. :(
I tried to go back through the 4.1-rc1 kdbus merge request saga, but Gmane is so slow as to be unusable today and the only other lkml archive I can find that shows me the whole discussion [0] doesn't appear to do threads. [1]
A search for "CRIU" reveals Lutomirski mentioning that a particular bit of info passed around by kdbus ("starttime") would break CRIU. Greg K-H says "'starttime' has been removed for a while now. You sure that you're looking at latest?" Lutomirski says something to the effect of: "No, I haven't taken a close look at latest, just checked to see if that thing that I thought was really bad was still in it, and it was."
Later, I see Lutomirski wondering if kdbus would interact with '/proc/<pid>/map_files' in such a way as to break CRIU, but Michal Hocko says something to the effect of "Eh, it looks like it shouldn't, but I can't know for sure.".
Because I don't have the thread structure, I can't be sure that there weren't replies to any of those messages from folks who felt that there was a problem for CRIU. However, as it stands now, it's pretty safe to say that I fabricated a bucket of mud and threw it around.
[1] I've just gotta ask: am I an idiot, or is Google Groups's mailing list search feature absolutely terrible? I ask it to give me all instances of "CRIU" in a particular LKML thread. It gives me a pointer to the start of that thread, leaving me to go through messages ~20 at a time to find the messages I care about.
500k in systemd itself, or inclusive of all the systemd-something pieces which, when debating with most, are part of it when convenient and separate when not?
It's not in question that systemd is infinitely more complicated than equivalent sysv or other script-based init system. Complexity means attack surface.
I wasn't aware it was in question a more complicated system is more difficult to throughly audit, and therefore more likely to have bugs, security bugs being a subclass of bug?
Unless the systemd devs are immune to the same problems that plague literally every other software developer
Apparently, we're speaking Lojban here, not English, so therefore I'm not allowed to use figures of speech as they may be evaluated pedantically by those who wish to win an argument in bad faith.
But fine, you wanted numbers. First, systemd's LoC count is 699,342 across 1,912 files as of a git clone of their SCM[1] and gitstats five minutes ago.
That number was around 550,000 back in May[2], so kindly update your knowledge before regurgitating it mindlessly.
Debian's (sid) sysvinit package[3], by the same methodology, is 32,151 lines across 235 files.
Systemd is therefore about 22x (21.75179621162639) more complicated than sysv.
I'd go run the numbers for Upstart, but I'm not going to go download bzr and appropriate profiling tool to conclusively prove something that's common knowledge to anyone not in an internet argument.
And kindly stop complaining about downvotes - it's against the site rules.
I don't think that's a fair comparison really - one of the main complaints against systemd is that it's taking on too many tasks, init plus NTP, cron, etc etc. Those modules are superficially optional, but in reality, a great deal of them will be turned on in any distro using systemd.
The init system should be an init system and nothing else, IMO, so targeting all that extra complexity (which was my original point) is fair game.
That's just it though—each task is a completely separate application. You're comparing all of these applications (one of which is an init system) to just an init system (sysvinit).
Whether a distribution chooses to include timesyncd or another ntpd doesn't magically make timesyncd an init system.
If sysvinit also included an ntpd client, _then_ it would be appropriate to include that module as well for a fair comparison of SLOC.
Another example would be to take vim, counting the SLOC including a bunch of downloaded plugins, _then_ comparing that number against emacs without any plugins.
It's comparing apples to apples and oranges simply because they're in the same basket.
Those "separate applications" are still tightly coupled to the main, and therefore they share the complexity/attack surface concerns.
If your system gets pwned, does it matter to the user if it was via the "systemd" binary itself or by one of the official "systemd-somethingd" binaries (that have no other purpose other than interoperation with the main app, developed by the same people) that come with every OS using it? Had the distro maintainers not shoveled an extra ~670,000LoC down their throat, they wouldn't have been pwned.
If systemd wants to replace my freaking cron, DNS, ntp, and logging daemons, and the majority of in-production deployments use the application like that, I will be evaluating the application's complexity against how it's used in the real world.
sysvinit doesn't attempt to replace resolvconf, ntpd, crond, and syslog-ng. systemd does.
> Those "separate applications" are still tightly coupled to the main, and therefore they share the complexity/attack surface concerns.
They have no access to systemd-init, only to an API presented by it (as does every other application if they so choose to use it), and thus this statement is incorrect. A bug in one of those applications cannot magically get access to systemd-init (which I'm assuming is what you meant by "main" here).
> If your system gets pwned, does it matter to the user if it was via the "systemd" binary itself or by one of the official "systemd-somethingd" binaries (that have no other purpose other than interoperation with the main app, developed by the same people) that come with every OS using it? Had the distro maintainers not shoveled an extra ~670,000LoC down their throat, they wouldn't have been pwned.
It absolutely matters! A bug in systemd-init can do much more damage than a bug in timesyncd could. Also, I really don't care about your anger towards an open source project, please refrain from expressing it to me if you'd like to continue this conversation.
> If systemd wants to replace my freaking cron, DNS, ntp, and logging daemons, and the majority of in-production deployments use the application like that, I will be evaluating the application's complexity against how it's used in the real world.
You may want to create a separate post somewhere which compares the SLOC of all the applications systemd attempts to re-create (choosing perhaps the most popular from each) along with the complete systemd package, but I personally wouldn't have any interest in it since it would be a factually correct comparison.
> sysvinit doesn't attempt to replace resolvconf, ntpd, crond, and syslog-ng. systemd does.
That's exactly right, but systemd-init doesn't attempt to replace those either (besides syslog-ng (journald), which I already mentioned as being fair to include with the SLOC count)*.
fwiw, this post has a good list of all the binaries included in the systemd project, and their reliance on PID1 or the systemd (systemd-init) binary (as they use the API).
Your comments are breaking the HN guidelines both by being needlessly abrasive and by going on about downvoting in a way you're explicitly asked not to. Please reread the guidelines and follow them.
Anything that requires DBUS is (at minimum, because it now depends on DBUS, which is -itself- rather complicated) substantially more complicated than a script that does
If that complexity isn't exposed, then it isn't more complicated. Lots of init scripts used to be quite different across distributions and very long. Nowaways the systemd unit files are shipped upstream and you just package them. That's vastly less complex.
It's less complex in the same way that a C program is less complex than equivalent assembly - the complexity is just moved one layer of abstraction down, but it still exists, and can still bite you in certain ways.
> If that complexity isn't exposed, then it isn't more complicated.
It may not appear to be more complex to the user, but when your software has a firm dependency on a complex system, it becomes more complex because it is affected by the failure modes, quirks, and restrictions of the software on which it depends. [0]
> Lots of init scripts used to be quite different across distributions and very long.
I've talked elsewhere in these threads about how systemd's unit files simply shift the complexity for services that need (for whatever reason) complex pre-start logic out of the unit file and into hand-maintained tools. See [1] for my most recent comment on the topic. Make sure to check the footnotes in that comment, too. :)
[0] Entirely valid anecdotal evidence on the relative stability of dbus doesn't invalidate my point. dbus is one of many complex software projects that others depend on. :)
Most people, including the authors of systemd would be very surprised if there were no problems with systemd.
Here's a couple of fun problems from years past.
Docs say NoNewPrivileges as yes "prohibits UID changes of any kind", but that's not exactly true.
It's spammed so much debug info the system fails to boot.
"systemd, when updating file permissions, allows local users to change the permissions and SELinux security contexts for arbitrary files via a symlink attack on unspecified files."
"The rm_rf_children function in util.c in the systemd-logind login manager in systemd before 44, when logging out, allows local users to delete arbitrary files via a symlink attack on unspecified files, related to "particular records related with user session."
It's complex software running on many different kernels, and many different distributions, doing many different things with lots of code written in C. There were over 80 people mentioned as contributing to the last release. Of course there are going to be problems!
I hereby coin Vacri's Law: When a comment-maker announces incoming downvoting, they're aware their argument is weak and is instead trying deflect criticism through shaming others.
This is a much more measured response to my comment than your first attempt which started "yeah, you got me there". This is what downvoting does - it makes you moderate what you have to say.
In any case, you're being a hypocrite. From your very first comment in this thread ("clearly done a full audit") you've been snarky, and now you're crying foul that people are being snarky back. I actually laughed out loud when you invoked the "good faith" HN guideline against Karunamon.
They have a bootloader now. I'm sure this is very beneficial for certain types of people.. my developer friends who are becoming sysadmins by way of 'devops' like systemd quite a lot.
Me? I've gone to freeBSD for personal things; until Apple successfully pushes launchd upstream..
If you want to see how far spread systemd is actually becoming install arch linux using the official guides.. it's very easy to set up- but you definitely can see how it's "everything that isn't linux that your computer needs to be functional"- I'm sure they'll have a wayland fork in no time.
Further still, jkh's primary dominion is over iXsystems products, not upstream FreeBSD itself. For instance, he's not on the FreeBSD Foundation's Board of Directors despite being a high-profile figure.
On the other hand, though desiring launchd is not insanity, I do consider it hasty and not well thought out, as you might know.
I'm still waiting on systemd getting its own text editor, with keybindings that are a random combination of emacs and vim ones, at which point all config files will switch to a binary format that cannot be read by normal editors.
Following that logic, it would make more sense for systemd team to get an own language, something with traits of rust, go and haskell, and port everything to that, at which point it wouldn't be much of a stretch to drop libc and with it all support for "unsafe" languages.
Quite an interesting update. Good to see ripping and replacing of code with external deps where needed, and exposing some more kernel functionality (around cgroups especially).
Pity the comments are largely the usual moronic and entirely non-technical nonsense that infests the topic at HN. Sub-reddit commentary quality.
Pity the comments are largely the usual moronic and entirely non-technical nonsense that infests the topic at HN.
I'd think I and others have been trying to inject some reasonable commentary. Any objections, or are we out on a political vendetta as is usual with systemd?
No more problems with overflowing logfiles, no more debugging undocumented shell scripts for booting, no more writing init scripts, no more debugging undocumented 200+ line init scripts, no more minute-long boot delays, no more brittle, half-assed distribution-specific networking solutions (also undocumented shell scripts), ...
For me as sysadmin, systemd solved a lot of long-standing problems, while so far introducing no new ones.
Not quite. The journal is prone to fragmentation, though it can be capped just as most sane loggers like multilog can.
no more debugging undocumented shell scripts for booting
no more writing init scripts
no more debugging undocumented 200+ line init scripts
Redundant complaints, and irrelevant. Not all script-based init systems are created equal, and many init systems throughout history have eschewed shell scripts [1].
no more minute-long boot delays
Incorrect. Boots can still stall from dependency loops, cyclical transactions, stuck start/stop jobs, an improperly calculated service ordering in the parallel dependency graph, etc.
no more brittle, half-assed distribution-specific networking solutions (also undocumented shell scripts)
Incorrect. networkd is tailored to specific purposes (largely containers) and not a general-purpose NetworkManager replacement, nor does it necessarily deprecate any ad-hoc network configuration scripts (which aren't even intrinsically bad).
Why, yes, I could clobber together my own server distribution with multilog and obscure init systems and whatnot, or… I could just use the distribution default, which massively improved over the old status quo with the introduction of systemd.
> The journal is prone to fragmentation, though it can be capped just as most sane loggers like multilog can.
So my choices are a) systemd + journald, which Just Works™ by always feeding units' stdout to it without any further configuration, b) syslog and logrotate and oh I have to patch software to use syslog in the first place and make sure logrotate rotates the files correctly if a daemon still its logfile open and… yeah, screw that, I've wasted enough of my life with that, c) something where I manually set up everything to use multilog.
> Redundant complaints, and irrelevant
From your ivory tower, maybe. Meanwhile I'm sitting in the trenches and banging my head against the wall because someone again managed to break their 300+ line init script (hi, MySQL!).
> Incorrect.
I did not mean failure cases. The average case, where every service starts up in a correct order on sysvinit/systemd, went from ~60 seconds to <20. Now, a reboot is usually faster than the failover timeout.
And unlike with Debian's "dependency-based sysvinit", systemd works. I've yet to see a failure case that wasn't caused by either hardware failure or gross operator error (e.g. referencing /var wrong in the fstab).
> Incorrect.
Well, it looks I have to throw away my perfectly working networkd setups then, if you say so.
> NetworkManager
I wasn't even talking about that. I was mainly referring to Debian's networking scripts (which have been broken and unfixed for half a decade now) and Arch's netcfg/netctl which all three never worked as advertised. If NetworkManager was actually usable on headless devices, I might prefer it over networkd, too (assuming I find a use case where either it or connman are insufficient).
No, but those of us who do generally get downvoted for mentioning it in these comment threads, so many just stay away.
Paradoxically, due to the positive feedback loop of people downvoting comments that they disagree with and thereby driving away dissenting voices, comment threads on polarizing topics like these tend to be a pretty poor representation of actual popular opinion, as opposed to the opinion of the loudest people in the room.
I mean, it's not that I'm going to rain on your parade if you want to use systemd. But by and large most systemd proponents seem to scantly have any idea why they like it beyond parroting misconceptions and half-truths, much as many critics aren't sure why they hate it.
I'd take it you've actually studied the Unit object, the internal graph engine, the transaction manager, the seven different job queuing modes, the contextual relationships between services, jobs and units, the semantics of the dependency and relationship model, how execution state is composed, etc.?
Those are seldom talked about, but are some of the primary reasons I object to systemd. Are you neutral on those aspects, or do you see them as definitively positive?
Not having looked at the internals of systemd, I'll take your word for it. The question is, is it possible to offer a similar feature set with a simpler model?
> ...you are implying he doesn't know what he's talking about...
It's good to first be charitable when attempting to divine a conversation partner's motives. If you go back and read more of what vezzy-fnord has said in the past, you'll find that he doesn't really rely on innuendo, and has no qualms with addressing false, misleading, or incorrect statements head-on. So, when he says:
> But by and large most systemd proponents seem to scantly have any idea why they like it beyond parroting misconceptions and half-truths, much as many critics aren't sure why they hate it.
he is stating the entirety of his observations, rather than slyly implying that his conversation partner doesn't know what he's talking about.
I think systemd has the misfortune/honor of touching decades-old subsystem interfaces. Any time anyone ever makes a change to something that's worked the same forever faces criticism. systemd's feature set is so attractive that it's getting/gotten adopted by the vast majority share of linux distros. The folks who didn't prefer it but thought that they could ignore it may have been surprised and vocal when it showed up in their favorite distro.
>undocumented 200+ line init scripts ... undocumented shell scripts
Maybe we've just traded those for undocumented C init programs?
A few people have attempted to use the systemd documentation to create a reimplementation. Bearing in mind that the task is complex and largely thankless, all the folks making attempts that I've heard about have reported that the available documentation does not provide enough information to complete such a task.
Note: I KNOW that documentation is seriously hard. But, one of the rebuttals of the "Systemd is complex and monolithic" assertion is "Systemd is fully documented, so you can replace any part of it with one that you've authored.". You can't honestly make that rebuttal if the state of your documentation doesn't support your claim.
That's because if the code is over some quality level (which can be measured with bugs count) then the source itself is implementation documentation. And systemd contains the more classic documentation of its behaviour for its end users.
> ...if the code is over some quality level (which can be measured with bugs count) then the source itself is implementation documentation.
Oh, you summer child.
In any complex software project, there's always a difference between the contract described by the system's specification and the contract described by the software as implemented.
The big difference between the explicit contract of the documentation and implicit contract of the code is that -in areas on which the documentation is silent- the contract described by the code is subject to change at any time, without warning. Raymond Chen [0] has spilled much ink over customers who have gotten badly burned by changes in the implicit contract some time after they made use of something that the code permitted, but the documentation did not describe as possible and/or supported.
[0] Microsoft most notable backwards-compatibility guru
> In any complex software project, there's always a difference between the contract described by the system's specification and the contract described by the software as implemented.
That's exactly the reason why source itself is better than implementation documentation - because it is always "current".
In the second story:
The scribe showed that source code may be too detailed, but at the same time he showed that past view of the system will not accurate now if not updated constantly. Which is the point I'm making.
If user behaviour doesn't change - why changes in implementation are wrong? Reimplementations can use the source code at one place in time and use that as a reference.
Personally, I hate not being able to dig into it when something goes wrong. But that familiarity will come with time.
What I absolutely LOVE is a much more robust ordering of system service turn ons. Start network BEFORE ntpd. Start rpcbind BEFORE nfs-server. A lot of hack solutions for this chicken & egg problem in the past were always very fragile.
FYI, all ordering options in systemd are relative and not absolute. You cannot guarantee in systemd that X will be started before Y, because that's not how it works. You declare an approximate dependency and ordering scheme, which is then fed into a graph, optimized and recalculated in memory on every bootup and state-modifying operation. There is no determinism in the boot order due to parallelism and some other constraints, so you can potentially have services be malignantly reordered or fail because of cyclical transactions, loops and stuck jobs.
The point is that, if you're thinking of it as a linear ordered sequence, you have the wrong mental model. On the other hand, systems like OpenRC do work that way.
> if you're thinking of it as a linear ordered sequence, you have the wrong mental model. On the other hand, systems like OpenRC do work that way.
Are you sure that OpenRC works in this way?
There may be some subtlety that I'm missing, but the documentation for systemd unit files [0] seems to describe a service dependency and ordering specification system that looks equivalent to what OpenRC provides with "need", "use", "before", and "after". (Search for "[Unit] Section Options" and read on from there.)
Again, don't be deceived into judging an architecture by its public interface. It certainly appears similarly, but the dependency and ordering primitives are interpreted much more loosely like tags than they are like absolute requirements.
Simplifying things, your unit file actually becomes a Unit object which queues a job (which is itself a Unit) which is scheduled and checked for consistency in a transaction, which encapsulates a service (itself a Unit and one that controls jobs). The final graph of dependency information largely does not reflect what you configured.
> It certainly appears similarly, but the dependency and ordering primitives are interpreted much more loosely like tags than they are like absolute requirements.
Ah. That would be the "subtlety" that I was missing. Thanks! :)
As an aside: how terrible it is to provide an interface that makes it look it gives you ordering guarantees when -in actuality- it gives you no such thing! :(
Well, in fairness, the developers are honest about it. But they perhaps do not make it as obvious they should, and a lot of people are fooled (I'd cynically argue to the benefit of the systemd developers).
Upstart did all those things without needing to worm it's way through the whole system like systemd does.
You're mistaking people's dislike for systemd with a love of sysv init - no-one liked sysv init and everyone wanted a change. It's just that systemd has taken a very broad remit, plus the maintainers have a 'fuck your use-case' attitude.
Logrotate solves this for me, for every logging method. How did it fail for you?
> ...no more minute-long boot delays...
I don't have this issue with OpenRC on Gentoo. shrug I do know that Kubuntu 15.04's boot time is irregularly slow, and that its shutdown process -every once in a while- hangs forever, requiring the use of SysRq keys. This didn't happen with the previous, Upstart-based version of Kubuntu. (Then again, there are many things wrong with Kubuntu 15.04, so this might not be entirely systemd's fault. ;) )
> ...no more debugging undocumented shell scripts for booting...
Scripts written for OpenRC are typically easy to read. Complex tasks -necessarily- require complex scripts, but you'd have to write and call out to those scripts with systemd, too!
> ...no more writing init scripts...
For services whose startup is as simple as what systemd handles out of the box, I can write a one-to-seven-line startup script for OpenRC and call it a day. Additionally every service packaged in my package manager has come with a init script. [0]
> ...no more brittle, half-assed distribution-specific networking solutions...
shrug Gentoo uses netifrc. It does appear that this is a distro-specific way to bring up the network, [1] but I assure you that it's full-assed, has a well documented and flexible config file, and is not brittle.
[0] Some of them -like the network startup scripts, and the OpenVPN startup scripts- let you make a symlink to the original script, which changes the behavior of the script.
In the case of the network startup scripts, this makes that script change the state of a particular network interface, virtual or otherwise.
In the case of OpenVPN's script, this causes the startup script to look for config files with a different name. In the OpenVPN case, you symlink /etc/init.d/openvpn to /etc/init.d/openvpn.newconfig . The symlinked openvpn startup script then uses the OpenVPN config file /etc/openvpn/newconfig.config to start the "openvpn.newconfig" service. It's really handy!
How would you do this in systemd? Make a new unit file?
[1] Only because -like all distro-specific things-, noone else has bothered to adopt it. ;)
Who needs calculators? The same can be done "by hands".
But the problem with that is that doing it many times is error prone. Why not let it be solved once for good?
Have you ever used the BSD's rc? How about launchd? SMF? The biggest problem with systemd is that it goes about trying to fix the problems with traditional sysv init in a rather inelegant, inconsistent way. In your sysadmin roles, how do you now aggregate your logs for later auditing? How often do you really reboot where you'll experience that minute long delay? There are definite problems with sysv init, and systemd is a bit better, but there are a lot of other systems that have solved those problems too, often in a lot cleaner fashion.
I like it, as well. It simplifies many things and no practical functionality is sacrificed.
Most arguments against sysyemd appear to be appeals to tradition, not based on any technical deficiency. In actual use I have found no fault with it and it does make some things easier.
The only argument against sysyemd that I find somewhat sympathetic is the fact that there is always a cost to pay when migrating to and learning about something new. This is certainly true. However, I find the user experience of systemd to be superior, and this small upfront price seems worth paying.
I actually like some things of the init daemon part of systemd.
But systemd is not only an init daemon anymore and the systemd developers work actively to cut off everything that is not systemd (see the udev saga as an example) or make idiotic decisions (NTP server defaults?). And that, in addition to some other things (why should udev be part of systemd? Why should be there an arbitrary limitation in core files size? See also [1] for more) left me very worried. And I don't like journald at all, rsyslog does almost everything already.
And the way Fedora integrated kdbus before it was put into the mainline kernel? That's ridiculous, sorry.
The need to make another 'su' because the older one was not good enough in the mind of the developers?
> But systemd is not only an init daemon anymore and the systemd developers work actively to cut off everything that is not systemd (see the udev saga as an example)
> or make idiotic decisions (NTP server defaults?).
Love the excuse for that. "Systemd is not meant for end users, and distros should set their own servers anyways".
Its like anyone thats not a developers working for/on the large name distros are users (aka consumers, aka mooo) and should shut up and accept what the distros ship.
PSA: Beware people running Debian Unstable and possibly other rolling release distros, lightdm seems to fail with 227. Be ready to downgrade if needed.
I did enjoy that the mention of it being policy to avoid unnecessary abbreviations was immediately followed by two unnecessary abbreviations of the word "compatibility". I suspect that to have been deliberate. (-:
154 comments
[ 3.1 ms ] story [ 215 ms ] threadYou still write shell scripts, but you also still have control.
Again, I'm not entirely opposed to the ideas behind systemd, but I am opposed to basically every fucking Linux server distribution breaking because of things people are doing for the desktop, which began with upstart.
There is also no love left to lose between me and sysV init scripts.
The only difference is acceptance.
I mean, you could just use Heroku, for free, most likely, to host what you are doing. Should all UNIX become Heroku? What happens if your blog starts doing a billion page views a day? When one of the systemd processes is taking up 95% CPU, what do you do?
I'm not saying it's bad, I'm saying these are the type of concerns its' critics have. I am a fan of a lot of the ideas. I have downgraded several Linux systems which I needed To Work(tm) because of systemd being completely half-assed, and there are some great audits of its' code which make it appear to have been written by kindergarten students.
It's pretty obvious you're not actually acquainted with the problem domain. Keep using whatever makes you feel comfortable, though.
EDIT: Let me join in on the moaning about downvotes. Not that I understand why.
I've used minimal shell based systems on BSD, Solaris SMF, and launchd, as well as Windows services, and maintain the successor to the now-unmaintained Unix Rosetta Stone. However a minimal shell-script based init is not something anyone on Linux is seriously considering, so I'm not going to debate it with you - that would be a waste of both our time.
But please, feel free to tell me about my own experiences. Since you're new here, have you read https://news.ycombinator.com/newsguidelines.html yet?
Have you taken the time to examine how a Linux system that uses a replacement for sysvrc written in the last ten years actually handles startup? Or what services people who aren't running a RedHat distro actually run on their Linux servers?
Or have you simply either taken the systemd folks at their word about the "sorry state" of sysvrc replacements, or -perhaps- glanced at a >10 line startup script and recoiled in horror?
2. Yes. Folks who aren't running 'RedHat' (sic) use systemd, sysvinit, or pretend to use upstart but as a thin wrapper to sysvinit.
3. No, I don't generally listen to Linux advocates as a way of evaluating something.
Good. Far too many people in these conversations have poorly-informed opinions.
> Folks who aren't running 'RedHat' (sic) use systemd, sysvinit, or ... upstart [sic] ...
These are init and RC systems, not services. :)
> ...or pretend to use upstart but as a thin wrapper to sysvinit.
I dunno. The Upstart project page says:
"Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.
It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init."
If I look at the file list for the upstart package, [1] I see that upstart provides its own /sbin/init, /sbin/halt, and -it seems- just about everything that's provided by the sys-apps/sysvinit package on my Gentoo system.
Also, sysvinit and sysvrc aren't the same thing. OpenRC -the default Gentoo RC system- is a sysvrc replacement. [2] It relies on sysvinit (or similar) to get its work done, and is an obvious improvement over sysvrc.
If you were aware of the distinction, and have the time, would you be so kind as to mention the top N worst things about sysvinit? If you weren't aware, and have the time, would you care to learn the distinction, and then bounce on back and answer the query?
Thanks in advance. :)
[0] http://upstart.ubuntu.com/
[1] https://packages.debian.org/jessie/amd64/upstart/filelist
[2] If I'm reading this right, Debian even maintains a sysvrc package: https://packages.debian.org/jessie/sysv-rc
Re further requests for info, I don't really want to continue the conversation, because you should have apologised by this point and haven't. Don't presume to know things about other people, it's rude and looks poor when you're wrong.
That would be why I started my initial comment with questions, rather than presuming that I knew your background and knowledge and launching into poorly-formed and ill-founded accusation and/or assertion. :)
I'm trying to have a serious, level-headed discussion about a topic that is -inexplicably- sensitive for a few groups of people. In order to do so, I need to discover the pertinent information about the background and knowledge of my conversation partners.
> If you actually meant services, then what's your point?
I did. That would be why I asked about services rather than RC systems. :) I know that many people don't work this way, but if you can try to keep in mind [0] that I try hard to say what I mean, and leave nothing but white noise in between the lines of my statements.
Anyway, my point is this. You said:
> I'd take systemd over sysv init, atd, crond, three or four watcher daemons, syslog, various other semi-maintained tools that replicate each other's functionality...
which I took to be a little strange. It seems like this is your understanding of the typical set of services a Linux system runs, but it doesn't jive with how either I or my handful of professional sysadmin friends run our systems. We generally don't run atd, leave any FS watching up to the software package that actually needs it, avoid using tools that duplicate each other's functionality like the plague, and tend to not use unmaintained software... unless it's clear that that software is "done".
I've known several friends and acquaintances who have gone off to work for Red Hat. They've -to a man- come back with some really strange, very hard-line positions about how Linux systems are "supposed" to work. One of them vehemently insisted that -because of the complexity of dependency management and testing- a highly-flexible, rolling-release distro like Gentoo Linux was impossible. [1] Because of my personal experience with how Red Hat seems to have enweirdened some of my friends and acquaintances, and because I knew next to nothing about your background, I wanted to ensure that you had indeed stepped out of the Red Hat bubble and seriously looked at what other companies and distros consider a typical set of services and the features provided by and operation of their sysvrc replacements.
[0] I won't fault you if you can't. I have quite a difficult time remembering what I had for breakfast, let alone the rhetorical styles of the couple-dozen one-off Internet conversation partners that I've had over the past several months. :)
[1] Note that his position was not "Only a fool would attempt to provide guaranteed enterprise support for any and all arbitrary Gentoo Linux configurations because of $COMPLEXITY."! I can't argue with that position. I asked him if perhaps he misspoke and that maybe this or something similar was closer to his actual position. He vehemently denied that it was, and doubled-down on his original statement.
I ask basic questions because I often have conversation partners who have not done their basic research. In the past, I've wasted a lot of time having mid-to-advanced level discussions about topics with people who -as it turns out- didn't have even the most rudimentary grasp of the fundamentals of the topic. I ask brief clarifying questions of this sort out of respect for the busy schedules of myself and my conversation partners.
More than that, I've found that presuming to know something about other people often leads to hurt feelings and wildly incorrect assumptions. :)
From the start, I've been behaving in exactly the way you claim that you want me to behave. I even explained -at length- the reasoning behind my first and second comments and my conversational MO.
After your reply to my first comment, you've failed to meaningfully engage in any further discussion, citing my failure to behave in the way that you want me to behave as the reason for your non-participation.
We're getting really far afield here, but do you make it a policy to apologise to people (who aren't paying you) who continue to fail to understand something long after you've made that thing clear to everyone else in the room?
These are all flaws in systemd with just one of those daemons you mentioned. Were I to get into init, I'd ask you about BSD-style rc files, which are tiny and very readable, as well as being shell scripts, so you can run more pre-execution tests to ensure your job will run correctly. With BSD atd and crond, I'd discuss how atd functionality has been merged in as a cron job. There are a lot of other solutions to sysv init than systemd. Unfortunately, the Linux monoculture sees bash as the only shell, sysv init as the only pre-systemd init, and the poorly configured rsyslogd distros ship with as the only syslog.
People like to make fun of Gentoo, but it does many things right. ;) OpenRC and netifrc are sane and flexible. syslog-ng is the "default" syslog and comes with a reasonable default config file. Use of logrotate is strongly recommended, etc, etc, etc.
As someone who has an on-going project to replace 157 rc.d scripts from FreeBSD 9/10, I can report that the generalizations about the Mewburn rc.d system that are often made, in order to contrast it with the egregious mess that one can find in the System V rc clone on Linux operating systems, are not in fact strictly accurate.
Yes, most of the rc.d scripts are small. Yes, the Mewburn rc.d system did almost 16 years ago what people are erroneously characterizing as novel to systemd: namely taking all of the common management code out of the individual service definitions, leaving just a few parameterizations in some cases. Yes, there is a definite common coding style that is a breath of fresh air when coming from the Linux world.
But no, the scripts are not universally tiny. One problematic example currently of interest is /etc/rc.d/bluetooth, weighing in at over 360 lines. Others include the subsystems that are split across multiple scripts, such as /etc/rc.d/atm1, /etc/rc.d/atm2, and /etc/rc.d/atm3.
Nor are they universally consistent, /etc/rc.d/serial being an example of one with quite different configuration semantics to the key=value-in-rc.conf{,.local} norm.
And whilst one might rail against the problems of having a common script for SuSE Linux, Debian Linux, and RedHat Linux, all of which have different sets of init.d script helper tools (resulting in "portable" init.d scripts ending up being a case "$distribution" in...esac statement for pretty much every step), one can equally point to the fact that OpenBSD's rc.d system (https://news.ycombinator.com/item?id=8506460) has entirely different function and parameter names to the NetBSD one (as adopted by FreeBSD/PC-BSD and DragonFly BSD) as well as different rules about rc.conf and different variable naming conventions.
Be very careful about attempting to claim some kind of moral high ground on interoperability and standards. (-:
Also be very careful about notions that the Linux world sees the Bourne Again shell as the only shell, some rather silly statements by some people who should know better notwithstanding. The adoption of the (Debian-modified) Almquist shell as /bin/sh by Debian and Ubuntu, and its significant effect on bootstrap speed, is quite famous. And my experience is that the Z Shell is quite popular, for starters.
Really though, the primary reason to cite rc.d is to debunk the pants-on-head claim of shell script-based rc being intrinsically spaghetti and voodoo.
Generally it's fairly easy to make a system user for an application if no special accesses are required - just make a custom system user for the application, and ensure it can access the various things it needs to touch (eg: logging).
A good question to ask yourself: "Presume that my daemon is compromised by an attacker ...
* ... what processes can it trace, debug, read/write the memory spaces and environments of, or send signals to?"
* ... what files, devices, and directories can it read and write?"
* ... what files, devices, and directories does it own, and can thus grant access to?"
Then ask: "How does the answer change if instead of all of my daemons running as the same (even if unprivileged) user they each run as different users, some (where appropriate) having no write access to anything and owning no files nor directories at all?"
It's not even the number of lines that matters -- because start-stop-daemon could probably accomplish all of that in "1" gigantic line.
It's the fact that it's incredibly readable. Someone with only a passing knowledge of UNIX and no knowledge of systemd could understand almost everything in this file at a glance.
In actual fact, the multi-user boot versus single-user boot notion goes back to pretty much the beginning (https://news.ycombinator.com/item?id=10207674), and the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism. It's been a Solaris SMF milestone name for over a decade. It was a concept used in the 4.1 Berkeley Distribution manual for init(8) in 1981.
Sure, but I had to do a bit of reading to understand that
meant: "This service should start in the runlevel called 'multi-user'".It also seems... backwards to put runlevel membership in a service definition file. Runlevels are an administratively controlled thing. Does systemd have two or more places a given service's runlevel membership can be specified, or is the only way to change a service's runlevel in its service definition file?
Yes, I know that the notion of "runlevels" is probably "legacy". However, assume that -on my OpenRC system- I have two particular runlevels, one for regular multi-user operation, and one for operation when I connect to hostile networks. Transitioning to this second runlevel shuts down all services that listen on anything other than localhost. Transitioning back brings those services back up.
Although this situation is a little contrived, this sort of flexibility is valuable.
What systemd actually does at startup, or when starting services individually, is look for symlinks. If I have a foo.service and a bar.service, and a symlink called foo.service.d/bar.service -> ../bar.service, then systemd will start bar.service automatically whenever I ask it to start foo.service. If it also sees a multi-user.wants.d/foo.service -> ../foo.service, then it will start foo (and also bar) whenever it enters the multi-user "runlevel". (This is almost exactly how sysv runlevels work, except that they're named rather than numbered, and they can depend on each other.)
I can make that symlink manually, if I want, or I can call "systemctl enable foo.service" in which case systemd will read the dependency information from the .service files and make the appropriate symlinks.
If I don't like how the distro or packager has set up the service file, I can override it. The service files live in /etc/systemd/system by default, but systemd also looks in /var/systemd/system for overrides. This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.
Putting the overrides in a separate directory is a pretty nice decision; it's always obvious what changes have been made, and upgrades never overwrite them. I can even put those overrides into version control and track changes to them over time.
Is this true of the identically-named keys in the Unit section?
> If I don't like how the distro or packager has set up the service file, I can override it. ... This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.
You should look at how OpenRC handles service dependency specification. It's at least as flexible as the system you describe in systemd. :)
> Putting the overrides in a separate directory is a pretty nice decision...
Yep. At dependency graph generation time, OpenRC examines its master config file at /etc/rc.conf as well as config files in /etc/conf.d/ for -among other things- additions to the need, use, before, &etc declarations in the depends section of a given service's init script. To augment -say- the "use" list in the openvpn service, either add the line
to /etc/conf.d/openvpn , or add the line to the OpenRC master config file at /etc/rc.conf . If I -say- wanted to override OpenVPN's default dependency on the "net" virtual service, I would add to /etc/conf.d/openvpn (or perform a similar transformation as before and store in /etc/rc.conf).> I can make that symlink manually, if I want, or I can call...
To add the openvpn service to the "custom_init" and "default" runlevels, I would simply do
If I omit the runlevel, the service is added to the current runlevel. I could screw around with symlinks if I cared to, but why would I? :)The OpenRC configuration processing system is substantially more powerful than I describe, handles the processing of OpenRC-standard config files [0] automatically, and lets you shift the complexity of a configurable init script that should be in a config file into a config file.
Because my overrides and augmentations to the distro-supplied init scripts are in text files stored in standard places, I -too- can easily put them in version control and track their changes over time.
[0] Which -shockingly enough- look exactly like what any Linux programmer would expect a non-XML non-JSON text config file to look like. ;)
Note the @ in the name, that means I reuse it as webapp@api.domain.name webapp@frontend.domain.name and so on.
----8<---- webapp@.service ---->8---- [Unit] Description=%i webapp
---- 8< ----/srv/webapp is a simple script that will do "exec $1/webapp" which is the actual executable.
I'm not trying to start a fight.
Why do you call out to a script, rather than having systemd make the call on its own?
systemd does not permit variable expansions or unit parameters in the first word of ExecStart.
Why do the thing Spidler described in his comment, rather than the thing I described in my comment?
Reliable per-service logging was largely perfected by multilog.
Judging complexity by observing external configuration is nonetheless exceptionally naive. One must factor in how the key-value pairs are built into an internal representation and dispatched, which is where things become an issue. Non-deterministic boot order (man 7 bootup) is one of many undesirable trade-offs.
Yeah, also something I found annoying with Upstart's induction into Ubuntu.
A predecessor at a place I worked at with a huge amount of traffic installed Ubuntu 9 on all of our memcache servers, and we found on a handful of other machines installed with Ubuntu 9 that non-deterministic boot order would randomly cause them to fail booting, and it was a coin-flip whether that would be a one-off problem or a forever problem requiring a reinstall.
When you have 4 8GB machines (this was some years ago) powering memcache for a billion requests a day you don't really want a power cycle to cause 25% cache misses for hours as you wrestle with hardware and boot config. :/
Actually the main problem is that a lot of modern systems do not make their dependency graphs well observable, but instead treat them as in-memory ephemera that are constantly recalculated and cannot e.g. be written to persistent storage as known-good checkpoints.
This is one area where s6-rc gets it right. You have to compile a manifest of your service dependencies and consistently boot from that well-known state, so for any changes it must be regenerated.
serel was an earlier example of a service dependency system and parallel launcher which directly had you specify the nodes and edges of a directed graph as an RDF manifest. Clever.
Try having a dozen different types of config-test for different programs, there is no way to extend ala ExecConfigTest
Folks gush about how concise systemd unit files are.
For services that require no complicated pre-start checks or configuration, equivalent service start scripts are more concise and just as legible in either OpenRC or -AIUI- Upstart. (Look at my conversion of a simple systemd unit file [0] to an OpenRC init script. [1])
What's more, for jobs that need complicated pre-start logic of one kind or another -AIUI- systemd provides no way to bundle that logic with your service description file. You have to put that logic in another script file which is then called by systemd.
EDIT: I can't speak to Upstart, but even if your service has dependencies or needs to be started before or after some other service or class of service, [2] specifying them is similarly concise and legible. Add
somewhere near the top of your script. For most services, this adds three non-whitespace lines. :)[0] https://news.ycombinator.com/item?id=10355588
[1] https://news.ycombinator.com/item?id=10357010
[2] For example: The network start scripts provide the "net" service. You might have a service that -for some reason- needs to start before or after the "net" service, but doesn't actually depend on it to start. This is what "before" and "after" are used for.
I just wish systemd itself had stayed there and that the additional functionalities it keeps developing lived in separate projects with well-defined interfaces.
Considering parallelism is about startup optimization, a much better option is to load a process from a checkpoint image, e.g. one taken by CRIU or DMTCP.
Almost completely unrelated: Two things that came out of the LKML examination of kdbus during the 4.1 merge request were:
1) kdbus makes assumptions that break CRIU and similar projects
2) The kdbus people didn't seem to care much about this fact
So... the "ultimate" systemd setup may very well preclude the use of process checkpointers to speed up startup. :(
I tried to go back through the 4.1-rc1 kdbus merge request saga, but Gmane is so slow as to be unusable today and the only other lkml archive I can find that shows me the whole discussion [0] doesn't appear to do threads. [1]
A search for "CRIU" reveals Lutomirski mentioning that a particular bit of info passed around by kdbus ("starttime") would break CRIU. Greg K-H says "'starttime' has been removed for a while now. You sure that you're looking at latest?" Lutomirski says something to the effect of: "No, I haven't taken a close look at latest, just checked to see if that thing that I thought was really bad was still in it, and it was."
Later, I see Lutomirski wondering if kdbus would interact with '/proc/<pid>/map_files' in such a way as to break CRIU, but Michal Hocko says something to the effect of "Eh, it looks like it shouldn't, but I can't know for sure.".
Because I don't have the thread structure, I can't be sure that there weren't replies to any of those messages from folks who felt that there was a problem for CRIU. However, as it stands now, it's pretty safe to say that I fabricated a bucket of mud and threw it around.
Sorry. Mea culpa. :(
[0] http://www.gossamer-threads.com/lists/engine?do=post_view_pr... (Print dialog warning)
[1] I've just gotta ask: am I an idiot, or is Google Groups's mailing list search feature absolutely terrible? I ask it to give me all instances of "CRIU" in a particular LKML thread. It gives me a pointer to the start of that thread, leaving me to go through messages ~20 at a time to find the messages I care about.
It's not in question that systemd is infinitely more complicated than equivalent sysv or other script-based init system. Complexity means attack surface.
Unless the systemd devs are immune to the same problems that plague literally every other software developer
But fine, you wanted numbers. First, systemd's LoC count is 699,342 across 1,912 files as of a git clone of their SCM[1] and gitstats five minutes ago.
That number was around 550,000 back in May[2], so kindly update your knowledge before regurgitating it mindlessly.
Debian's (sid) sysvinit package[3], by the same methodology, is 32,151 lines across 235 files.
Systemd is therefore about 22x (21.75179621162639) more complicated than sysv.
I'd go run the numbers for Upstart, but I'm not going to go download bzr and appropriate profiling tool to conclusively prove something that's common knowledge to anyone not in an internet argument.
And kindly stop complaining about downvotes - it's against the site rules.
[1]:http://anongit.freedesktop.org/git/systemd/systemd.git
[2]:http://www.phoronix.com/scan.php?page=news_item&px=MTY5NjM
[3]:https://anonscm.debian.org/git/collab-maint/sysvinit.git
Single out the init management portion of systemd (and journald since it's a dependency), and compare those numbers.
Everything else is an optional module.
The init system should be an init system and nothing else, IMO, so targeting all that extra complexity (which was my original point) is fair game.
Whether a distribution chooses to include timesyncd or another ntpd doesn't magically make timesyncd an init system.
If sysvinit also included an ntpd client, _then_ it would be appropriate to include that module as well for a fair comparison of SLOC.
Another example would be to take vim, counting the SLOC including a bunch of downloaded plugins, _then_ comparing that number against emacs without any plugins.
It's comparing apples to apples and oranges simply because they're in the same basket.
If your system gets pwned, does it matter to the user if it was via the "systemd" binary itself or by one of the official "systemd-somethingd" binaries (that have no other purpose other than interoperation with the main app, developed by the same people) that come with every OS using it? Had the distro maintainers not shoveled an extra ~670,000LoC down their throat, they wouldn't have been pwned.
If systemd wants to replace my freaking cron, DNS, ntp, and logging daemons, and the majority of in-production deployments use the application like that, I will be evaluating the application's complexity against how it's used in the real world.
sysvinit doesn't attempt to replace resolvconf, ntpd, crond, and syslog-ng. systemd does.
They have no access to systemd-init, only to an API presented by it (as does every other application if they so choose to use it), and thus this statement is incorrect. A bug in one of those applications cannot magically get access to systemd-init (which I'm assuming is what you meant by "main" here).
> If your system gets pwned, does it matter to the user if it was via the "systemd" binary itself or by one of the official "systemd-somethingd" binaries (that have no other purpose other than interoperation with the main app, developed by the same people) that come with every OS using it? Had the distro maintainers not shoveled an extra ~670,000LoC down their throat, they wouldn't have been pwned.
It absolutely matters! A bug in systemd-init can do much more damage than a bug in timesyncd could. Also, I really don't care about your anger towards an open source project, please refrain from expressing it to me if you'd like to continue this conversation.
> If systemd wants to replace my freaking cron, DNS, ntp, and logging daemons, and the majority of in-production deployments use the application like that, I will be evaluating the application's complexity against how it's used in the real world.
You may want to create a separate post somewhere which compares the SLOC of all the applications systemd attempts to re-create (choosing perhaps the most popular from each) along with the complete systemd package, but I personally wouldn't have any interest in it since it would be a factually correct comparison.
> sysvinit doesn't attempt to replace resolvconf, ntpd, crond, and syslog-ng. systemd does.
That's exactly right, but systemd-init doesn't attempt to replace those either (besides syslog-ng (journald), which I already mentioned as being fair to include with the SLOC count)*.
https://news.ycombinator.com/item?id=9517040
https://news.ycombinator.com/newsguidelines.html
Anything that requires DBUS is (at minimum, because it now depends on DBUS, which is -itself- rather complicated) substantially more complicated than a script that does
Surely you can't argue with that?It may not appear to be more complex to the user, but when your software has a firm dependency on a complex system, it becomes more complex because it is affected by the failure modes, quirks, and restrictions of the software on which it depends. [0]
> Lots of init scripts used to be quite different across distributions and very long.
I've talked elsewhere in these threads about how systemd's unit files simply shift the complexity for services that need (for whatever reason) complex pre-start logic out of the unit file and into hand-maintained tools. See [1] for my most recent comment on the topic. Make sure to check the footnotes in that comment, too. :)
[0] Entirely valid anecdotal evidence on the relative stability of dbus doesn't invalidate my point. dbus is one of many complex software projects that others depend on. :)
[1] https://news.ycombinator.com/item?id=10357379
Here's a couple of fun problems from years past.
Docs say NoNewPrivileges as yes "prohibits UID changes of any kind", but that's not exactly true.
It's spammed so much debug info the system fails to boot.
"systemd, when updating file permissions, allows local users to change the permissions and SELinux security contexts for arbitrary files via a symlink attack on unspecified files."
"The rm_rf_children function in util.c in the systemd-logind login manager in systemd before 44, when logging out, allows local users to delete arbitrary files via a symlink attack on unspecified files, related to "particular records related with user session."
It's complex software running on many different kernels, and many different distributions, doing many different things with lots of code written in C. There were over 80 people mentioned as contributing to the last release. Of course there are going to be problems!
I hereby coin Vacri's Law: When a comment-maker announces incoming downvoting, they're aware their argument is weak and is instead trying deflect criticism through shaming others.
In any case, you're being a hypocrite. From your very first comment in this thread ("clearly done a full audit") you've been snarky, and now you're crying foul that people are being snarky back. I actually laughed out loud when you invoked the "good faith" HN guideline against Karunamon.
Let's see he can take over cron and make it 10 times more complicated... what else is left...
Me? I've gone to freeBSD for personal things; until Apple successfully pushes launchd upstream..
If you want to see how far spread systemd is actually becoming install arch linux using the official guides.. it's very easy to set up- but you definitely can see how it's "everything that isn't linux that your computer needs to be functional"- I'm sure they'll have a wayland fork in no time.
The systemd people have a Grub/Grub2/LILO replacement!?
Would you be so kind as to link to details? :)
It's (unsurprisingly) much simpler than GRUB2 (as is every other bootloader in past, present, and presumably the near future).
On the other hand, though desiring launchd is not insanity, I do consider it hasty and not well thought out, as you might know.
He could always go into hardware.
Pity the comments are largely the usual moronic and entirely non-technical nonsense that infests the topic at HN. Sub-reddit commentary quality.
I'd think I and others have been trying to inject some reasonable commentary. Any objections, or are we out on a political vendetta as is usual with systemd?
No more problems with overflowing logfiles, no more debugging undocumented shell scripts for booting, no more writing init scripts, no more debugging undocumented 200+ line init scripts, no more minute-long boot delays, no more brittle, half-assed distribution-specific networking solutions (also undocumented shell scripts), ...
For me as sysadmin, systemd solved a lot of long-standing problems, while so far introducing no new ones.
Not quite. The journal is prone to fragmentation, though it can be capped just as most sane loggers like multilog can.
no more debugging undocumented shell scripts for booting
no more writing init scripts
no more debugging undocumented 200+ line init scripts
Redundant complaints, and irrelevant. Not all script-based init systems are created equal, and many init systems throughout history have eschewed shell scripts [1].
no more minute-long boot delays
Incorrect. Boots can still stall from dependency loops, cyclical transactions, stuck start/stop jobs, an improperly calculated service ordering in the parallel dependency graph, etc.
no more brittle, half-assed distribution-specific networking solutions (also undocumented shell scripts)
Incorrect. networkd is tailored to specific purposes (largely containers) and not a general-purpose NetworkManager replacement, nor does it necessarily deprecate any ad-hoc network configuration scripts (which aren't even intrinsically bad).
[1] http://blog.darknedgy.net/technology/2015/09/05/0/
> The journal is prone to fragmentation, though it can be capped just as most sane loggers like multilog can.
So my choices are a) systemd + journald, which Just Works™ by always feeding units' stdout to it without any further configuration, b) syslog and logrotate and oh I have to patch software to use syslog in the first place and make sure logrotate rotates the files correctly if a daemon still its logfile open and… yeah, screw that, I've wasted enough of my life with that, c) something where I manually set up everything to use multilog.
> Redundant complaints, and irrelevant
From your ivory tower, maybe. Meanwhile I'm sitting in the trenches and banging my head against the wall because someone again managed to break their 300+ line init script (hi, MySQL!).
> Incorrect.
I did not mean failure cases. The average case, where every service starts up in a correct order on sysvinit/systemd, went from ~60 seconds to <20. Now, a reboot is usually faster than the failover timeout.
And unlike with Debian's "dependency-based sysvinit", systemd works. I've yet to see a failure case that wasn't caused by either hardware failure or gross operator error (e.g. referencing /var wrong in the fstab).
> Incorrect.
Well, it looks I have to throw away my perfectly working networkd setups then, if you say so.
> NetworkManager
I wasn't even talking about that. I was mainly referring to Debian's networking scripts (which have been broken and unfixed for half a decade now) and Arch's netcfg/netctl which all three never worked as advertised. If NetworkManager was actually usable on headless devices, I might prefer it over networkd, too (assuming I find a use case where either it or connman are insufficient).
No, but those of us who do generally get downvoted for mentioning it in these comment threads, so many just stay away.
Paradoxically, due to the positive feedback loop of people downvoting comments that they disagree with and thereby driving away dissenting voices, comment threads on polarizing topics like these tend to be a pretty poor representation of actual popular opinion, as opposed to the opinion of the loudest people in the room.
Those are seldom talked about, but are some of the primary reasons I object to systemd. Are you neutral on those aspects, or do you see them as definitively positive?
That in turn invoke the old chestnut about how 99% of MS Office users use 1% of its features, but each use a different 1%.
It's good to first be charitable when attempting to divine a conversation partner's motives. If you go back and read more of what vezzy-fnord has said in the past, you'll find that he doesn't really rely on innuendo, and has no qualms with addressing false, misleading, or incorrect statements head-on. So, when he says:
> But by and large most systemd proponents seem to scantly have any idea why they like it beyond parroting misconceptions and half-truths, much as many critics aren't sure why they hate it.
he is stating the entirety of his observations, rather than slyly implying that his conversation partner doesn't know what he's talking about.
I think systemd has the misfortune/honor of touching decades-old subsystem interfaces. Any time anyone ever makes a change to something that's worked the same forever faces criticism. systemd's feature set is so attractive that it's getting/gotten adopted by the vast majority share of linux distros. The folks who didn't prefer it but thought that they could ignore it may have been surprised and vocal when it showed up in their favorite distro.
>undocumented 200+ line init scripts ... undocumented shell scripts
Maybe we've just traded those for undocumented C init programs?
Note: I KNOW that documentation is seriously hard. But, one of the rebuttals of the "Systemd is complex and monolithic" assertion is "Systemd is fully documented, so you can replace any part of it with one that you've authored.". You can't honestly make that rebuttal if the state of your documentation doesn't support your claim.
Oh, you summer child.
In any complex software project, there's always a difference between the contract described by the system's specification and the contract described by the software as implemented.
The big difference between the explicit contract of the documentation and implicit contract of the code is that -in areas on which the documentation is silent- the contract described by the code is subject to change at any time, without warning. Raymond Chen [0] has spilled much ink over customers who have gotten badly burned by changes in the implicit contract some time after they made use of something that the code permitted, but the documentation did not describe as possible and/or supported.
[0] Microsoft most notable backwards-compatibility guru
That's exactly the reason why source itself is better than implementation documentation - because it is always "current".
http://thecodelesscode.com/case/16
http://thecodelesscode.com/case/69
If user behaviour doesn't change - why changes in implementation are wrong? Reimplementations can use the source code at one place in time and use that as a reference.
What I absolutely LOVE is a much more robust ordering of system service turn ons. Start network BEFORE ntpd. Start rpcbind BEFORE nfs-server. A lot of hack solutions for this chicken & egg problem in the past were always very fragile.
The point is that, if you're thinking of it as a linear ordered sequence, you have the wrong mental model. On the other hand, systems like OpenRC do work that way.
Are you sure that OpenRC works in this way?
There may be some subtlety that I'm missing, but the documentation for systemd unit files [0] seems to describe a service dependency and ordering specification system that looks equivalent to what OpenRC provides with "need", "use", "before", and "after". (Search for "[Unit] Section Options" and read on from there.)
[0] http://www.freedesktop.org/software/systemd/man/systemd.unit...
Simplifying things, your unit file actually becomes a Unit object which queues a job (which is itself a Unit) which is scheduled and checked for consistency in a transaction, which encapsulates a service (itself a Unit and one that controls jobs). The final graph of dependency information largely does not reflect what you configured.
Ah. That would be the "subtlety" that I was missing. Thanks! :)
As an aside: how terrible it is to provide an interface that makes it look it gives you ordering guarantees when -in actuality- it gives you no such thing! :(
You're mistaking people's dislike for systemd with a love of sysv init - no-one liked sysv init and everyone wanted a change. It's just that systemd has taken a very broad remit, plus the maintainers have a 'fuck your use-case' attitude.
Logrotate solves this for me, for every logging method. How did it fail for you?
> ...no more minute-long boot delays...
I don't have this issue with OpenRC on Gentoo. shrug I do know that Kubuntu 15.04's boot time is irregularly slow, and that its shutdown process -every once in a while- hangs forever, requiring the use of SysRq keys. This didn't happen with the previous, Upstart-based version of Kubuntu. (Then again, there are many things wrong with Kubuntu 15.04, so this might not be entirely systemd's fault. ;) )
> ...no more debugging undocumented shell scripts for booting...
Scripts written for OpenRC are typically easy to read. Complex tasks -necessarily- require complex scripts, but you'd have to write and call out to those scripts with systemd, too!
> ...no more writing init scripts...
For services whose startup is as simple as what systemd handles out of the box, I can write a one-to-seven-line startup script for OpenRC and call it a day. Additionally every service packaged in my package manager has come with a init script. [0]
> ...no more brittle, half-assed distribution-specific networking solutions...
shrug Gentoo uses netifrc. It does appear that this is a distro-specific way to bring up the network, [1] but I assure you that it's full-assed, has a well documented and flexible config file, and is not brittle.
[0] Some of them -like the network startup scripts, and the OpenVPN startup scripts- let you make a symlink to the original script, which changes the behavior of the script.
In the case of the network startup scripts, this makes that script change the state of a particular network interface, virtual or otherwise.
In the case of OpenVPN's script, this causes the startup script to look for config files with a different name. In the OpenVPN case, you symlink /etc/init.d/openvpn to /etc/init.d/openvpn.newconfig . The symlinked openvpn startup script then uses the OpenVPN config file /etc/openvpn/newconfig.config to start the "openvpn.newconfig" service. It's really handy!
How would you do this in systemd? Make a new unit file?
[1] Only because -like all distro-specific things-, noone else has bothered to adopt it. ;)
systemd start openvpn@<config_name>
[0] http://www.freedesktop.org/software/systemd/man/systemd.unit...
Most arguments against sysyemd appear to be appeals to tradition, not based on any technical deficiency. In actual use I have found no fault with it and it does make some things easier.
The only argument against sysyemd that I find somewhat sympathetic is the fact that there is always a cost to pay when migrating to and learning about something new. This is certainly true. However, I find the user experience of systemd to be superior, and this small upfront price seems worth paying.
But systemd is not only an init daemon anymore and the systemd developers work actively to cut off everything that is not systemd (see the udev saga as an example) or make idiotic decisions (NTP server defaults?). And that, in addition to some other things (why should udev be part of systemd? Why should be there an arbitrary limitation in core files size? See also [1] for more) left me very worried. And I don't like journald at all, rsyslog does almost everything already.
And the way Fedora integrated kdbus before it was put into the mainline kernel? That's ridiculous, sorry.
The need to make another 'su' because the older one was not good enough in the mind of the developers?
And now 'systemctl is-system-running'? Wat?
1. http://suckless.org/sucks/systemd
http://www.landley.net/notes-2015.html#05-07-2015 is a lovely entry from Rob Landley about trying to maintain an alternative to udev, and getting the runaround by GregKH and Kay Sievers.
> or make idiotic decisions (NTP server defaults?).
Love the excuse for that. "Systemd is not meant for end users, and distros should set their own servers anyways".
Its like anyone thats not a developers working for/on the large name distros are users (aka consumers, aka mooo) and should shut up and accept what the distros ship.