We have a similar problem when deploying Openstack using puppet - all the services need to be brought up after config that sets things like the central DB and AMQP locations. It means that a deployed site will always have errors in their logs right from the start, which confuses customers and makes it harder to spot when there is actually an error in the deployment tool.
If you're using Ubuntu, the .deb packages need a lot of work. I've been playing around with this for the past week and there are all sorts of little glitches and gotchas. A big problem is that the package maintainers didn't use debconf, so you end up having to install, set some settings and then bring the service up.
This is definitely not the debian/ubuntu way of doing things.
If I find some time I'll see if I can repackage things to properly use debconf and push things upstream.
I'm not quite sure I'm parsing your sentence correctly. If you're preseeding with debconf the value is that you got to preseed the correct configuration options. This means the debs will work with anything, not just the flavour of the week devops tool. Juju, Puppet and Chef aren't the only games in town. I'll agree that they are all very useful tools, but you shouldn't need to write complex scripts/manifests/whatever around them to stand up services. That should happen out of the box without requiring anyone to jump through 50 hoops. It also means that you can scale from 1 instance all the way up to 5000 instances which is a lot better than optimizing for the 5000 instance case.
I remember installing Red Hat 4, going on IRC and getting rooted. Red Hat's approach comes from lots of learning, and I think it's smart. Thanks for the post.
Years ago, Linux distributions defaulted to installing everything and the kitchen sink and enabling all those services automatically.
After years of "OOTB pwnage", they started to make things a bit more secure by default and they eventually got to the point we're at today (where most installations are relatively secure by default, OOTB).
(Side note: I believe it was RHL 5.2 that shipped with wu-ftpd enabled out of the box. By the time it shipped (or shortly afterwards, I forget which), there was a remote root exploit for wu-ftpd. Thus, if you installed RHL 5.2, you'd likely get pwned in short order.)
This is exactly correct. I was a Slackware zealot for a long time but frequently tried Red Hat because my boss back then was a big fan. I didn't actually convert to appreciating Red Hat until I worked there and learned the advantages of using RPM and other tools included in Red Hat that were absent in Slackware. In either scenario, and this is still true today, if you plan to connect your system to the Internet, you need to know your configuration and care for the system after it is live.
Modern distros help with this a lot by offering rapid package updates, automatic updating and smarter firewall and security measures. A lot of this stuff was controlled by very esoteric text configuration files in the past, which made it hard even for a savvy person to get right every time.
When packaging a daemon for Debian, the maintainer should add sufficient questions during installation, and sane default values, to start the service.
Contrary to the article, I think this is a rather good policy. It makes sure packages are usable as they are, rather than the normal process of config, make, make-install, and last adding user configuring before anything will start up. Debian is not Gentoo.
For those daemons which this isn't possible, package maintainers has a harder time. One of the better ways is simply to have it run, but in a safe way which do nothing. That would solve the issues the article describes, and would not be harmful.
What does packages using non-standard hacks that break puppet scripting have to do with configuring and building source code? That's not related to the issue at all...
The article talks about two issues. One is that debian has policy to default run daemons after installation. The second issue the article brought it up is the issues with puppet.
Regarding the first issue, sane defaults and usable packages directly after installation came as a result of the config, make, and make-install process. Users should get a useful package directly without needing to spend days adding all the default settings to start a simple mail or web daemon.
The second issue has nothing to do with config, make, and make-install. It has to do with non-standard hacks. If nagios3 had simply started up like apache, this would have been a non-issue.
> One of the better ways is simply to have it run, but in a safe way which do nothing. That would solve the issues the article describes, and would not be harmful.
I'd argue it would be worse. I'd rather have the program not run at all than have it preconfigured in a way that contradicts its regular operation and having to reconfigure it to make it work normally.
If you have to configure it in such way that it can't be done through sane defaults and install questions, then there are no normal operation. Any configuration is going to be unique to that installation.
As such, there is no sane configuration that contradicts its "regular" operation, if there is no regular operation to begin with.
Apache is a typical example here, as default installation provide no personal website, no web frame work, no kind of web site to speak of. Default here, is to do basically nothing and wait for the user to configure and install.
I posted this as a comment on the blog, but I'm duplicating here:
RHEL and Debian (and their respective derivatives) have different philosophies on packaging, which is why you see this result.
The RHEL philosophy is simple; package installation only installs the package, starting daemons is a separate process.
The Debian philosophy is that installing a daemon should also make that daemon run, unless it's not possible the make the daemon run securely by default. Things like the /etc/default RUN="no" aren't the package maintainer trying to be clever, but rather them following Debian packaging guidelines. There also the reason you see things like MySQL listening only on the loopback interface by default on Debian.
One you understand this the decisions seem a bit less crazy.
For me, the big issue with the Debian approach isn't on installation, but rather on upgrade. On Debian, running 'apt-get upgrade' on a live server can cause an unspecified amount of downtime. If you're upgrading MySQL, MySQL will get restarted. If you're upgrading a library on which multiple daemons rely, they will also get restarted. yum upgrade on RHEL doesn't suffer from the same problem, and you can schedule the restart yourself.
Again, this is because of a philosophical difference. On Debian, the idea is that if you've asked to upgrade something, it should be upgraded. On RHEL, you can upgrade a package, but continue to run the old (potentially insecure version).
> If you're upgrading MySQL, MySQL will get restarted.
Or specifically, it will get stopped, your packages will spend 15 minutes updating (or longer if there's an interactive prompt mid way and you aren't paying attention), and then it will get started again.
I'm quite happy with the debian way of doing things, but I'd really like it if apt-get was smart enough to stop / upgrade / start individual services, rather than stop-all / upgrade-all / start-all...
It depends on the upgrade sequence; I've definitely seen the behavior of a service getting temporarily shut down during something like preinst upgrade, a bunch of packages being installed, and only at the end during configure them all being restarted.
Report a bug then, or check if this is really possible to restart the service after installation. In some cases that may be the only reasonable way to do it. For example services based on scripts/libraries which get dynamically reloaded on each request cannot be updated while they're running, because it would cause mismatch between the components. Killing them, updating all files and starting again is the only way to do it.
As you said - it's not a debian-specific thing. Some packages may do it if they explicitly require it (and I haven't seen one doing this in the official repos so far).
I did not and do not consider it a bug; it was just inconvenient; it was really really inconvenient once when I did an upgrade involving hundreds of packages.
If you have very large buffers for InnoDB to flush and safely write out to disk it can take a long time. Usually not 15 minutes but it can be several minutes.
I hate to say it, but this is just a rant by a RedHat/CentOS admin who has started using Debian/Ubuntu for the first time and just doesn't know why things are done differently there.
Author here. You're 99% correct. I'm historically an RHEL-based admin, who has started using Debian/Ubuntu for the first time (relatively, for about a year), but I totally get why these things are done. What I deeply disagree with is how they're done.
Why are there so many non-standard ways that a package can decide to not start? I really appreciate that the package maintainer is trying to do the right thing, but it's ridiculous that there isn't a way to do it - there's the Debian default and then there's whatever the packager thought up.
That's nuts, and needing to manage the same resource twice in puppet isn't a tragedy or big deal, but it's a symptom that something is wrong.
You keep saying things like "so many non-standard ways that a package can decide to not start" and "whatever the packager thought up". Can you provide examples of what you mean?
In your blog post you only mention one way of controlling startup- via an environment variable sourced from a file in /etc/defaults. This is the standardized, documented place in Debian for controlling the behavior of init scripts. Offhand, I can't think of any other ways I've seen packager maintainers controlling startup. I'd really need to see more evidence before agreeing that things are "ridiculous", "nuts", or "wrong".
The file is a standard, but the contents are not. For the most part, that makes sense, but it would be nice if there were a few standards there. Eg, DISABLE_ALL_SERVICES=true|false. :)
As it is, figuring out exactly how to do this has to be done for each service. It's tedious, IMO, though I understand the history of why this is the case.
Sure, I can see the value in making service configuration more standard. E.G. after getting over the learning curve I've found working with systemd more pleasant than SysVinit.
I disagree that this is currently really more of a problem on Debian/Ubuntu that it is on Red Hat. E.G. If I want to manage BIND via puppet on Red Hat, I'm going to have to care about /etc/sysconfig/named.
Maybe it would be a good idea to provide the reason why things are done differently. He seems happy to accept the Debian standards despite disagreeing with them. Help the guy (and the rest of us who has asked the same question) out.
What I don't understand is why every maintainer adds a (unneeded) layer for enable/disable.
If you want to run a service, you have a runlevel editor.
This is, chkconfig in RH based, and update-rc.d in debian based.
That is the tool for the job.
Nowadays it seems that everybody prefers to invent an additional switch, this is a shell boolean or string, in /etc/sysconfig (RH) or /etc/default (debian).
This has been a long-standing bug in Debian which, due to their glacial processes, will probably prevail eternally.
In theory it is possible to disable the post installation auto-start with a specially crafted 'policy-rc.d'. In practice you don't want to go there, as it tends to break in unexpected ways.
Every package maintainer needs to make some decisions when packaging software. Debian has very specific guidelines about packing, so if you don't like it may be Debian is the wrong distribution _for you_.
Recently a workmate installed nginx from the upstream repo and... surprise: nginx packages things different to Debian. You may like it more or less, but there are different solutions for the same problem.
Author of the article here. Thanks, and I agree with you. I have some disagreements with the way Ubuntu does things like this, and I'd prefer an RHEL clone (which, of course, has tradeoffs too).
That being said, I'm in the environment relatively recently and it's established, and I'm the one who has to learn to adapt since I don't see our infrastructure changing dramatically soon. Other than finally adding configuration management, that is.
Debian's thinking on this is that a user won't install a daemon unless he wants it to run, and that a package should get everything into an immediately useable state by default. Therefore, a daemon should be started automatically when a package is installed.
I agree with the objection: I think that times have moved on, and in reality today people use configuration management tools and almost always want to change the default configuration on a server daemon. Therefore, it doesn't make much sense to start it automatically.
As bashtoni says: this isn't a problem with package maintainers. It's a problem with Debian Policy.
A stepping stone to a full solution is to arrange a standard to configure the system not to start a daemon by default, and arrange for packages to follow that standard. policy-rc.d is just such a standard.
Debian always think they know better than the authors of the software they're distributing (the worst, but by no means only incident being their key generation bug). When I maintained a linux program I received twice as many bugs from Debian users as anything else, because Debian repackaged KDE in a way that broke its autoconf macros. And anyone who's ever tried to follow a tutorial for e.g. Tomcat will know how radically Debian rearranges its files.
IMO it's not worth maintaining Debian packages of one's software or supporting it on Debian. Let Debian have their own packager for it if they want to, and direct support requests to them.
Right, but the excess was all people with the same problem. I added a note to the homepage ("debian users: you need to install libblah-devel, configure will not warn you about this because of how debian packages KDE") but it didn't seem to make any difference.
It seems like a lot of these comments are missing the point by discussing virtues of various packaging systems and auto-start behaviors.
It seems to me that the author was saying that consistency is important - installed services are expected to have a consistent behavior on a given platform.
A lot of tooling is built based on this, because that behavior is defined by the respective systems.
When people force override of these behaviors, it does break things as well as cause frustration.
The article isn't a criticism of any one packaging system or autostart choice of a distribution, but rather of individual packagers who impose their own choices over those of the system they're packaging for.
I believe ElasticSearch will automatically connect to any nodes with the same cluster-name. The package installs it with the default cluster-name. Since it starts on install, you don't have a chance to change this name before it's done that...
Apache's Cassandra deb that a similar behaviour and has a greater impact than ES's super-hero node names. A newly started Cassandra node will absent a token declaration in its yaml file, auto-assign a token. This is never what you want in a cluster distribution. So the workaround is either to patch the deb to not autostart or let it start, stop it, and blow away the data directories. This allows cluster management tools (such as Priam) to control the token allocation first, and then re-start Cassandra.
This is effectively how Debian/Ubuntu work. Once you know it, you can work around it. That said, it would be nice to have packages that don't run their service.
42 comments
[ 2.5 ms ] story [ 108 ms ] threadThis is definitely not the debian/ubuntu way of doing things.
If I find some time I'll see if I can repackage things to properly use debconf and push things upstream.
If you are using debconf interactively, it simply doesn't scale - if you are preseeding - or using juju, puppet or chef - what value does debconf add?
Please don't assume that it is simply wrong, just because it doesn't do x or y.
The debconf templates will always be trying to play catchup, and the transition of options between components doesn't make a lot of sense.
It would be really good to know what issues you have identified, as we take quality very seriously in this area.
However, your help with packaging will be most appreciated. Feel free to give me a shout on IRC, and we'll go through it.
I'll definitely give you a shout on IRC.
After years of "OOTB pwnage", they started to make things a bit more secure by default and they eventually got to the point we're at today (where most installations are relatively secure by default, OOTB).
(Side note: I believe it was RHL 5.2 that shipped with wu-ftpd enabled out of the box. By the time it shipped (or shortly afterwards, I forget which), there was a remote root exploit for wu-ftpd. Thus, if you installed RHL 5.2, you'd likely get pwned in short order.)
Modern distros help with this a lot by offering rapid package updates, automatic updating and smarter firewall and security measures. A lot of this stuff was controlled by very esoteric text configuration files in the past, which made it hard even for a savvy person to get right every time.
Contrary to the article, I think this is a rather good policy. It makes sure packages are usable as they are, rather than the normal process of config, make, make-install, and last adding user configuring before anything will start up. Debian is not Gentoo.
For those daemons which this isn't possible, package maintainers has a harder time. One of the better ways is simply to have it run, but in a safe way which do nothing. That would solve the issues the article describes, and would not be harmful.
Regarding the first issue, sane defaults and usable packages directly after installation came as a result of the config, make, and make-install process. Users should get a useful package directly without needing to spend days adding all the default settings to start a simple mail or web daemon.
The second issue has nothing to do with config, make, and make-install. It has to do with non-standard hacks. If nagios3 had simply started up like apache, this would have been a non-issue.
I'd argue it would be worse. I'd rather have the program not run at all than have it preconfigured in a way that contradicts its regular operation and having to reconfigure it to make it work normally.
As such, there is no sane configuration that contradicts its "regular" operation, if there is no regular operation to begin with.
Apache is a typical example here, as default installation provide no personal website, no web frame work, no kind of web site to speak of. Default here, is to do basically nothing and wait for the user to configure and install.
RHEL and Debian (and their respective derivatives) have different philosophies on packaging, which is why you see this result.
The RHEL philosophy is simple; package installation only installs the package, starting daemons is a separate process.
The Debian philosophy is that installing a daemon should also make that daemon run, unless it's not possible the make the daemon run securely by default. Things like the /etc/default RUN="no" aren't the package maintainer trying to be clever, but rather them following Debian packaging guidelines. There also the reason you see things like MySQL listening only on the loopback interface by default on Debian.
One you understand this the decisions seem a bit less crazy.
For me, the big issue with the Debian approach isn't on installation, but rather on upgrade. On Debian, running 'apt-get upgrade' on a live server can cause an unspecified amount of downtime. If you're upgrading MySQL, MySQL will get restarted. If you're upgrading a library on which multiple daemons rely, they will also get restarted. yum upgrade on RHEL doesn't suffer from the same problem, and you can schedule the restart yourself.
Again, this is because of a philosophical difference. On Debian, the idea is that if you've asked to upgrade something, it should be upgraded. On RHEL, you can upgrade a package, but continue to run the old (potentially insecure version).
Or specifically, it will get stopped, your packages will spend 15 minutes updating (or longer if there's an interactive prompt mid way and you aren't paying attention), and then it will get started again.
I'm quite happy with the debian way of doing things, but I'd really like it if apt-get was smart enough to stop / upgrade / start individual services, rather than stop-all / upgrade-all / start-all...
As you said - it's not a debian-specific thing. Some packages may do it if they explicitly require it (and I haven't seen one doing this in the official repos so far).
You can examine those scripts with "rpm -q --scripts mysql-server".
http://serverfault.com/a/526123/76707
I'm a longtime Redhat/CentOS guy recently moved over to Ubuntu. It has definitely been a bit weird getting adjusted.
One counterpoint to the OP's rant? SSH.
Why are there so many non-standard ways that a package can decide to not start? I really appreciate that the package maintainer is trying to do the right thing, but it's ridiculous that there isn't a way to do it - there's the Debian default and then there's whatever the packager thought up.
That's nuts, and needing to manage the same resource twice in puppet isn't a tragedy or big deal, but it's a symptom that something is wrong.
In your blog post you only mention one way of controlling startup- via an environment variable sourced from a file in /etc/defaults. This is the standardized, documented place in Debian for controlling the behavior of init scripts. Offhand, I can't think of any other ways I've seen packager maintainers controlling startup. I'd really need to see more evidence before agreeing that things are "ridiculous", "nuts", or "wrong".
http://www.debian.org/doc/manuals/maint-guide/dother.en.html...
More examples here: http://serverfault.com/questions/526075/fixing-services-that...
As it is, figuring out exactly how to do this has to be done for each service. It's tedious, IMO, though I understand the history of why this is the case.
I disagree that this is currently really more of a problem on Debian/Ubuntu that it is on Red Hat. E.G. If I want to manage BIND via puppet on Red Hat, I'm going to have to care about /etc/sysconfig/named.
If you want to run a service, you have a runlevel editor.
This is, chkconfig in RH based, and update-rc.d in debian based.
That is the tool for the job.
Nowadays it seems that everybody prefers to invent an additional switch, this is a shell boolean or string, in /etc/sysconfig (RH) or /etc/default (debian).
Why ?
This has been a long-standing bug in Debian which, due to their glacial processes, will probably prevail eternally.
In theory it is possible to disable the post installation auto-start with a specially crafted 'policy-rc.d'. In practice you don't want to go there, as it tends to break in unexpected ways.
Recently a workmate installed nginx from the upstream repo and... surprise: nginx packages things different to Debian. You may like it more or less, but there are different solutions for the same problem.
That being said, I'm in the environment relatively recently and it's established, and I'm the one who has to learn to adapt since I don't see our infrastructure changing dramatically soon. Other than finally adding configuration management, that is.
I agree with the objection: I think that times have moved on, and in reality today people use configuration management tools and almost always want to change the default configuration on a server daemon. Therefore, it doesn't make much sense to start it automatically.
As bashtoni says: this isn't a problem with package maintainers. It's a problem with Debian Policy.
A stepping stone to a full solution is to arrange a standard to configure the system not to start a daemon by default, and arrange for packages to follow that standard. policy-rc.d is just such a standard.
IMO it's not worth maintaining Debian packages of one's software or supporting it on Debian. Let Debian have their own packager for it if they want to, and direct support requests to them.
It seems to me that the author was saying that consistency is important - installed services are expected to have a consistent behavior on a given platform.
A lot of tooling is built based on this, because that behavior is defined by the respective systems.
When people force override of these behaviors, it does break things as well as cause frustration.
The article isn't a criticism of any one packaging system or autostart choice of a distribution, but rather of individual packagers who impose their own choices over those of the system they're packaging for.
This is effectively how Debian/Ubuntu work. Once you know it, you can work around it. That said, it would be nice to have packages that don't run their service.