It's an aesthetic thing mainly, and a bit silly I'm sure, but dammit, I find it unsettling to have my Python 3 stuff running, and then the legacy Python 2, simply for the sake of supervisord.
Likewise, I don't need any Python on my Go-server, please.
That is sort of a misguided attitude. If you run Ubuntu, you run plenty of systems stuff on Python. Also Perl and bash and C. If the piece of software works and doesn't require you to do extra shit to set it up, who cares what it's written in?
I can see caring about performance. A Python process will be more memory hungry than something hand coded in C. But for the rest of it, who gives a crap if t is packaged well. And if it's not, it's probably not worth using it at all.
Supervisord doesn't respect pam, limits.conf, etc which is a deal breaker for most of the stuff we do. I'm not sure if immortal solves it, but I might look into it.
This appears to be aware that it is solving the same problem as daemontools / runit / nosh / s6... but doesn't seem to offer anything new or better.
And it presents a solved problem (some daemons fork themselves off and the parent exits, so tell them not to do that or use fghack) as a new thing that Immortal can solve.
Is there something actually new and interesting about this that I'm missing?
The "execution" seems better at first blush. The documentation is short and to the point, which means it's something which would lead me to use it, especially since I still haven't figured out how to easily deploy most of the others you mention.
The ability to have go produce cross platform binaries would ease a lot of deployment issues as well. Along with distributed logging, it's something I'll probably try using. Unless any of the above have advantages, primarily in the ease of use category.
Instead of using named pipes for managing the process, a Unix socket is been used, so if required this opens the possibility to manage via a web JSON API the services an example about how to use Nginx for this is here https://immortal.run/post/nginx/
The idea behind was to avoid a developer to deal with a DevOps/operator to just stop/restart services, something useful on staging environments.
For starting, stopping a process there is no need to unlink/link a file, this simplifies things a little bit, for example, https://immortal.run/post/ansible/ by only updating the file contents or "touching" the file it can be started or re-started.
When following a PID, signals can still be sent to the process, these things become handy while working with a lot of microservices.
The docs need to be improved, but any feedback, comments or even desired feature is more than welcome.
Here are some differences, from a cursory reading:
* There is no composable toolset, but rather one monolithic program that does only a small part of the job. envdir and setuidgid are -u and -e options to the supervise analogue that is that monolithic program, "immortal". All of functionality of the other daemontools tools, such as envuidgid and softlimit, let alone the likes of runit's chpst, is missing. Not only are there no composable tools for it, the monolithic program simply lacks their functionality entirely. One can extend this functionality with the composable toolsets, with tools to handle things from scheduling classes to control groups. The monolithic program lacks all of this.
* There is no equivalent of the service restart/finalization mechanisms of runit, perp, s6, or nosh; which permit taking various actions dependent from how a service program terminates.
* Nor is there an equivalent of the at-stop and at-start mechanisms of s6 or nosh.
* There is no service readiness mechanism, as s6 and systemd have.
* s6, nosh, and systemd all have the concept of "oneshot" services. immortal does not cater for them.
* The svscan analogue, named "immortaldir", lacks an equivalent to the "down" file mechanism from daemontools, runit, perp, s6, et al.. All services always come up at bootstrap. There is no enable/disable mechanism at all.
* The author hasn't borne in mind the design lessons from svscan, the reasons why it works as it does, such as only having symbolic links in the scan directory, not the actual service definitions themselves. There is a race condition between copying or saving the YAML file into its directory and immortaldir trying to open and read it, which can result in immortal reading a part-written configuration file. There is also the potential for hidden files to be recognized as services, something that svscan explicitly prohibits.
* The author hasn't borne in mind the design lessons from s6-svscan. immortaldir wakes up and polls every few seconds instead of being event-driven, which as M. Bercot notes is a consideration on systems that want to save power.
* immortal does not have logger services, as perp, s6, daemontools, daemontools-encore, freedt, nosh, and runit all do. Instead, it has only the options of the service manager itself writing the log file, or a non-supervised non-manageable logging command. It is thus not possible to control logging through service management.
* There is no equivalent to multilog's rotation scripts, filtering, multiple log directories, or automatic timestamping.
* The log rotation uses the old .1 .2 system. The author hasn't borne in mind the design lessons from multilog, the reasons why it did the things that it did. So it doesn't ensure that logs are safely written to disc at rotation time, for starters.
* The service management API is not compatible with daemontools, daemontools-encore, runit, and nosh; which all share a common binary-compatible subset (the daemontools service management API). This API is, rather, a WWW server. Where daemontools et al. send simple one-byte commands; this API uses JSON and the Internet Message format meaning that a simple command to send a signal is actually an entire HTTP/TCP transaction, with all of that parsing and marshalling going on, under the covers. This flies in the face of the "do not parse at runtime" dictum followed by the other toolsets. There is an entire HTTP parser and server running with superuser credentials inside the service manager here. Ironically, this is a criticism that people vehemently (but erroneously) level at systemd. This service manager actually does do this, though.
* It also contains an entire YAML parser.
* There is no service interdependency mechanism; at all. There aren't even the tools for the old "thundering herd" approach of starting all services and having them manually check their depe...
Many thanks for your time on this extended feedback, is very nice to read and know more about other supervisors and what is expected to accomplish with this kind of tools.
I apologize in advance because unfortunately, the current documentation of the project don't cover all cases and because of that, some assumptions can be made until going deep into the code or testing, I will try to clarify some point hoping I could transmit a better idea of what the goal was and how is working, so pease bear with me.
Same with the enable/disable that in immortal is "start/stop" or also -x that is "exit"
You indeed can send a signal STOP, mainly because of this when using [immortalctl](https://immortal.run/post/immortalctl/) the option "stop" is used instead of down:
When following a pid, you indeed can still send signals to the "adopted PID" if we may call it that way.
Now immortal is totally capable of knowing the process ID of the daemon, but the methods you described as for reading PIDS from a file and checking if process are up and running, does methods are mainly used within the `-f PID` follow PID option in where you want to supervise or know status about a PID that the supervisor didn't create and, for this in BSD systems Kqueue is used instead of long polling https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
I have some animations here http://immortal.run/about/ maybe they are not very clear but hope they could clarify this.
In case you know a better way of knowing if a process exists, something like asked here: http://stackoverflow.com/a/15210305/1135424, I will be more than happy to implement it.
Regarding the login options, I would say it was the main motivation to create `immortal`, http://serverfault.com/questions/752839/how-to-make-svlogd-s..., these days dealing with microservices and try to follow a 12-factor approach force you to innovate, in this case the idea of delegating a logger to a 3rht party is because doesn't limit your options, for example, if you want you could just pipe directly to [filebeat](https://www.elastic.co/products/beats/filebeat) and at the same time continue writing files to disk and rotate them.
The idea of not using link files was to simplify more the process, is true that while writing to the file there could be some issues, I will have to check/test more in detail, but the approach I took, was to keep checksum of the files to guarantee integrity and also based on the atime take actions: https://github.com&#x...
I agree with Jonathan on everything here, with the additional caveat that the Go runtime is particularly heavy, and thus not really suitable for low-level system software (which should use resources sparingly, leaving them for production applications). So I don't think Go is a suitable language choice for a supervisor in the first place; and apart from the language choice, all of Jonathan's criticism applys.
I strongly recommend the author of immortal to join the supervision mailing-list on skarnet.org, follow the discussions there and ask questions if needed. We have about 19 years of experience with process supervision, there has been a lot of prior art and successive refinements, and we hold extensive knowledge about what constitutes good practice. If someone aims to design a process supervisor, even if it's a completely new take with a specific spin, it would be a mistake not to tap into our pool of experience. We always want users to benefit from good, rigorous software.
Hi, so assuming immortal was created using the best in the class programming language for its purpose, and let's say was following all the existing best practices, I still have one question that never really found a proper solution, which is:
For cases in where is required to monitor services like unicorn/gunicorn/Nginx that demonize them self and have the capability to fork how to make aware the supervisor of does changes and to continue monitoring the new process without entering into a race condition, to be more clear here is an example of this problem https://asciinema.org/a/80371, and work around's here http://serverfault.com/a/587065/94862.
Type=
If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.
PIDFile=
Takes an absolute file name pointing to the PID file of this daemon. Use of this option is recommended for services where Type= is set to forking. systemd will read the PID of the main process of the daemon after start-up of the service. systemd will not write to the file configured here, although it will remove the file after the service has shut down if it still exists.
There is no good solution for servers that, mistakenly, background themselves; the practice of backgrounding itself comes from a time where supervision was not as widely known as it is today, and nowadays there is no excuse for a daemon to fail to at least provide an option to avoid backgrounding itself.
The best thing to do is to contact the daemon's authors and insist they change that. In nginx's case, they think it's ok if they perform their own supervision, but it would be nice if they also could integrate with existing supervision systems.
In the meantime, as a workaround, the accepted practice for supervising forking daemons is known as "fghack": maintaining a process keeping track of the forking daemons via a few pipes that close when the daemons die. s6, for instance, provides such a tool: https://skarnet.org/software/s6/s6-fghack.html . AFAIK, so does nosh.
I really like that it has a user-mode incarnation that looks in ~/.immortal. One of the things I really enjoy about Master Control Processes is having a well-declared place where one can expect to find a manifest of processes.
This is pretty much a non-problem for us anymore. We use nomad (www.nomadproject.io) as a distributed process manager(and scheduler), it also does lots and lots of other cool things as well, but it also happens to handle this use case, as a sort of side-effect of what it does.
Is there more comprehensive docs anywhere? I want to know what options to pass, and more detailed descriptions of what things do. The initial docs page looks like but after clicking through it seems like they all link to each other... so in summary I cant find anything I want :/
That currently would be the primary barrier to usage -- the docs arent very helpful.
"A Distributed, Highly Available, Datacenter-Aware Scheduler" is a bit more than what I want. I just want a cross-platform way to install/remove (and query the status of) a daemon process (i.e. run your process as a Windows service, launchd service, or linux equivalent: systemd, ...).
I agree, it's a little over-kill for your particular use-case, but it works well, and it can be run in -dev mode, so you don't have to use the whole distributed, HA and datacenter part(s).
I don't have much experience with it on windows, but I use it regularly on *nix platforms and it works wonderfully.
Is anyone ignorant enough to believe that any monitor can survive general process failure or contribute to recovery of a system with abnormal function or resource starvation?
And that is the basic reason for these monitors! garbage.
My last experience was with bluepill which was this mess of
ruby nonsense. Anyone with any experience knows you can't trust a system to correct it's own failure. Why are we hammering this nail over and over to catch a 30% recoverable transient...management.
I think that's why they are external "monitoring" tools to check how the status of your services and rather than just restarting a daemon, the ones if required could add more instances, restart them or rebuild everything, maybe something like https://epazote.io/
You need to read the archives of the supervision mailing-list at list.skarnet.org, as well as http://skarnet.org/software/s6/overview.html - there is a lot of value in process supervision, it's just that most people who whip up their own solution end up doing it wrong. Like crypto, in a less dramatically noticeable way.
Regarding if is mature enough, strictly talking still not a stable version, current beeing `0.11.0` and not jet a `1.x.x` but the more you could test and evaluate could be nice.
36 comments
[ 5.4 ms ] story [ 102 ms ] threadIt's an aesthetic thing mainly, and a bit silly I'm sure, but dammit, I find it unsettling to have my Python 3 stuff running, and then the legacy Python 2, simply for the sake of supervisord.
Likewise, I don't need any Python on my Go-server, please.
I can see caring about performance. A Python process will be more memory hungry than something hand coded in C. But for the rest of it, who gives a crap if t is packaged well. And if it's not, it's probably not worth using it at all.
As I said, it's a question of aesthetics, and probably quite irrational, possibly OCD, but I do like my servers tight.
And it presents a solved problem (some daemons fork themselves off and the parent exits, so tell them not to do that or use fghack) as a new thing that Immortal can solve.
Is there something actually new and interesting about this that I'm missing?
Mostly, though, covering the point about parent forks and exits that you mention.
The ability to have go produce cross platform binaries would ease a lot of deployment issues as well. Along with distributed logging, it's something I'll probably try using. Unless any of the above have advantages, primarily in the ease of use category.
The idea behind was to avoid a developer to deal with a DevOps/operator to just stop/restart services, something useful on staging environments.
For starting, stopping a process there is no need to unlink/link a file, this simplifies things a little bit, for example, https://immortal.run/post/ansible/ by only updating the file contents or "touching" the file it can be started or re-started.
When following a PID, signals can still be sent to the process, these things become handy while working with a lot of microservices.
The docs need to be improved, but any feedback, comments or even desired feature is more than welcome.
* There is no composable toolset, but rather one monolithic program that does only a small part of the job. envdir and setuidgid are -u and -e options to the supervise analogue that is that monolithic program, "immortal". All of functionality of the other daemontools tools, such as envuidgid and softlimit, let alone the likes of runit's chpst, is missing. Not only are there no composable tools for it, the monolithic program simply lacks their functionality entirely. One can extend this functionality with the composable toolsets, with tools to handle things from scheduling classes to control groups. The monolithic program lacks all of this.
* There is no equivalent of the service restart/finalization mechanisms of runit, perp, s6, or nosh; which permit taking various actions dependent from how a service program terminates.
* Nor is there an equivalent of the at-stop and at-start mechanisms of s6 or nosh.
* There is no service readiness mechanism, as s6 and systemd have.
* s6, nosh, and systemd all have the concept of "oneshot" services. immortal does not cater for them.
* The svscan analogue, named "immortaldir", lacks an equivalent to the "down" file mechanism from daemontools, runit, perp, s6, et al.. All services always come up at bootstrap. There is no enable/disable mechanism at all.
* The author hasn't borne in mind the design lessons from svscan, the reasons why it works as it does, such as only having symbolic links in the scan directory, not the actual service definitions themselves. There is a race condition between copying or saving the YAML file into its directory and immortaldir trying to open and read it, which can result in immortal reading a part-written configuration file. There is also the potential for hidden files to be recognized as services, something that svscan explicitly prohibits.
* The author hasn't borne in mind the design lessons from s6-svscan. immortaldir wakes up and polls every few seconds instead of being event-driven, which as M. Bercot notes is a consideration on systems that want to save power.
* immortal does not have logger services, as perp, s6, daemontools, daemontools-encore, freedt, nosh, and runit all do. Instead, it has only the options of the service manager itself writing the log file, or a non-supervised non-manageable logging command. It is thus not possible to control logging through service management.
* There is no equivalent to multilog's rotation scripts, filtering, multiple log directories, or automatic timestamping.
* The log rotation uses the old .1 .2 system. The author hasn't borne in mind the design lessons from multilog, the reasons why it did the things that it did. So it doesn't ensure that logs are safely written to disc at rotation time, for starters.
* The service management API is not compatible with daemontools, daemontools-encore, runit, and nosh; which all share a common binary-compatible subset (the daemontools service management API). This API is, rather, a WWW server. Where daemontools et al. send simple one-byte commands; this API uses JSON and the Internet Message format meaning that a simple command to send a signal is actually an entire HTTP/TCP transaction, with all of that parsing and marshalling going on, under the covers. This flies in the face of the "do not parse at runtime" dictum followed by the other toolsets. There is an entire HTTP parser and server running with superuser credentials inside the service manager here. Ironically, this is a criticism that people vehemently (but erroneously) level at systemd. This service manager actually does do this, though.
* It also contains an entire YAML parser.
* There is no service interdependency mechanism; at all. There aren't even the tools for the old "thundering herd" approach of starting all services and having them manually check their depe...
I apologize in advance because unfortunately, the current documentation of the project don't cover all cases and because of that, some assumptions can be made until going deep into the code or testing, I will try to clarify some point hoping I could transmit a better idea of what the goal was and how is working, so pease bear with me.
For example, there is a "down" option, same as the "once" options https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
Same with the enable/disable that in immortal is "start/stop" or also -x that is "exit"
You indeed can send a signal STOP, mainly because of this when using [immortalctl](https://immortal.run/post/immortalctl/) the option "stop" is used instead of down:
https://github.com/immortal/immortal/blob/master/cmd/immorta...
When following a pid, you indeed can still send signals to the "adopted PID" if we may call it that way.
Now immortal is totally capable of knowing the process ID of the daemon, but the methods you described as for reading PIDS from a file and checking if process are up and running, does methods are mainly used within the `-f PID` follow PID option in where you want to supervise or know status about a PID that the supervisor didn't create and, for this in BSD systems Kqueue is used instead of long polling https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
I have some animations here http://immortal.run/about/ maybe they are not very clear but hope they could clarify this.
In case you know a better way of knowing if a process exists, something like asked here: http://stackoverflow.com/a/15210305/1135424, I will be more than happy to implement it.
Regarding the login options, I would say it was the main motivation to create `immortal`, http://serverfault.com/questions/752839/how-to-make-svlogd-s..., these days dealing with microservices and try to follow a 12-factor approach force you to innovate, in this case the idea of delegating a logger to a 3rht party is because doesn't limit your options, for example, if you want you could just pipe directly to [filebeat](https://www.elastic.co/products/beats/filebeat) and at the same time continue writing files to disk and rotate them.
The idea of not using link files was to simplify more the process, is true that while writing to the file there could be some issues, I will have to check/test more in detail, but the approach I took, was to keep checksum of the files to guarantee integrity and also based on the atime take actions: https://github.com&#x...
I strongly recommend the author of immortal to join the supervision mailing-list on skarnet.org, follow the discussions there and ask questions if needed. We have about 19 years of experience with process supervision, there has been a lot of prior art and successive refinements, and we hold extensive knowledge about what constitutes good practice. If someone aims to design a process supervisor, even if it's a completely new take with a specific spin, it would be a mistake not to tap into our pool of experience. We always want users to benefit from good, rigorous software.
For cases in where is required to monitor services like unicorn/gunicorn/Nginx that demonize them self and have the capability to fork how to make aware the supervisor of does changes and to continue monitoring the new process without entering into a race condition, to be more clear here is an example of this problem https://asciinema.org/a/80371, and work around's here http://serverfault.com/a/587065/94862.
Immortal and Systemd seems to fix this problem, but based on JdeB comments, they are employing the "old broken idea - Witness https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
Therefore wondering, what is the best approach, solutions for this cases?
Or could it be than Immortal is not that wrong, at the end, is solving a problem, maybe broken or not but is working and could be improved for sure:
https://asciinema.org/a/80360
As a reference, from systemd man page for options
Type= If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.
PIDFile= Takes an absolute file name pointing to the PID file of this daemon. Use of this option is recommended for services where Type= is set to forking. systemd will read the PID of the main process of the daemon after start-up of the service. systemd will not write to the file configured here, although it will remove the file after the service has shut down if it still exists.
The best thing to do is to contact the daemon's authors and insist they change that. In nginx's case, they think it's ok if they perform their own supervision, but it would be nice if they also could integrate with existing supervision systems.
In the meantime, as a workaround, the accepted practice for supervising forking daemons is known as "fghack": maintaining a process keeping track of the forking daemons via a few pipes that close when the daemons die. s6, for instance, provides such a tool: https://skarnet.org/software/s6/s6-fghack.html . AFAIK, so does nosh.
That currently would be the primary barrier to usage -- the docs arent very helpful.
I don't have much experience with it on windows, but I use it regularly on *nix platforms and it works wonderfully.
Is anyone ignorant enough to believe that any monitor can survive general process failure or contribute to recovery of a system with abnormal function or resource starvation?
And that is the basic reason for these monitors! garbage. My last experience was with bluepill which was this mess of ruby nonsense. Anyone with any experience knows you can't trust a system to correct it's own failure. Why are we hammering this nail over and over to catch a 30% recoverable transient...management.
I want to use Immortal instead of systemctl. Just a question - is this mature enough to be used in production systems ?
the readme file has some instructions about how to do it: https://github.com/immortal/immortal/
Regarding if is mature enough, strictly talking still not a stable version, current beeing `0.11.0` and not jet a `1.x.x` but the more you could test and evaluate could be nice.
Please feel to raise any issue if needed here: https://github.com/immortal/immortal/issues