systemd handles service monitoring natively, as well as socket management and many aspects of container management. It's a superset of most of the tools listed.
Right now, I use Upstart (and thus Ubuntu) because it can--kinda-sorta--do the same "supervision-tree" thing that Erlang/OTP does; you can have a virtual "service" that runs a script on startup to get a list of tokens, then starts a set of dynamic children using those tokens ("initctl start myservice/worker FOR=bob", that kind of thing), monitors them, and restarts them when they crash; bringing down the supervisor cleanly shuts down its currently-running dynamic children, and so on.
Can systemd do that? I like everything else I've heard about it, but I haven't seen any documentation regarding this sort of usage.
systemd can, right now, launch "template" services based on a common template with a parameter. For instance, the usual set of text login prompts are services named getty@tty1.service, getty@tty2.service, ..., getty@tty6.service, all spawned from a template getty@.service.
systemd is currently adding support for dynamic service "instances", which would allow you to launch and close such services on the fly.
So, depending on how dynamic you need the list of virtual services to be, the answer is either "yes, that works right now" or "yes, that's available in the latest systemd release, but probably not in your distro yet".
Funnily enough, last time I set up a supervision tree hierarchy with Upstart, I was thinking the whole time, "this would be so much nicer with Erlang." I think that was my a-ha moment for OTP.
For most standalone systems though, Upstart works nicely enough as long as services don't need too much coordination.
I'd love to try systemd in the future but right now we deploy in Debian and for us is easier to go supervisord (and all our devs "speak" Python, that's a plus).
My main concern is that systemd handles too much and I feel it's going to be hard to change. I guess we can start using systemd without removing supervisord and move from there.
Urgh, that doesn't sound great. We haven't encountered that issue, but we also only use it for a largely inconsequential service.
I think it's a bit unfair to downvote me, the OP has posted a Poll for what people use, not what they recommend. We use God, so I've asked for God to be added to the list.
I've been using god for two years and have never had this problem, or any problem with it. Maybe the god load command has problems, but I can't think of any reason to use it.
Bottom line - try a few monitoring tools out and use what works for you.
I've been seeing this problem in prod for the last 8 months, consistently. I literally have one process that doesn't start 3/4ths of the time. My start up script has to iterate through all the processes god is supposed to start and if they're not all running kill god and try to start again. Not inspiring a lot of confidence in god as a monitoring solution.
Monit is by far your best bet. Easy to install, packaged on most distros and it performs reactive monitoring as opposed to most traditional monitoring systems like Nagios. Plus you can open up a web interface if you want to allow for easy browsing of monitored processes.
As one of the guys behind Monit, thanks for that! We're about to improve Monit further by adding realtime (ms resolution) scheduling and monitoring (libev or libuv) which we hope will make Monit even more useful.
Hey.. We use monit and mmonit on all our servers for a lot of varied services, and it is truly a life saver! I only wish the web console for mmonit was a little more detailed and showed more system information. It is also sometimes helpful to see a graph of memory, cpu usage, and server response time over the time axis to see the current system load.
Systemd for most Servers, svc / rundir ( a version of daemontools ) from busybox on embedded services.
The code is designed to be shot, and will always recover after a restart. rundir/svc will neatly reap a process and re-start it again. And can be used separately.
I use runit in production for http://typing.io. I appreciate runit's strong unix philosophy (shell scripts instead of dsls). However, I'm starting to experiment with systemd because of features like properly tracking and killing services [1]. This feature would be useful with a task like upgrading an nginx binary without dropping connections [2]. This isn't possible with runit (and most process monitors) because nginx double forks, breaking its supervision tree.
Supervision has many benefits besides automatic restarts after crashes. Supervising programs provide a consistent way to start, monitor, and log long running programs. Nginx reinvents its own interface for some of this functionality (like daemonizing, log rotating/compression, conf reloading, etc.), but it's useful for all services to work under the same interface. This is especially true for monitoring nginx's status, where a supervisor like runit is much nicer than 'pgrep nginx' or 'ps aux | grep $(cat /where/nginx/dumps/its/pid)'.
There are some good points there, but they're not particularly applicable to nginx. nginx's relative robustness, its need to log to more than one file, and its need to fork itself in order to perform graceful configuration state transitions suggests that it's not really a low-hanging fruit for robust process supervision.
Besides, its control interface is the same as any other subsystem if you've configured it correctly -- i.e., "service nginx <verb>".
I agree that nginx needs supervision less than most processes because it reinvents many wheels. However, supervision is still nice, e.g. your 'service nginx' example that uses Ubuntu's supervisor Upstart.
I agree it's not worth straining to make nginx's binary upgrade work with arbitrary supervision. However, if someone created a supervisor that solves this problem (systemd), I might give it a try.
Like with all software... it doesn't until it does. And when it does you wish you had an auto-restart system in place.
It doesn't even have to be nginx's fault. It could be that some other process started fork-bombing the system and OOM-killer decided that killing nginx is the way to resolve it, before trying the actual offender.
Disabling daemonization allows nginx to be monitored initially and through conf reloads but not binary upgrades (e.g. nginx 1.5.1 -> nginx 1.5.2). The binary upgrade process forks the master process, which it 'orphans' from runit and reparents under pid 1.
And because you're telling Google more when you log into a site with OpenID (or any Google-specific mechanisms) than you're telling anyone when you log in with Persona.
Anyone whose email provider provides persona automatically, or who runs their own identity provider, or who has taken 10 seconds to create an account on Mozilla's.
I don't, currently, because I don't have to monitor services, but if I did, I think I'd likely use daemontools, based on the fact that djb really, really understands how to write Unix software.
I use daemontools to manage all the servers for SlickDNS (https://www.slickdns.com). Trivial to install and configure. Most of my run scripts are one liners, and of course like anything written by DJB daemontools just runs and runs. (Naturally I'm also using tinydns on the SlickDNS name servers.)
djb however doesn't know how to get on with other people. While the software is pretty neat, it does piss all over your disks and leave a hard to untangle mess.
upstart because I wouldn't use a job control system in prod that isn't included with the base distribution (do any base unix distros use monit or supervisord?) It's just too much useless work to rewrite job control logic for daemons when the OS already gives them to you, and I've been quite surprised with the feature completeness of upstart.
Folk on Solaris / Illumos would probably like SMF to be added to the list.
I'd go on to mention various z/OS subsystems, but that's a bit esoteric even for HN :D
(Process management ties into my larger rant that nobody has properly combined it with configuration management. But nobody has time for this nonsense.)
Do any of these integrate with cgroups? I've found myself wanting to specify some rules about resource usage on occasion, and cgroups seems conceptually nice, but I'm not sure how to work it in nicely to my other tools, short of writing custom shell scripts to manipulate /proc/cgroups.
Systemd provides many knobs for tuning cpu, memory, and io settings [1] using cgroups. This deep integration is one of the reasons why systemd only runs on linux.
While it's true that "nothing" or "custom" should be poll options, I don't agree with your reasons.
Supervisor programs aren't just about restarting processes if they die. They manage dependencies, reloading, a central authority for starting/stopping and other administrative tasks. Some additionally do things like logging.
And besides, even the most stable software can't be guaranteed bug-free (though I do agree that a better action might be to stop and yell loudly rather than blindly restart). Why forego a safety net?
I think there are some problems with your statement. All these process-reloading and management options are a symptom of a much larger problem. The problem is simply:
People have stopped writing software designed to run continuously.
Some notes from 20 years of Unix fudging:
If a process dies, it's a problem with the process, not an external problem. Make it reliable. Software shouldn't fail at all. PostgreSQL, Postfix, init never crash from experience. Why shouldn't your software do the same?
Dependencies: Dependencies are hell. Having used BIG Unix kit with power sequencers and stuff, I wouldn't ever take on a dependency. If I did, the dependent processes should fail gracefully and retry.
Central authority: service xxx restart/reload again...
Logging: syslog? On our kit, we use Splunk but it's pretty much overkill for most people.
If you can't sleep at night because your processes crash, there's no good employing someone (a supervisor) to go around and do CPR on them. Fix the root problem.
Funny you should mention PostgreSQL, since it includes a monitoring process - the postmaster - that restarts the workers if they die ;) (and they do, occasionally).
That's because the child processes can load and run third party code, which obviously can crash them. Just like apache modules can crash httpd worker processes. So why run an extra layer of monitoring on top of the existing one for apps like those?
All the software in the poll is general purpose monitoring/restarting software. It isn't software designed to be multi-process and monitor its own child processes.
How do you cope with hardware failure, corrupt memory banks, temporary network failures, you also just buy "hardware that never fails"?
Just testing something and deeming it reliable enough to "just run" is a horrible practice.
There is no software on earth that is bug-free, you do realize that, yes?
Why do you think the NASA builds rockets with many many safety measures? Because they use the latest and greatest hardware and software? Or because they KNOW that at some point something WILL fail. It's not a only a question of picking stable software. The skill is to be prepared for the emergency at any time, even if it is on a rare occasion.
If your only safety measure is to "just pick stable stuff", i'd surely never buy your service.
Firstly, calm down. I gave you a small peek into my world and you're drawing a lot of bad assumptions from it.
We mitigate hardware issues with either hot spares or clustering. We have three datacentres distributed geographically with entire redundant sets of equipment (4x 42U racks in each).
With respect to software failures, we test everything thoroughly including failover conditions etc. Everything is load tested as well.
We are prepared for emergency. We have dedicated people ready to jump on that.
However preventing these things ever being needed is a professional responsibility which is my point.
The poll is about the fad of using a program to monitor your known to crash/leak/whatever app, generally starting it, monitoring it, and restarting it. Hardware failure is completely irrelevant to the discussion.
Yes I have. A whole ISP in fact, including hosting corporate email for hundreds of small to medium businesses. Postfix never crashed. Openldap never crashed. Courier never crashed. What is monitoring and restarting your monitoring/restarting daemon?
1. daemontools and runit are practically identical. I do prefer runit somewhat, as svlogd has a few more features than multilog (syslog forwarding, more timestamp options), and sv has a few more options than svc (it can issue more signals to the supervised process).
2. Among the criteria I look for in a process manager are: (1) the ability to issue any signal to a process (not just the limited set of TERM and HUP), and (2) the ability to run some kind of test as a predicate for restarting or reloading a service. The latter is especially useful to help avoid automating yourself into an outage. As far as I'm aware, none of the above process supervisors can do that, so I tend to eschew them in favor of initscripts and prefer server implementations that are reliable enough not to need supervision.
Systemd provides a hook (ExecStartPre [1]) for running test commands before starting a service, e.g. checking nginx configuration before starting. However, I also prefer the flexibility of shell scripts over something like systemd's INI service definition format.
129 comments
[ 3.2 ms ] story [ 168 ms ] threadmonit: http://mmonit.com/monit
supervisord: http://supervisord.org
daemonize: http://bmc.github.com/daemonize
runit: http://smarden.org/runit/
perp: http://b0llix.net/perp/
DJB's daemontools: http://cr.yp.to/daemontools.html
systemd: http://www.freedesktop.org/wiki/Software/systemd
god: http://godrb.com
upstart: http://upstart.ubuntu.com
Can systemd do that? I like everything else I've heard about it, but I haven't seen any documentation regarding this sort of usage.
systemd is currently adding support for dynamic service "instances", which would allow you to launch and close such services on the fly.
So, depending on how dynamic you need the list of virtual services to be, the answer is either "yes, that works right now" or "yes, that's available in the latest systemd release, but probably not in your distro yet".
For most standalone systems though, Upstart works nicely enough as long as services don't need too much coordination.
My main concern is that systemd handles too much and I feel it's going to be hard to change. I guess we can start using systemd without removing supervisord and move from there.
Also I'll put in a shameless plug for my side project, a service management tool for multi project development; Hack On[2]
[1] http://godrb.com/ [2] https://github.com/snikch/hack
For reference see this bug which has been open for 2 years.
https://github.com/mojombo/god/issues/51
I think it's a bit unfair to downvote me, the OP has posted a Poll for what people use, not what they recommend. We use God, so I've asked for God to be added to the list.
EDIT: I liked it so much (and it was so easy) that I wrote a blog post expounding how much I liked it and how to use it. http://moduscreate.com/monit-easy-monitoring/
The code is designed to be shot, and will always recover after a restart. rundir/svc will neatly reap a process and re-start it again. And can be used separately.
[1] http://0pointer.de/blog/projects/systemd-for-admins-4.html
[2] http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_...
Besides, its control interface is the same as any other subsystem if you've configured it correctly -- i.e., "service nginx <verb>".
I agree it's not worth straining to make nginx's binary upgrade work with arbitrary supervision. However, if someone created a supervisor that solves this problem (systemd), I might give it a try.
It doesn't even have to be nginx's fault. It could be that some other process started fork-bombing the system and OOM-killer decided that killing nginx is the way to resolve it, before trying the actual offender.
oops, edit: pid 0 -> pid 1
Anyone whose email provider provides persona automatically, or who runs their own identity provider, or who has taken 10 seconds to create an account on Mozilla's.
https://news.ycombinator.com/item?id=1368855
I'd go on to mention various z/OS subsystems, but that's a bit esoteric even for HN :D
(Process management ties into my larger rant that nobody has properly combined it with configuration management. But nobody has time for this nonsense.)
Systemd was pretty stable until user mode flat out broke in 205. I use it to manage my entire desktop session.
[1] http://0pointer.de/blog/projects/resources.html
Supervisor programs aren't just about restarting processes if they die. They manage dependencies, reloading, a central authority for starting/stopping and other administrative tasks. Some additionally do things like logging.
And besides, even the most stable software can't be guaranteed bug-free (though I do agree that a better action might be to stop and yell loudly rather than blindly restart). Why forego a safety net?
If a process dies, it's a problem with the process, not an external problem. Make it reliable. Software shouldn't fail at all. PostgreSQL, Postfix, init never crash from experience. Why shouldn't your software do the same?
Reloading: system-wide: "service whatever restart/reload"
Dependencies: Dependencies are hell. Having used BIG Unix kit with power sequencers and stuff, I wouldn't ever take on a dependency. If I did, the dependent processes should fail gracefully and retry.
Central authority: service xxx restart/reload again...
Logging: syslog? On our kit, we use Splunk but it's pretty much overkill for most people.
If you can't sleep at night because your processes crash, there's no good employing someone (a supervisor) to go around and do CPR on them. Fix the root problem.
We pick software that is reliable and trustworthy then test it thoroughly.
Those are forgotten arts outside of the enterprise. Elsewhere, any new technology that falls from the sky is picked whether it works properly or not.
We mitigate hardware issues with either hot spares or clustering. We have three datacentres distributed geographically with entire redundant sets of equipment (4x 42U racks in each).
With respect to software failures, we test everything thoroughly including failover conditions etc. Everything is load tested as well.
We are prepared for emergency. We have dedicated people ready to jump on that.
However preventing these things ever being needed is a professional responsibility which is my point.
I've been dogfooding it in a production environment for a couple years and it's been pretty solid.
1. daemontools and runit are practically identical. I do prefer runit somewhat, as svlogd has a few more features than multilog (syslog forwarding, more timestamp options), and sv has a few more options than svc (it can issue more signals to the supervised process).
2. Among the criteria I look for in a process manager are: (1) the ability to issue any signal to a process (not just the limited set of TERM and HUP), and (2) the ability to run some kind of test as a predicate for restarting or reloading a service. The latter is especially useful to help avoid automating yourself into an outage. As far as I'm aware, none of the above process supervisors can do that, so I tend to eschew them in favor of initscripts and prefer server implementations that are reliable enough not to need supervision.
[1] http://www.freedesktop.org/software/systemd/man/systemd.serv...