Ask HN: How do you monitor your systemd services?

I push telemetry to Amazon CloudWatch (my infrastructure is on AWS) and then setup alarms accordingly. If I'm concerned about a service failing or becoming unresponsive, it's easy to create an alarm based on the existence or non-existence of data.

This thread is one of those cases where you read something and realize you've been completely missing something. I don't monitor these as much as I should

Servers/services? Definitely - take your pick. Timers/jobs, particularly those on my system? Nothing!

With the right directives laid out ('Wants/Requires/Before/After'), they can be pretty robust/easily forgotten.

I've been lucky in this regard; I check 'systemctl list-timers' just to be sure - but they always run

I run the Prometheus node_exporter on my servers. That has a systemd collector for the state of services.

That reports the state of all systemd services to a central Prometheus and alertmanager cluster, which has various alert rules.

Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don't forget.

Long answer: Whenever I've started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to start from the beginning with an extensible system. For me, Prometheus has been the best option: easy to configure, lightweight, doesn't even need to run in the host, and can monitor multiple systems. You just have to configure which exporters you want it to pull data from. In this case, prometheus_node_exporter has a massive amount of stats about a system (including SystemD), and there are default alarms and dashboards out there that will help you create basic monitoring in a minute.

You can choose to use Grafana for visualization, and then either the integrated Grafana alerting or use the Prometheus alerting + Prometheus Alertmanager. I think in the latest versions Grafana Alerting includes basically an embedded AlertManager so it should have the same features.

Regarding the type of alert itself, I send myself mails for the persistence/reminders + Telegram messages for the instant notifications. I find it the best option tbh.

Run the job via cron, if it fails you'll get an email sent according to your system's alias file. You can also grep the logs for failure if you think you didn't get the email.

If you are ok with a Saas and if it's just scheduled jobs that you are monitoring, there are a number of monitoring tools where you tell when job completes (with a http request) and a missing ping (after a grace period) means that it failed.

I think https://deadmanssnitch.com/ may have been the original service for this.

https://healthchecks.io/ has a fairly generous free tier that I use now.

There are others that do the same thing Sentry, Uptime Robot, ...

I generally just call up goggle and ask them how my system is doing...

In general this evolves to a SIEM-like solution in IT or gets added to the tag menagerie in OT.

If you're focused on "notifications are bad" note that notifications are push, and pull solutions are possible. Tail logs (or journalctl) and post significant events to Redis (https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) for example.

I was building an elaborate job monitoring system, but then I realized that what I really need is monitoring the actual end to end functionality.

For example, instead of monitoring my Minecraft server process that OpenRC spawns, I have a dedicated monitoring server that actually queries the server for version, number of players, etc. Same for websites, etc. Think of it as periodically running an integration test on a live system.

This way I get much more confidence that the service is doing what it should.

I'm not a big fan of over complicated monitoring systems - I simply have a script that builds a HTML status page with enough information to know when something goes wrong.

@wh33zle For work, well, i have to follow already-established convention (some that others have noted). but for personal machines, i have not rolled out too many comprehensive monitoring solutions or platforms. Rather, i add focus on moitoring specific jobs/tasks, and as such leverage cron to run the job, and use basic, old school sorts of bash scripts to assess success or failure. I'm statrting to look into leveragin more systemd as you noted.

Now, specific to alerting, well, i have rolled out my own solution...Caution: self-promotion coming next...

I stopped relying on email being sent from servers since i've had too many annoyances, constraints in my history. Also, nowadays email is a medium that is slow for me...that is, i treat it like its non-time-sensitive3 messaging (for the majortiy of the time). So, for system alert-style messagings, I use my own little python script that sends messages into a dedicated matrix room. Since, i'm always on matrix, its a place where i can quickly see a new system alert messaage (matrix clients like Element allow you to adjust visibility - i think they call it noise level - of which messages are given higher or lower priority for the client vieew, etc.). And, those messages tend to be ephemeral, since they're just alerts, and such messages do not pollute my email inbox. There are plenty of options in this space of course. Mine is not the only one, but i also wanted to learn how to make apps for matrix ecosystem, etc. Here's a link to my little notification app/script that leverages the matrix network chat ecosystem: https://github.com/mxuribe/howler

`0 * * * * journalctl --since="61 minutes ago" --priority=warning --quiet`

In crontab piped to a bunch of grep -v for the things I want to ignore

So basically the email approach, just have to be religious about marking unread if not immediately actioned

I don't monitor services at that level at all, because it means basically nothing. More acutely: the the lack of a notification doesn't tell mean everything is "ok".

I tend to monitor the actual service. If it's a web server, have something checking that a specific URL is working (tip: use something specific, not /). Likewise any other network service is pretty easy to monitor.

For backups, check the date on the most recent file in the backup target location. If that date is older than "x", something is broken. This can apply to most other types of backend apps too -- everything has some kind of output.

It's when these checks fail that you can investigate deeper and start diagnosing systemd or whatever. It's also possible there's a bigger problem -- like DNS got messed up, or the hardware died -- and checking the final outcome will catch all this.

Basically explicitly checking systemd is a lot of extra work for no real added benefit. If your systemd service is failing often enough that knowing that is the problem immediately (at the alert level) IMHO you'd be better off to spend the time fixing the service definition so it doesn't fail.

I use Nagios, easy, lean and gets the job done

Only send alerts from the end user perspective. In your case the end user would most likely go into the backups and list them. So I would have a job that lists the backups every day and if something is missing it alerts.

See the difference here is that you don't monitor the systemd backup job, you monitor the backup backend instead. Because systemd can be configured to retry a job, the end result is in the backend.

And in other cases I do have monitoring for individual services, but I only send alerts if the end user experiences an issue. So a web server process/systemd unit is being monitored, but the alert is on a different monitor that checks if the website returns 200, or if it contains a keyword indicating it works.

This combo does the job for me: grafana + riemann + influxdb and collectd as the main agent. collectd bundles many plugins so you can watch logs, monitor running processes or have something custom [1]. This setup is very light to start with and can scale well (up until you hit influxdb limits :D).

[1] https://github.com/mbachry/collectd-systemd

I love https://ntfy.sh/ for my services running on headless servers - it lets me ping my phone with messages of varying urgency, and even duplicate the notifications to email for particularly information-dense messages.

> "I have a backup job that is triggered by a timer. I want to know when that job fails so I can investigate and fix it."

This is really more in the realm of a shell script.

You could do this verbosely:

  #!/bin/sh

  /path/to/my/backup_job

  if [ $? -ne 0 ]
  then /path/to/my/failure_alert
  fi

...or, you could do this tersely:

  #!/bin/sh

  /path/to/my/backup_job || /path/to/my/failure_alert

The wrapper script would go into your timer unit. I like dash.

Everyone seems to be talking about production services & headless servers, but my impression is that you meant on the desktop?

I wrote a little script that puts a failed service count in waybar, and throws up a dismissable swaynag message with buttons to 'toggle details', and reset or restart the failed system/user units.

It's a bit noisy at the moment - but I think that's probably just a helpful indication of units I need to sort out/make a bit more robust anyway.

I think this is a great question. Consider https://blog.wesleyac.com/posts/how-i-run-my-servers. His unit files do not mention monitoring.

I have a carefully designed alert service for every project, checking various aspects of the system. It periodically checks heartbeats from various systems to make sure everything is in order. It sends alerts to UI via websocket, and to slack channels and makes calls to twilio numbers if things do not self-recover in time. I only check if the alert system is running via cron.

I’d go with sending myself a push notification on the phone through a dedicated service file and then call it with this in my unit file:

OnFailure=send-push-notification.service

Perhaps via a WhatsApp notification or any other instant message [0] or any other service such as matrix as said in another comment.

[0] https://developers.facebook.com/docs/whatsapp/cloud-api/get-...

> Notifications are [...] ephemeral, meaning I might forget about it if I don't deal with it immediately.

If you do like the notification method aside from this issue, try passing "--urgency=critical" or "--expire-time=0" to notify-send. Either (or both) of those should make the notifications stay popped up, assuming your notification daemon is doing something reasonable with those hints.

(Disclosure: I'm the author of xfce4-notifyd, which does behave in this way; other daemons may do other things.)

I use the `OnFailure` property to trigger a service that emails me for failed services like backups which are run as system timers + service.

I also use `failure-monitor` which is Python service that monitors `journald`.

Files on Github for those interested:

https://github.com/kylemanna/systemd-utils

We don't use systemd, so we haven't had issues with it.

Things get deployed by the automatic deployment system. If they go in cron, they are supervised by a program called errorwatch which does all the things that you want in a one-shot supervisor: logging, error codes, time bounds, checking for right output, checking for wrong output. If they are daemonic, they get /etc/init.d/ start/stop scripts that have been tested.

If they have a habit of dying and we can't afford that and we can't fix it, we run them from daemontools instead of init.d.

I've been using Checkmk (raw - e.g. free) to monitor stuff in my home lab (mostly for other things.) It has notified me of some failed Systemd services.