HN Explorer

Ask HN: How to monitor periodic short-lived processes?

4 points by sharmi ↗ HN
Hi HN,

    We are two founders starting up a business. We need to run multiple programs everyday (around 25) that will be run periodically. They are not always alive though and will often be done in 10 minutes to  an hour.
Unfortunately as a two person company, keeping track of all the programs' statuses is hard.

Is there a way to keep track (and if possible, also spawn them on schedule) of the processes and give regular updates? Like a dashboard of sorts, that can show which programs have been running successfully and which have failed to finish or didn't start?

12 comments

[ 3.5 ms ] story [ 43.9 ms ] thread
DamonHD
For a *nix box, the simple/historic way has been to run the tasks from cron and append log output from them to text files or the system logs. If you really mean statuses, as in exit statuses, wrap the programs in shell scripts which print out the programs' exit codes.

It's unfashionable these days I think, but still works for me.

verdverm
Cron is the term you are looking for and there are many options. As another commenter said, there are Linux system level solutions. There are also Cron concepts in kubernetes, aws lambda, zapier. If they don't have a dashboard, I'm sure there is another SaaS that does and plenty of open source solutions for that as well
yuppie_scum
Also curious what the best way to visualize cron status is, for example using timeseries data from kube or a syslog.
verdverm
ah, we don't care when it runs as much as when it doesn't run, so we slack on failure
sharmi
I do use system level cron. The problem was in monitoring them. Maybe like the other commentor said, one could parse syslog to monitor.
2143
You might have to come up with something custom.

Here's one idea:

1. Wrap your processes in systemd services or something similar, where it's possible to determine the status by pinging it.

2. Write a script to periodically (via cron maybe) ping all relevant services, and store the statuses somewhere (file, DB..)

3. Another script (or the previous one) to read the data from 2 and spit it to a locally served webpage.

> if possible, also spawn them on schedule

Cronjobs.

sargstuff
Make use of dynamic tracing and write/customize relevant to needs.

Potential examples:

https://twitter.com/freebsdfrau/status/1238541870612926465 modify to just focus on cron related stuff.

https://www.brendangregg.com/blog/2016-02-01/linux-wakeup-of...

Programming approaches:

ebpf : https://ebpf.io/projects/

Dtrace : https://www.brendangregg.com/dtrace.html

Ylang : https://blog.openresty.com/en/ylang-intro-part1/

Program example: Use execsnoop (https://www.brendangregg.com/dtrace.html ) & add wrapper/filter to feed relevant "live" info to dashboard.

sargstuff
turnkey program(s) per google search:

Cronitor : https://cronitor.io/docs/using-cronitor-cli

Nagios : https://exchange.nagios.org/directory/Tutorials/Other-Tutori...

https://jaytuckey.name/2020/10/30/new-project-coconut-deadma...

https://www.saashub.com/compare-cronhub-vs-zabbix?ref=sideba...