But it can also make things more stable. I laughed when 30 years ago people recommended to restart Windows NT to make it more stable. I now have some Go websites running for a long time with daily Systemd restarts.
I believe this is one of the claims of chaos engineering. If you randomly restart services, you have to build them in a way that they're resilient to random failures. If you do this to entire servers, VM, or containers, whatever your unit of OS userspace is, you can also kill an attacker's foothold if they manage to get one. Sort of how if humans were capable of respawning in a healthy base state but with their memories intact, you could stop the spread of a disease by just killing everyone.
Heroku notably does (did? It's been a while) daily restarts. I believe they started it to take away the support hassle of folks whose apps had memory leaks, but it's also a convenient mechanism to allow cycling the infrastructure without needing live migration.
> I now have some Go websites running for a long time with daily Systemd restarts.
I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now).
No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them.
Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly.
And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.
Or it can make things worse. I tried a load balancer once that had a reboot once a month. Not worth it when the underlying service hosts had failures more like once every three months. Round robin DNS was better than that.
Another interesting thing you may want to check for is coredumpctl. At $job, I'm installing a custom handler for those, extracting the stacktrace and submitting it to Rollbar with the rest of the app events. It only fired once so far, but it's something you want to know about. (Same should be trivial to do with Sentry or other error reporting services)
auto-restarting after a failure is a reasonable first order workaround for transient environment issues (network or external dependency goes away for a bit) or perhaps rare per-request issues that aren't handled correctly. arguably it'd be better to understand and fix or mitigate the issue properly, but until you actually have time to do that, it's nice for the service to autonomously attempt to be available.
...except when that causes even more problems! one example i remember from a hobby project was a worker service that would make requests to an external service's API. I'd implemented a throttle in the worker to limit the number of external requests per second made against the external service, where the throttle state was stored in process memory and not persisted anywhere -- seemed to be a pragmatic design tradeoff. you probably see where this is going.
there was an exciting interaction where an external API response caused the worker's API client to throw an unhandled exception, which took down the worker service. then systemd would diligently restart the worker service immediately per the restart policy. the throttle working state had been lost, so upon being restarted the worker service immediately fired another request to the external API, which then issued the same API response, which caused the worker service to fail in the same way, ... luckily noticed by systemd and immediately restarted.
combine with a lack of monitoring + alerting and you get a mechanism where your worker service can make about 100,000x as many external API requests over a few days as it was meant to.
I mean, if you fail to code it correctly in the first place then fail to set up reasonable auto restart policy then fail to monitor it as well, yeah, shit will eventually break
I think they rather received a phonecall/pager on the weekend than having it restart. ;-)
This is actually part of the self-healing aspect. One way to see if a service restarts is using the following command:
sudo systemctl show [servicename].service -p NRestarts
I prefer to have this set as
Restart=on-failure
Also note, there are OnFailure to trigger an additional log message, or recovery service, and FailureAction to allow actions to happen, such as the suggested 'reboot' ;-)
I think a better way (as mentioned by the parent) is to use FailureAction to do something (like send an event to an alerting system). Granted, you have to modify all the .service files that you care about, which is annoying.
You would be selective. For OS services you should be able to rely on them...
You also don't want to deal with all of them. For this you could otherwise just use journalctl. You also have OOM issues, and this has its own facility.
> Whether you want to monitor for this sort of thing (and how) is an open question. It's certainly possible that this is one of the times where your monitoring isn't going to be comprehensive, because it's infrequent enough, low impact enough, and hard enough to craft a specific alert.
my preferred way to alert on this is to use the process start time that's included by default in most process-level Prometheus metrics (and is trivial to implement yourself, if you need to):
> curl -s localhost:9100/metrics | grep process_start_time_seconds
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68957669109e+09
on a stable system, this metric will be very close to static. you can feed it through the PromQL changes() function to get a count of how many restarts have happened in a given time window.
in my experience, for anything with an "alert if it's down for X minutes" rule, you probably also want an "alert if it restarts N times in Y minutes" rule.
Mh, good points. I'll have to take a look to include a restart count in our zabbix systemd template. Should be a cheap metric to track with deduplication but valuable to create tickets upon if it changes too much in a short timeframe.
A whole lot of alerting is made better by generally considering whether to alert on large changes in in almost any kind of event, and sometimes even by measuring the changes in the rate of change, because it has a better chance of surviving "alert fatigue".
"That process that restarts spuriously once in a while has restarted again" stops being noteworthy very quickly if it's deemed low priority, and often even might lead to alerting thresholds being change because it becomes a nuisance.
"That process that restarted every now and again now restarts 5 time as often as it used to" on the other hand is a lot more likely to get attention.
The reasonable way to notice is to have alerts for any unexpected restarts. Relying on noticing intermittent service disruption is bound to fail. And so is "remembering to check for this":
> in the future I'm going to want to remember to check for this
Whenever you think that sentence, you should notice this as a red flag and re-think your approach. You will forget. And if not you, then somebody else in your team. You need automation for things you can forget, otherwise your mental checklists will grow too large to handle and are just a distraction.
What is the best way to set up those alerts? Specifically how do you set something up that knows if unexpected restarts happened? Is there a dbus or similar event you can listen for?
Worth looking into prometheus. in basic form that'd be gathering metrics from https://github.com/prometheus-community/systemd_exporter on your hosts, and configure alerting in grafana or prometheus alertmanager to notify when a threshold is exceeded
> The reasonable way to notice is to have alerts for any unexpected restarts.
Yes, and to have alerts, I have resorted to write my own small tools [0][1] at the end of the day. Railgun proved to be very useful, but for smaller things, I'm writing the second one.
On the topic of alerts/notifications, does anyone know of a project or method for minimal selfhosted cross device notifications? Say I run a daemon on my home server which listens on a Unix or inet socket(s) for generic JSON messages which could come from anywhere, like an IRC plugin, bash command wrapper, or anything I can think of that I want to get a notification for (I choose JSON over sockets so that is quick to implement and can be easily embedded in whatever random environment that I can expose the socket to, even sandboxed ones with no internet connection). Then it exposes another socket where a daemon on my desktop and laptop can receive notifications from, and convert it to a notify-send command so that it's displayed on the desktop. And since the daemon on the home server saves the messages in say an sqlite database, my desktop/laptop can fetch messages and I can print a backlog if they were offline. Otherwise, I'll probably write something like this because I'd find it useful. Email seems to be the closest thing but I don't want to rely on an external service or run a whole internal mail server.
In a kubernetes environment, systemd auto-restarting a process inside of a container can hide problems like the article says - if it successfully restarts the process before a liveness probe can pick it up, even if you monitor something like container restarts, you could easily miss this.
> The reasonable way to notice is to have alerts for any unexpected restarts. Relying on noticing intermittent service disruption is bound to fail.
I'd argue that unexpected restarts should alert beyond a threshold. Alerting on every occurrence is too noisy. If an individual unit failure causes a service disruption architecture improvements are needed.
Don't you want to know if something restarts unexpectedly? It's a bug that should be understood and fixed. (If it's not a bug then it's not unexpected.)
But if you have too many alerts, you have to remember to check those, and the temptation is to skim over them, and then you'll inevitably miss stuff. Then you need an alert system to alert you to the really important alerts.
There are plenty of similar examples out there. Take, for instance, Kubernetes indefinitely failing to start a pod and just retrying while taking no action to notify whoever started the process (just hanging).
So, I can see two different aspects to this problem:
* Lack of alerts.
* Lack of RCA automation.
For the first one -- it would've been nice if systemd had a "brother" program that could be used for alerts, so that no custom solution was necessary and that services could properly report intermittent problems etc.
The second is contingent on several factors: re-envisioning error handling, declarative debugging and general popularization of the concept. There are several major problems with error handling in system programming languages today. Due to poor language runtime design, programmers learn to think about any and every error as essentially fatal. Recovery is typically seen as impossible, and if there's any sort of recovery code put in place, it's usually the one that tries to do the cleanup and start fresh. It's never really about fixing the problem. So, popularizing something like Common Lisp restart system with the ability to traverse the program stack back to the failing frame would've been a good first step in this direction.
Declarative debugging, on the other hand, could be taking another step forward, where it could be made into a separate program which describes a complex recovery scheme. I.e. the idea of declarative debugging is that the programmer needs to describe the program in terms of constraints that should be checked when the program fails to identify the problematic place. The step forward would be to add automation for the cases when the failed constraint is discovered.
Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it.
But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.
I'm in the storage business, probably about a decade. This is the first time I hear that database management systems don't bother with recovery... where are you getting this from? I'm pretty sure it's the exact opposite.
Nobody in system (storage / network / compute) or any other specialized highly-reliable system -- unlikely.
In highly reliable systems this is already being done, except it's usually an afterthought, ad hoc, functionality added without any system or framework, and so it's hard to judge its performance because it's often partially interactive, and very uneven across different components.
Any system that advertises itself as "HA" (for highly-available) will have some sort of a protocol for complex error recovery, except it's usually not a single program that's explicitly written with the purpose of definition and documentation of recovery process, instead different bits of this functionality are embedded in many different places across the entire surface of a program.
A very common example that I had to implement myself multiple times: HTTP errors. These are usually very poorly reported by the HTTP client libraries. TLS handshake failed? -- you get some numerical exit code instead of proper error report, and so once you realize that this error happens every so often, you start writing code that digs up the internal state of the TLS library at the time of the error, finds the certificates involved and tries to trace down the chain of events leading to the problem.
Similarly, various parsing and de/serialization errors... this stuff is usually atrocious in popular libraries. And in order to figure out why some JSON didn't parse the way you wanted, you need to write a lot of code to investigate and report the error properly. The most common cases I've encountered is when a client program expects a server to reply with a structured error message, but the message comes back botched, or an HTML page is sent instead etc. A lot of these things are quite predictable after some time of using the client, but in a situation where you wanted to properly report an error that was supposed to be encoded in some JSON message and instead you got an HTML page -- typically you end up reporting a wrong (and useless) error of failure of parsing HTML as JSON. To prevent that, you'd need a complicated recovery scheme that would anticipate such errors.
I think the feature exists primarily because there is quite a lot of flaky software out there that gets „miraculously cured” with a restart. Not having to nanny them until a proper fix is a boon, especially when you know the proper fix isn’t coming ever, or is not possible.
That said, there exist options to determine that restarts are happening too quickly, indicative of a misconfiguration or that it just won’t work, period. See Timeout*.
If this is not what you want, you can always change the restart policy, or specify the maximum number of restarts.
Could be, though, that some software ships with crappy defaults which make it crash all the time and systemd just always restarting it and reporting everything is ok. Not a problem of systemd per se, though.
I had the exact same challenge with my daemons/scripts that could crash on bugs of mine.
My solution: (Shameless plug) Create a small daemon that monitors the systemd-journal and email me a summary every 5-minutes (if any and not on blacklist).
This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting.
This is just another reason to track and alert on error rates.
It also depends how sensitive you are to failures, and required availability.
But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.
Observability is so key for successful infrastructure. A poor infrastructure with good observability is going to be more successful with than good infrastructure and no observability
Auto restarting is actually intended to hide problems! ...from the user. The user, who probably isn't you if you're developing or administrating a service.
Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it.
The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting.
And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia.
If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted?
At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required.
Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically.
> and alerting on set thresholds that seem aberrant.
I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not.
I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
>However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not.
Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
Yes. I mean anomaly detection based on identifying what’s unusual behavior not just typical.
We have computers to use their CPUs, RAM, and disk. Using it isn’t weird.
Detecting actual unwanted or dangerous behavior is bad.
For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”.
RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health.
CPU at 90 is not one of them.
As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
I like to investigate every random nuisance error and restart, if I have nothing else to do. Otherwise, any system simple enough to be 100% but free without millions of dollars of work and avionics level formal proofs, might be a system that could probably be replaced with a yellow legal pad and a phone call and might not even need to exist.
What I hate is that everything with systemd requires software it access (instead of plain text log files). It would be nice if there was a way to configure log file type/format, like "ICanHazPlainLogs=yespls"
And tbis is why we have logging system. A decent automated query of your syslog output should be able to show you any flapping services. A setup like this gives you not only self-healing (as described in the post) but also visibility into erroneous systemd units.
I always thought a good monitoring system had three components. Polling, streaming data and log analysis. A lot of times folks don't bother with the logging and miss issues like what is described in the post.
Yea I was a little surprised to see the comment "It wouldn't hurt to look at the logs or the metrics, just in case" paired with the previous comment "there's not quite enough information exposed in the Prometheus host agent's systemd metrics to make it easy".
Not everything needs to be monitored in a single way via a single platform. A cron job that greps/awks syslog for relevant strings and sends a slack message, email, sms, something would be a step function in the right direction.
It's even easier than that, the functionality exists inside rsyslog. All you have to do is set up a template in /etc/rsyslog.d and point it towards a mail relay...
if $syslogseverity <= 3 then {
action(type="ommail" server="127.0.0.1" port="25"
mailfrom="rsyslog@localhost"
mailto="root@localhost"
subject.template="mailSubject"
template="mailBody"
action.execonlyonceeveryinterval="3600")
}
While I generally like to dump all logs into elasticsearch/opensearch, it's also really handy to have this set up for one-off machines so that critical/error logs that usually come from things breaking & restarting still get seen.
Semi-related but it's shame default cron in most (all?) kinda fucking sucks at everything.
There is no builtin spreading so */5 job on 500 nodes will get you nasty spike in CPU load. Logging options are pretty much "parse emails and hope for best" and some syslog spam that's mostly noise if everything is working fine. And fucked up environment vars that trip many script-makers. No metrics whatsoever
Oh definitely. This actually bit me a few years ago. Had a distributed set of embedded systems (tens of thousands of wifi routers) that were phoning home which I didn't realize was being done through a cron. It was something we inherited with the BSP. Looking at our analytics for the relevant API zoomed out to 1-5 minute resolutions, our RTTs looked like they were monotonically increasing over time. We had to hookup real time analytics to one of the nodes in our production cluster to be able to catch the spikes at the exact second of the relevant minute boundary.
The solution in our case was to build a tiny launcher for anything cron-related that delayed the script executions randomly within a range of milliseconds.
Because it's written by an experienced and highly-respected academic systems administrator who's been continuously publishing his high-quality, ad-free blog for longer than Hacker News has been around.
Strangely back then the sky did not fall, automatic restarts were Windows’ solution to paper over its poor reliability.
I think we either started demanding more from our services or just accepting worse quality. And with the mainstream accepting silent restarts it’s only going to get worse.
> You do get the node_systemd_service_restart_total metric, which counts how many times a Restart= is triggered, but that doesn't necessarily say why
Wouldn't journald have details? Of course those might not be hooked up to the main reporting flow, but the debugging info is available once you discover the problem
Not if you're monitoring your systems. True engineering means you have a feedback loop into your process. Otherwise you're 'just a programmer'.... :)
You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter.
Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s).
s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/
Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).
Both of your assumptions are completely incorrect. One should _never_ "monitor" a system like that. Fortunately, the wise Unix wizzards of yesteryear already thought of the solution: uptime counters.
An uptime counter counts the number of milliseconds a process is alive. So if it resets to a value lower than the previous recorded one, it's a blue star thats impossible to miss.
TSDBs don't work like that. They generally will choose to store a slope, not absolute values. They're very efficient at storing tons of data like this.
systemd sucks, but this feature isn't the worst thing in the world, even if it's primitive. Probably the best implementation I've seen is in erlang ( https://www.erlang.org/doc/design_principles/sup_princ.html ) in which supervisor processes get notified of child death and can use several different strategies to figure out what to do next (e.g. die and expect the supervisor's supervisor to handle it, terminate all children and restart, ...)
Although you do lose state when a process dies, if your processes are sufficiently lightweight and/or stateless, it's possible for the problem to be irrelevant until the next software release can garden/amend/curate it.
The win that you get in erlang/elixir/etc. is that you can just code the happy path most of the time, and then figure out which sad paths are the ones you need to deal with once you've seen production under load. If you've ever programmed in go, you understand how freeing that is.
systemd has nothing to do with this. In erlang/elixir if you configure your processes to automatically restart and you never check your logs, you'll have exactly the same issue.
I feel it's the same with the whole kerberos / "cattle not pets" cloud mindset.
Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then.
Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.
Definitely have multiple replicas, and recreate misbehaving ones, saving logs and data for later analysis.
If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers.
In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.
This is really confusing when I am trying to debug why a service failed and checking the logs the last errors are because the service is re-starting too quickly.
Looking at the history, I don't think so (though he has high karma value, I don't know if that is considered by the algo since it's secret). His failure rate is pretty typical. He just submits a lot of posts.
Nothing about the 4-paragraph blog posts he makes is advanced. It is in fact a remarkably basic lesson of "monitor your services" that even a 16 year old developer would know. I think he is just used to talking down to idiots because of his job, maybe.
No offense to the dude if he happens to be reading this, but less quantity and more quality please. I'm sure you're very smart.
Alternatively, maybe HN can let us blacklist domains?
107 comments
[ 3.7 ms ] story [ 160 ms ] threadAre the restarts needed because there's malicious over-size content being sent to it, trying to exhaust the server resources?
If so, then "http.MaxBytesReader" might be helpful:
https://github.com/sqlitebrowser/dbhub.io/blob/5c9e1ab1cfe0f...
https://en.wikipedia.org/wiki/Altered_Carbon_(TV_series)
I went a step further and all of my servers do staggered restarts at the start of every month (previously daily/weekly, but once I worked most stuff out, a month is enough now).
No more update related issues catching me off guard when I suddenly NEED to do a restart in short order, no issues regarding manual processes or a service that doesn't start automatically after a restart/crash, or other things that you might not catch otherwise, load balancing etc. Even stuff like restarting the Swarm cluster leader/followers close to one another, yet expecting them to eventually communicate with one another properly and schedule all of the containers, pull them and launch them as needed, join them to the networks and actually communicate properly to them.
Knowing that restarts are inevitable (sometimes after a crash), might as well make sure that they work properly.
And when you discover a new failure mode and something indeed doesn't work, it's nice to test your monitoring and alerting, as well as learn how to deal with that particular failure.
...except when that causes even more problems! one example i remember from a hobby project was a worker service that would make requests to an external service's API. I'd implemented a throttle in the worker to limit the number of external requests per second made against the external service, where the throttle state was stored in process memory and not persisted anywhere -- seemed to be a pragmatic design tradeoff. you probably see where this is going.
there was an exciting interaction where an external API response caused the worker's API client to throw an unhandled exception, which took down the worker service. then systemd would diligently restart the worker service immediately per the restart policy. the throttle working state had been lost, so upon being restarted the worker service immediately fired another request to the external API, which then issued the same API response, which caused the worker service to fail in the same way, ... luckily noticed by systemd and immediately restarted.
combine with a lack of monitoring + alerting and you get a mechanism where your worker service can make about 100,000x as many external API requests over a few days as it was meant to.
You can then also set "OnFailure" to trigger another unit if the failure state is reached, e.g. to trigger a notification.
E.g.:
This is actually part of the self-healing aspect. One way to see if a service restarts is using the following command:
I prefer to have this set as Also note, there are OnFailure to trigger an additional log message, or recovery service, and FailureAction to allow actions to happen, such as the suggested 'reboot' ;-)For reference: https://www.freedesktop.org/software/systemd/man/systemd.uni...
You also don't want to deal with all of them. For this you could otherwise just use journalctl. You also have OOM issues, and this has its own facility.
my preferred way to alert on this is to use the process start time that's included by default in most process-level Prometheus metrics (and is trivial to implement yourself, if you need to):
on a stable system, this metric will be very close to static. you can feed it through the PromQL changes() function to get a count of how many restarts have happened in a given time window.in my experience, for anything with an "alert if it's down for X minutes" rule, you probably also want an "alert if it restarts N times in Y minutes" rule.
"That process that restarts spuriously once in a while has restarted again" stops being noteworthy very quickly if it's deemed low priority, and often even might lead to alerting thresholds being change because it becomes a nuisance.
"That process that restarted every now and again now restarts 5 time as often as it used to" on the other hand is a lot more likely to get attention.
> in the future I'm going to want to remember to check for this
Whenever you think that sentence, you should notice this as a red flag and re-think your approach. You will forget. And if not you, then somebody else in your team. You need automation for things you can forget, otherwise your mental checklists will grow too large to handle and are just a distraction.
Away from things that can be automated and are important you have a checklist and tick things off with a pen. Add $this to the list.
Atul Gawande is worth reading in general and on this topic. [1] He turned it into a book I haven't yet read.
[1] https://www.newyorker.com/magazine/2007/12/10/the-checklist
Yes, and to have alerts, I have resorted to write my own small tools [0][1] at the end of the day. Railgun proved to be very useful, but for smaller things, I'm writing the second one.
[0]: https://git.sr.ht/~bayindirh/railgun
[1]: https://sr.ht/~bayindirh/nudge/
I'd argue that unexpected restarts should alert beyond a threshold. Alerting on every occurrence is too noisy. If an individual unit failure causes a service disruption architecture improvements are needed.
So, I can see two different aspects to this problem:
* Lack of alerts.
* Lack of RCA automation.
For the first one -- it would've been nice if systemd had a "brother" program that could be used for alerts, so that no custom solution was necessary and that services could properly report intermittent problems etc.
The second is contingent on several factors: re-envisioning error handling, declarative debugging and general popularization of the concept. There are several major problems with error handling in system programming languages today. Due to poor language runtime design, programmers learn to think about any and every error as essentially fatal. Recovery is typically seen as impossible, and if there's any sort of recovery code put in place, it's usually the one that tries to do the cleanup and start fresh. It's never really about fixing the problem. So, popularizing something like Common Lisp restart system with the ability to traverse the program stack back to the failing frame would've been a good first step in this direction.
Declarative debugging, on the other hand, could be taking another step forward, where it could be made into a separate program which describes a complex recovery scheme. I.e. the idea of declarative debugging is that the programmer needs to describe the program in terms of constraints that should be checked when the program fails to identify the problematic place. The step forward would be to add automation for the cases when the failed constraint is discovered.
Database management systems don't bother with it anymore, your almost stateless deamon has no hope of gaining anything from it.
But wide logging and error identification are things too; and very useful ones. And those plug on the same spots that were created for error recovering. Because of this, those are also not very common on our current system software.
Nobody in system (storage / network / compute) or any other specialized highly-reliable system -- unlikely.
In highly reliable systems this is already being done, except it's usually an afterthought, ad hoc, functionality added without any system or framework, and so it's hard to judge its performance because it's often partially interactive, and very uneven across different components.
Any system that advertises itself as "HA" (for highly-available) will have some sort of a protocol for complex error recovery, except it's usually not a single program that's explicitly written with the purpose of definition and documentation of recovery process, instead different bits of this functionality are embedded in many different places across the entire surface of a program.
A very common example that I had to implement myself multiple times: HTTP errors. These are usually very poorly reported by the HTTP client libraries. TLS handshake failed? -- you get some numerical exit code instead of proper error report, and so once you realize that this error happens every so often, you start writing code that digs up the internal state of the TLS library at the time of the error, finds the certificates involved and tries to trace down the chain of events leading to the problem.
Similarly, various parsing and de/serialization errors... this stuff is usually atrocious in popular libraries. And in order to figure out why some JSON didn't parse the way you wanted, you need to write a lot of code to investigate and report the error properly. The most common cases I've encountered is when a client program expects a server to reply with a structured error message, but the message comes back botched, or an HTML page is sent instead etc. A lot of these things are quite predictable after some time of using the client, but in a situation where you wanted to properly report an error that was supposed to be encoded in some JSON message and instead you got an HTML page -- typically you end up reporting a wrong (and useless) error of failure of parsing HTML as JSON. To prevent that, you'd need a complicated recovery scheme that would anticipate such errors.
That said, there exist options to determine that restarts are happening too quickly, indicative of a misconfiguration or that it just won’t work, period. See Timeout*.
If this is not what you want, you can always change the restart policy, or specify the maximum number of restarts.
Could be, though, that some software ships with crappy defaults which make it crash all the time and systemd just always restarting it and reporting everything is ok. Not a problem of systemd per se, though.
If you need alerts, you should config monitoring for the logs.
My solution: (Shameless plug) Create a small daemon that monitors the systemd-journal and email me a summary every 5-minutes (if any and not on blacklist).
If you're interested (it's BSD2-licensed): https://github.com/mpdroog/hfast/tree/master/contrib/deltajo...
This is just another reason to track and alert on error rates.
It also depends how sensitive you are to failures, and required availability.
But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.
Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it.
The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia.
If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted?
At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required.
Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically.
> and alerting on set thresholds that seem aberrant.
I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not.
I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
You really care about time to fill (speed), and whether your system has gone nuts and is logging faster and faster than normal (acceleration).
Disk usage is X says do little.
Disk utilization rate is Y therefore will fill by YYYYMMDD is great.
Disk utilization rate change has gone off the chart so a process crapped itself and will fill the disk in 5 mins is life saving.
We have computers to use their CPUs, RAM, and disk. Using it isn’t weird.
Detecting actual unwanted or dangerous behavior is bad.
For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”.
RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health.
CPU at 90 is not one of them.
As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
Of course not. Log it, and restart.
I know there are problems in the software I am running, but I don't what to think about it. I want it hidden from myself.
I always thought a good monitoring system had three components. Polling, streaming data and log analysis. A lot of times folks don't bother with the logging and miss issues like what is described in the post.
Not everything needs to be monitored in a single way via a single platform. A cron job that greps/awks syslog for relevant strings and sends a slack message, email, sms, something would be a step function in the right direction.
https://docbot.onetwoseven.one/services/syslog/
There is no builtin spreading so */5 job on 500 nodes will get you nasty spike in CPU load. Logging options are pretty much "parse emails and hope for best" and some syslog spam that's mostly noise if everything is working fine. And fucked up environment vars that trip many script-makers. No metrics whatsoever
The solution in our case was to build a tiny launcher for anything cron-related that delayed the script executions randomly within a range of milliseconds.
It was so common to not restart that many services now have a "clear error logs on startup" policy.
I think we either started demanding more from our services or just accepting worse quality. And with the mainstream accepting silent restarts it’s only going to get worse.
Wouldn't journald have details? Of course those might not be hooked up to the main reporting flow, but the debugging info is available once you discover the problem
You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter.
Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s).
s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/
Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).
a) It's highly likely that your monitoring is not going to spot an occasionally restarting service
b) It's probably a good choice not to waste cycles on that sort of monitoring
An uptime counter counts the number of milliseconds a process is alive. So if it resets to a value lower than the previous recorded one, it's a blue star thats impossible to miss.
Although you do lose state when a process dies, if your processes are sufficiently lightweight and/or stateless, it's possible for the problem to be irrelevant until the next software release can garden/amend/curate it.
The win that you get in erlang/elixir/etc. is that you can just code the happy path most of the time, and then figure out which sad paths are the ones you need to deal with once you've seen production under load. If you've ever programmed in go, you understand how freeing that is.
Usually when a service conks out, operations will just kill it and roll out another instance. Nice, but you have no way of finding out what actually happened then.
Sometimes this is caused by someone looking for exploits (and actually finding one), or other stuff you'd really rather know about.
Operation teams (and automation) have usually a primary mandate of availability above all, not to investigate any possible failure.
If you can't have that (and budget allows) keep unhealthy replicas alive and pull them off the load balancers.
In my experience option one works best, and makes option two redundant. Unknown unknowns are a thing, but that's true even when the unhealthy replicas are kept around.
No offense to the dude if he happens to be reading this, but less quantity and more quality please. I'm sure you're very smart.
Alternatively, maybe HN can let us blacklist domains?