Mistake 3: Putting different people in charge of alert-setting than alert-responding
In the best case, it increases the risk of alarm-fatigue. In the pathological cases, it means the technical system becomes a cross-group battleground for shifting blame and workload.
Hah! That's a very good point. I can think of several instances I've been involved in where the person who decided what triggers the alarm never actually had to respond to the alarm. I guess it's relevant for anything you are building... not just alarms. The end users of the software need to be involved in the building of it.
I think it's a matter of that most elusively difficult-to-engineer qualities: Trust. (Making sure the two people are in the same team are just an easier way to get it.)
The setter trusts that they don't have to micromanage or over-stimulate responders to react and fix the problems. The person dealing with the alarm trusts that the setter isn't crying-wolf or shirking their own duties to minimize alerts.
> The end users of the software need to be involved in the building of it.
That's my personal bugbear at work right now. My team was literally (albeit indirectly) told to stop talking to users and to route everything through the same "proper channels" which we had written off as clueless.
I'm gradually working my way free of "maybe they'll change this time" narrative that's kept me there since my original employer was acquired and merged into the blob.
I've seen this go badly even when the first responders are the ones configuring the alerts.
For example those getting paged for slow app response times want DBAs to be paged for slow query performance who in turn want System Admins to be paged for high disk latency, etc.
These are all reasonable requests but weren't always implemented due to poor communication and/or team relationships.
Pages [alerts] should be urgent, important, actionable, and real.
They should represent either ongoing or imminent problems with your service.
Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring.
You should almost always be able to classify the problem into one of: availability & basic functionality; latency; correctness (completeness, freshness and durability of data); and feature-specific problems.
Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.
Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes.
The further up your serving stack you go, the more distinct problems you catch in a single rule. But don't go so far you can't sufficiently distinguish what's going on.
If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical.
11 comments
[ 3.0 ms ] story [ 35.8 ms ] threadIn the best case, it increases the risk of alarm-fatigue. In the pathological cases, it means the technical system becomes a cross-group battleground for shifting blame and workload.
The setter trusts that they don't have to micromanage or over-stimulate responders to react and fix the problems. The person dealing with the alarm trusts that the setter isn't crying-wolf or shirking their own duties to minimize alerts.
> The end users of the software need to be involved in the building of it.
That's my personal bugbear at work right now. My team was literally (albeit indirectly) told to stop talking to users and to route everything through the same "proper channels" which we had written off as clueless.
I'm gradually working my way free of "maybe they'll change this time" narrative that's kept me there since my original employer was acquired and merged into the blob.
For example those getting paged for slow app response times want DBAs to be paged for slow query performance who in turn want System Admins to be paged for high disk latency, etc.
These are all reasonable requests but weren't always implemented due to poor communication and/or team relationships.
(D) "Here's a truck. Also, it's a BLUE truck."
(CD) "Here's a truck. Also, it's a BLUE truck, not a RED one."
But on a practical level I doubt anybody cares about this contradistinction in terms. :P
—an English major
https://docs.google.com/a/gravitant.com/document/d/199PqyG3U...
Particularly:
Pages [alerts] should be urgent, important, actionable, and real.
They should represent either ongoing or imminent problems with your service.
Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring.
You should almost always be able to classify the problem into one of: availability & basic functionality; latency; correctness (completeness, freshness and durability of data); and feature-specific problems.
Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.
Include cause-based information in symptom-based pages or on dashboards, but avoid alerting directly on causes.
The further up your serving stack you go, the more distinct problems you catch in a single rule. But don't go so far you can't sufficiently distinguish what's going on.
If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical.