the problem is you will probably think of monitoring the stuff that you wished you had monitored only _after_ the crisis happened, people are built in a way that makes it hard for us to think of what may go wrong.
So another simple rule I learned with time is to trust/understand the defaults,plugins,knobs,metrics that come with well known monitoring systems ("why the hell should I monitor _that_?"). This way you use the experience of other people as a backup for your own.
What I have learned: Take Munin (or your solution of choice) and install all plugins for infrastructure to use. It's hard to monitor too much, only too little.
How about starting with the application/business metrics first (as those are presumably easier to articulate). As things fail over time move down the stack (infra/system) to get earlier warnings?
This is a really good introduction to the different levels of infrastructure monitoring and their various pros/cons.
http://librato.com (full disclosure: I hack there) is a startup with a new kind of entry in the process-level monitoring/management space. Would love any feedback the infrastructure-minded part of the community here might have.
This is great! Very clear explanation. Have you looked at AppFirst's (http://appfirst.com) solution? (fyi. I work there)
We have a monitoring solution that covers the application, process, and server level as well as integrating with nagios to cover any other level you have scripts for. It would be great to hear what you have to say about us!
6 comments
[ 3.4 ms ] story [ 17.4 ms ] threadRule of thumb: Think of everything you wish you had monitored after a crisis has happened (which might be "everything").
So another simple rule I learned with time is to trust/understand the defaults,plugins,knobs,metrics that come with well known monitoring systems ("why the hell should I monitor _that_?"). This way you use the experience of other people as a backup for your own.
http://librato.com (full disclosure: I hack there) is a startup with a new kind of entry in the process-level monitoring/management space. Would love any feedback the infrastructure-minded part of the community here might have.
We have a monitoring solution that covers the application, process, and server level as well as integrating with nagios to cover any other level you have scripts for. It would be great to hear what you have to say about us!