Ask HN: Best monitoring system?
At my company I'm considering switching us from Nagios to another monitoring system and starting to do some research. What's the best monitoring solution out there today? I'm pretty impressed by Prometheus, but just like to get some more opinions.
127 comments
[ 0.31 ms ] story [ 157 ms ] threadhttps://news.ycombinator.com/item?id=12073675
You want metrics from counters you build in your app? (see statsd?)
You want to aggregate and do analysis on logs? (see ELK stack?)
You want to monitor cloud infrastructure (see stackdriver?)
You want to run end to end tests on your application to ensure it's behaving? (see runscope?)
As your application grows, you probably want a blend of tools to see inside your app.
My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to ideal as I can come up with.
Of course, that requires a fair bit of up-front knowledge to stand up and operate, but they're so rock solid (and scale like mad) I have a hard time not recommending them.
Why in the world would you want to manage (which I'm reading as throttle) the ingest rate of a monitoring service? That strikes me as a recipe for missing important events.
Monitoring should be a fairly stable load rate. If you're setting up monitoring, and your machine can't handle the load of all the data points coming in, you need to shard out, not drop data points.
> Active polling scales _better_ than the alternative
Exceptional claims require exceptional evidence. Personally, I have never heard of anything that is actively pinging outside services performing better than receiving and processing data passively.
Prometheus will scale better than Nagios running active checks since it won't be using subprocesses, but it is still going to require more overhead than a service receiving passive reports.
A single big Prometheus server can easily do millions of time series, and in an older record, 800,000 samples stored per second. You could monitor e.g. 10,000 hosts with that with quite some detail, and the bottleneck is still not the pull aspect.
If you need to shard, no other binaries are involved, just a couple of changed configuration files and some SSL certs. Passive reports can go to any node; a simple load balancer will work fine without any form of hashing or federation required.
> There is nothing inherent in a polling model that limits its ability to scale
Aside from the additional processing requirements, such as SSH, NRPE, subprocesses, etc, you are also limited in that a polling process must be told in advance of new systems or services, whereas it's fairly easy to just have a new service or system start reporting and be immediately monitored.
How do you re-combine and alert on data for one service that got reported to multiple nodes? This would imply to me a database of some form, which is going to need hashing or federation or other distributed systems approaches to be scaled out.
> Aside from the additional processing requirements
Push or pull, there's additional processing requirements when you scale. It's the same bytes on the wire and broadly the same amount of processing power required.
> you are also limited in that a polling process must be told in advance of new systems or services
That's not a scaling limit, that's a more fundamental issue that isn't different between push and pull.
For a push system you need to have a list of all systems and services in order to be able to alert on systems that never reported, or are no longer reporting.
https://www.circonus.com/pully-mcpushface/
Pull works fine at 2 million machines" is a great statement to make, but it would be much stronger with more details. How many machines are doing the pulling? How often? Are they using subprocesses or threads or greenthreads? How do they handle timeouts? How many metrics per machine? How many pulls per metric grab?
We also have an FAQ in Prometheus about why we prefer pull: https://prometheus.io/docs/introduction/faq/#why-do-you-pull...?
In my experience, pulling is operationally much nicer than pushing, and I've worked with both. It also gives you somewhat less of an accidental DDoS exposure.
> [Grafana+Ichinga] they're so rock solid (and scale like mad) I have a hard time not recommending them.
As a Prometheus developer I have seen a significant number of users who moved from Graphite because they found it doesn't scale and was far from rock solid for them, requiring regular manual care and feeding. By contrast Prometheus seems to be working pretty well for them at what we would consider to be a moderate load. I've heard similar about Nagios/Ichinga.
Push vs. pull is largely not relevant for scaling (pull is slightly better in this regard, but only slightly). I've been involved with some extremely large scale monitoring systems, and the fact they were pull was never relevant to scaling them.
May I ask what you consider to be a high level of scale?
> It won't scale without a number of workarounds.
There are very very few systems who won't scale without workarounds. That's the nature of scaling a non-trivial system.
Can you afford time but not money? Try Sensu or Nagios.
Do you have money and not time? Try datadog.
Like someone else mentioned here, if you're looking to alert off of logs from ELK, try Elastalert.
If you have money and no time: DataDog
If you don't mind putting in a little time: Sensu
Sensu is straightforward to deploy if you use Chef/Ansible/Puppet. It also supports running Nagios plugins which is pretty useful.
I also disagree that setting up Sensu takes a "little" time. What is a "little" to an inexperienced Sensu administrator? A day? A week? Several weeks? Quantifying it would be valuable to the reader.
Good documentation, UI, many, many plugins and fair pricing (IMO).
https://www.datadoghq.com/
(Im not affiliated with in any way other than using their product on a pet project with many moving parts).
As far as Datadog goes, it's the most team friendly dashboard system we've used. We had a specialty monitoring system for one application stack previously, and no one made custom dashboards there or even just looked at the data. Now we've got custom dashboards out the nose and we're gradually consolidating to a "best of" dashboard for each service.
Are you affiliated with Wavefront?
(I've learned not to trust numbers that seem too good to be true unless they're contractually obligated.)
Wavefront doesn't publish pricing, but if we take Librato's pricing as a general indication you're talking several million dollars a month.
I used it up until two months ago when I left that job. There was 2000~ servers monitored I think.
I'm asking because I will be designing a similar page soon (that's also billed per host) and I'd like to avoid the same mistakes.
1. VERY clearly state that when you sign up for the service, then you are on the hook for up to $18*500 = $9000 + tax in charges for any month. Even Google compute engine (and Amazon) don't create such a trap, and have a clear explicit quota increase process.
2. Instead of "HUGE $15" newline "(small light) per host", put "HUGE $18 per host" all on the same line. It would easily fit. I don't even know how the $15/host datadog discount could ever really work, given that the number of hosts might constantly change and there is no prepayment.
3. Inform users clearly in the UI at any time how much they are going to owe for that month (so far), rather than surprising them at the end. Again, Google Cloud Platform has a very clear running total in their billing section, and any time you create a new VM it gives the exact amount that VM will cost per month.
4. If one works with a team, 3 is especially important. The reason that I had monitors on 50+ machines is that another person working on the project, who never looked at pricing or anything, just thought -- he I'll just set this up everywhere. He had no idea there was a per-machine fee.
Thinking you could get away with your current setup for $15 is very naive of you. It looks like you red it well but your refused to acknowledge it?
FYI prices for some popular services:
- BMC (direct competitor to datadog) is $12 per host per month
- SignalFx (other competitor) is using a $ price per metric stream per month (this one is tricky but they suggest to plan for $12 per host per month)
- NewRelic is $75 per host per month
- Dynatrace/Appdynamics are about $1000 per host per month
- AWS/GCE have pricing per hour (which is annoying to convert per month)
I agree on one thing though (and I did tell that to datadog guys directly). The price displayed in big should be $18 per month per host and not $15. The cheaper one if for an annual subscription which is not the most common case.
It uses Nagios under the hood, it's basically an automation system that generates those Nagios systems. The GUI is amazing, because it uses a plug-in so you don't have to edit files on disk to group your hosts or tweak the alerts. Those configs are snapshotted automatically at every change, and you can replicate that configuration automatically to remote servers. Download it from the upstream site instead of relying on distro package repositories.
Caveat, the documentation sucks, the GUI can be nonintuitive and it's hard to Google problems. It takes time to fully tune. Out of the box you'll probably still be impressed though.
The English documentation isn't that great, the German one is better. That being said, it's mostly self-explanatory and all checks are very well documented in man pages.
My favorite features:
- Auto discovery for literally everything, including SNMP interfaces.
- Fine grained rule system for customizing check threshold and parameters
- All configuration is automatically versioned and you can integrate it with Git - this includes the changes you make in the web interface.
- It's very easy to set up a distributed monitoring system (multisite) with a central node which aggregates all states and replicated configuration changes, yet each site is fully autonomous.
- The agent takes zero network input, so no attack surface.
- Even though it's extremely featureful, it's architecture is very simple and it's easy to contribute code and write custom checks.
Their Git is public: http://git.mathias-kettner.de/git/?p=check_mk.git;a=shortlog...
It works well with Naemon and Nagios 4. Been using it for a number of projects, ask me anything!
Monitoring systems not to use:
I like Icinga a lot. I won't bother reviewing it; is is very well known. Professionally, my last two gigs have used Zabbix.
Zabbix, architecturally, is a nightmare. Uses an RDBMS for storing time-series data, so it wastes a ton of space on historic data while managing to be far slower than it needs to be when querying larger ranges. Uses an agent. Has a proxy-agent that, while handy, encourages all sorts of sketchy, error-prone monitoring topologies. With 3.0, the UI has crawled out of the awful range, and is now merely annoying. Takes the all-singing, all-dancing monolithic approach for the main app, including features for drawing maps on big-screens.
For all that, it works well. Give it the hardware it wants, be sane in setting it up[1], ignore the goofy features (maps, inventory, screens - I guess someone must of requested those), and it is very solid and very powerful.
[1] The template system, pseudo-language for triggers, naming convention for variables and method of creating custom monitors take some getting used to. Expect to take the time to actually read the docs, and most likely to throw out your templates the first time you model your systems.
Even if you don't have access to the server so you can monitor it, you can use the "host" concept as containers for your services: "api.mycorp.com", "tasks.mycorp.com", "backups.mycorp.com" are great starting points.
Actually, no.
When you monitor state of a cluster (e.g. node count), you don't have a server, you have plenty of servers and a cluster (completely different thing).
When you monitor temperature in your server room, you don't have a server, you have a server room.
When you monitor exchange rate, you don't have a server.
When you monitor a website, you still don't have a server.
And now add all the AWS Lambdas and other serverless rage.
Notion that everything works on a (single!) server was never valid, and today it's even more visible than it was twenty years ago, when Nagios was state of the art.
What matters is that the alert about the issue is raised and relayed to the proper notification channels. Since sensu doesn't concern itself with a fancy dashboard, it doesn't really matter if the alert pertains to the host or not.
Any decent monitoring will have customized the alert handling based on what's alerting, so there's some amount of post-processing possible.
In practice it doesn't matter if you name a file handle "juju" and a database query "peach" in your code.
It's a matter of calling things what they are instead of forcing them into a mismatched data scheme by creating artificial hosts.
Borg inspired Kubernetes. Borgmon inspired Prometheus. So naturally it works well together with a dynamically scheduled world.
This is yet another point where DevOps is not "devs doing ops" but "operations building and deploying with all the tools of modern software development". You need a subject matter expert.
What are you monitoring? Do you care about availability or performance or both? Scale? Do you have services or servers? Do you manage the underlying hardware? Do you need to track which hardware boxes have which VMs or containers?
There are a million questions to answer. One big set of them: what do you dislike about Nagios? Make sure that you don't get those problems with the next one, but also make sure you get something that does what you need as well as what you want.
Benefits: shared expertise. Common language. Propagation of alerts up from hardware and down from services. Better root cause analysis. If you have a good culture, faster resolution time and better understanding.
Spot on. Too many people think that monitoring is about slapping a piece of code on some hosts. Monitoring is data science.
I'm also interested in prometheus but haven't gotten to try it out yet. Anyone reading this have experience with both? How do they compare?
EDIT
Should have mentioned how well it pairs with Grafana ;)
Also take a look at Riemann which is system monitoring written in Clojure. Riemann should be good for monitoring latency of the system.
If it helps here is Slidedeck from Spotify how they do their monitoring https://www.netways.de/fileadmin/images/Events_Trainings/Eve...
I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console.
Glad you enjoyed the book!
For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it.
I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.