When I started out as a beginner understanding what are metrics, why is Prometheus used was something difficult for me to understand . After spending a lot of time and learning from the material out there I created something that I wish I had when I started learning Prometheus.
PS: This is my first tutorial kind of material, open to feedback on how to improve.
I wonder if it should be used at all for beginners and smaller teams. I thought it is something like the new industry standard but even to get it running in production is not trivial and had the feeling this is something for teams who are deeply committed to monitoring and developing their own metrics, views and alarms.
My conclusion is the majority (or at least smaller teams) are maybe better served with something like Datadog. It's cheaper as long as you don't have a high number of machines. Speaking of that I'm still missing the easy middle ground of monitoring which is modern, easy to use and works for a higher number of machines (and even something like Docker Swarm).
Basic Prometheus, with vanilla storage engine, is quite easy to set up (IMO).
On a Debian system it takes about 15-30 minutes to install and do a basic enough configuration to get it running. Another 30 minutes to get a Grafana instance sitting in front of it - and then it's incremental to get the node_exporter (also packaged up) out to your systems.
The two time consuming bits are a) customising the metrics that you can scrape, though there's plenty of templates out there, and if you're not trying to do any fancy application-layer metric scraping you're probably going to be happy out of the box.
And b) evaluating and then setting up longer / larger storage engine on the backend - but if you've committed to Prometheus after doing your low-effort PoC, then this is a simple cost benefit evaluation.
Metrics-as-a-service are an interesting proposition, but I'm very cautious about them
> Speaking of that I'm still missing the easy middle ground of monitoring which is modern, easy to use and works for a higher number of machines (and even something like Docker Swarm).
What's modern mean in this context, and just how easy do you want?
In my experience monitoring systems are relatively easy to deploy, unless you're huge. The difficult bit is working out (with platform owners) everything they need to monitor, what their key health indicators are, and what thresholds to apply.
IMHO, Prometheus is not complicated. In fact, it was designed to be simple and dumb. I mean, it does not provide any production grade scalability out of box, and people have to implement their own add-ons and external storage solutions (thanos, cortex, M3DB...) to make Prometheus even usable in big clusters. Just because you can buy DataDog and be done doesn't mean the paid product itself is easy to understand.
What I like most about Prometheus, are PromQL, pull-based model, and its open source ecosystem. I use it both at work and in my homelab (which is just a bunch of Intel NUC bare metal boxes), and it has been great. I can't speak for any other individuals or companies, but there are tons of learning materials and active community where you can get help.
Also, when it comes down to application level metrics, you have to educate your developers to add instrumentation in their own code anyways. And I think exposing a small HTTP server is far easier than pushing data to a remote server.
I never found Prometheus that difficult. I would say I've spent a moderate amount of time over the past few years reading and using it (almost exclusively just reading the docs and blog). I haven't used Datadog. The places I've used Prometheus tended not to use Saas, which meant trying to get the wallet opened or approval for sending data offsite were basically non-starters. I know that's not an issue for most here. I set it up as a side project because we were basically blind and reactive to issues. The other options I was looking at was Nagios and Cacti.
Machine metrics are basically plug-and-play. The only real downside with Prometheus is that you have to install the pieces yourself (and know enough to do it). After you understand how the pieces fit together and get basic knowledge of each piece the time suck is writing and grokking new metrics and noodling with a dashboard--which I figure any solution would have.
This is probably home-hobby geekery, but I'm running Prometheus at home. My Synology has Prometheus and Grafana running in docker containers. The only gnarly part was getting dd-wrt's SNMP data and creating a dashboard of router metrics.
Prometheus is just a storage engine, with a debugging page to display one metrics (it's what you'd get if you did a one week hackathon to make a metrics system really).
Datadog is a complete solution to collect metrics, visualize, dashboards, search and alerting. There is really no comparison possible (if datadog is a car, prometheus is an engine).
Prometheus is easy to setup but it barely does anything. It gets really messy when one needs to gather application metrics (postgresql, cassandra, haproxy, java, etc...) and have dashboards and alerts going to different places (slack, email, etc...). Trying to find and combine a variety of plugins and open source tools to fill the gaps.
Any company irrelevant of size should have a metrics/dashboard/monitoring/alerting solution. IMO datadog because it's better in all aspects. If the company doesn't want to pay or doesn't want SaaS, it should at least setup prometheus to have basic system metrics somewhere, rather than flying blind.
prometheus is an amazingly versatile, performant, and high quality piece of software that does more than just storage (eg. service discovery, scraping, etc).
it’s tsdb engine (specifically the new one in 2.0) is pretty much state of the art.
i don’t think it’s fair to say that it’s the kind of thing that could be created in a one week hackathon.
I tried to suggest something off the shelf to our place, as we have a lot of code and a small team. Our prima donna tech lead insisted that we need something custom for one small case (like everything) so we have Prometheus.
Prometheus scales down very poorly for small teams. Other monitoring systems are effective in few hour of work and require installing only one component.
Prometheus requires a plethora of tools around it to achieve feature parity and plenty of manual work to create dashboards, alerts and so on.
Not to mention the need for custom exporters for each application (!), contrasted to agent-based platforms that have built in plugins and autodetect what need to be monitored.
That's also my conclusion. I want something of the shelf which offers some basic monitoring and working alarms to my email for small or tiny projects. I guess it's also doable with Prometheus if you figured it out once and then reuse the configs/templates then <- not everybody has time (or man power) for that though.
If you use GCP's Monitoring for dashboards and alerts on metrics from applications running in GKE, the easiest way to collect those seems to be running Prometheus with a sidecar container that ships Prometheus' metrics to StackDriver [1].
We're running such a setup currently, i.e. we installed Prometheus via Helm, using the Helm chart's sidecar config option to add the sidecar container, and the rest works more or less automagically (k8s workloads with Prometheus endpoints are detected based on annotations and their metrics show up in Monitoring)
We have somewhat relaxed availability constraints for the project this is part of, so we're using the opportunity to gain some experience with this setup and find out whether it's good enough for highly-available prod clusters. So far it's all nice and easy and works very well.
I think Prometheus works well for getting infra metrics, but getting application metrics in Prometheus is non-trivial.
DataDog on the other hand, provides complete application monitoring and tracing.
Agree with you that teams who want to get things up & running quickly should try tools like DataDog which are simpler to get started with
This doesn’t seem to touch the subject of Promgen (or other Prometheus configuration management software that I’m not aware of). Configuring it via yaml files, especially the alerts in alert manager and queries in the sql exporter, requires more steps than I’d like to do. Any recommendations on how to manage Prometheus configs? Especially having multiple Prometheus instances where the master node scraps all the data from other nodes
You can run and interface with statsd_exporter to switch your Prometheus to a push-based model.
If you want to get away from Prometheus, InfluxDB and its TICK stack is great.
VictoriaMetrics ingests data in a variety of popular formats and offers a superset of the Prometheus query language to query the data. Super easy to run.
25 comments
[ 3.3 ms ] story [ 63.9 ms ] threadPS: This is my first tutorial kind of material, open to feedback on how to improve.
My conclusion is the majority (or at least smaller teams) are maybe better served with something like Datadog. It's cheaper as long as you don't have a high number of machines. Speaking of that I'm still missing the easy middle ground of monitoring which is modern, easy to use and works for a higher number of machines (and even something like Docker Swarm).
On a Debian system it takes about 15-30 minutes to install and do a basic enough configuration to get it running. Another 30 minutes to get a Grafana instance sitting in front of it - and then it's incremental to get the node_exporter (also packaged up) out to your systems.
The two time consuming bits are a) customising the metrics that you can scrape, though there's plenty of templates out there, and if you're not trying to do any fancy application-layer metric scraping you're probably going to be happy out of the box.
And b) evaluating and then setting up longer / larger storage engine on the backend - but if you've committed to Prometheus after doing your low-effort PoC, then this is a simple cost benefit evaluation.
Metrics-as-a-service are an interesting proposition, but I'm very cautious about them
> Speaking of that I'm still missing the easy middle ground of monitoring which is modern, easy to use and works for a higher number of machines (and even something like Docker Swarm).
What's modern mean in this context, and just how easy do you want?
In my experience monitoring systems are relatively easy to deploy, unless you're huge. The difficult bit is working out (with platform owners) everything they need to monitor, what their key health indicators are, and what thresholds to apply.
What I like most about Prometheus, are PromQL, pull-based model, and its open source ecosystem. I use it both at work and in my homelab (which is just a bunch of Intel NUC bare metal boxes), and it has been great. I can't speak for any other individuals or companies, but there are tons of learning materials and active community where you can get help.
Also, when it comes down to application level metrics, you have to educate your developers to add instrumentation in their own code anyways. And I think exposing a small HTTP server is far easier than pushing data to a remote server.
Machine metrics are basically plug-and-play. The only real downside with Prometheus is that you have to install the pieces yourself (and know enough to do it). After you understand how the pieces fit together and get basic knowledge of each piece the time suck is writing and grokking new metrics and noodling with a dashboard--which I figure any solution would have.
This is probably home-hobby geekery, but I'm running Prometheus at home. My Synology has Prometheus and Grafana running in docker containers. The only gnarly part was getting dd-wrt's SNMP data and creating a dashboard of router metrics.
Datadog is a complete solution to collect metrics, visualize, dashboards, search and alerting. There is really no comparison possible (if datadog is a car, prometheus is an engine).
Prometheus is easy to setup but it barely does anything. It gets really messy when one needs to gather application metrics (postgresql, cassandra, haproxy, java, etc...) and have dashboards and alerts going to different places (slack, email, etc...). Trying to find and combine a variety of plugins and open source tools to fill the gaps.
Any company irrelevant of size should have a metrics/dashboard/monitoring/alerting solution. IMO datadog because it's better in all aspects. If the company doesn't want to pay or doesn't want SaaS, it should at least setup prometheus to have basic system metrics somewhere, rather than flying blind.
it’s tsdb engine (specifically the new one in 2.0) is pretty much state of the art.
i don’t think it’s fair to say that it’s the kind of thing that could be created in a one week hackathon.
Prometheus requires a plethora of tools around it to achieve feature parity and plenty of manual work to create dashboards, alerts and so on.
Not to mention the need for custom exporters for each application (!), contrasted to agent-based platforms that have built in plugins and autodetect what need to be monitored.
Cloud you maybe link some of those other tools, because my team is currently looking into monitoring software for our applications.
Telegraf makes easy work of monitoring all kinds of things and has a built-in StatsD server.
Chronograf has a nice GUI for browsing through your metrics and can GROUP-BY and filter on a specific value with just a click or two.
We're running such a setup currently, i.e. we installed Prometheus via Helm, using the Helm chart's sidecar config option to add the sidecar container, and the rest works more or less automagically (k8s workloads with Prometheus endpoints are detected based on annotations and their metrics show up in Monitoring)
We have somewhat relaxed availability constraints for the project this is part of, so we're using the opportunity to gain some experience with this setup and find out whether it's good enough for highly-available prod clusters. So far it's all nice and easy and works very well.
[1] https://cloud.google.com/monitoring/kubernetes-engine/promet...
Maybe ask over at https://reddit.com/r/PrometheusMonitoring
I was extremely happy with this setup, although influxdb is a bit lacking (it doesn't allow summing up two metrics for example).
If you want to get away from Prometheus, InfluxDB and its TICK stack is great.
VictoriaMetrics ingests data in a variety of popular formats and offers a superset of the Prometheus query language to query the data. Super easy to run.