Does the new plugin architecture allow me to provision plugins from a config file on startup? Like installing the Kubernetes App without `grafana-cli plugins install kubernetes-app`
Are there plans for letting power-users style their own custom panels using html, css and js using just the web interface? Meaning without having to develop a plugin.
A few things have been added that may help:
1. only visible data is queried and loaded
2. If you have multiple panels that use the same data, you can now just reuse the results (or a subset of the results) in a new panel. This will cut down on additional queries that may not be necessary
First off, Grafana is amazing! So keep up the good work.
Question: Currently alerts are fairly limited both in applicable panels and how they export/share relevant data to the chosen notification channel, as well as more advanced conditions. Is there any work going on with improving alerts?
Also to mention frequently asked feature: alerts using queries and templated variables [1][2]. Creating even simple generic dashboards with alerts is not possible, unless variables are hardcoded in hidden queries.
Agreed. If Grafana alerts supported templated variables I would migrate a decent chunk of my monitoring stack. But without templated variables the alert system feels half baked and I almost prefer it not even be a feature.
The cost aspect is one thing. Vendor lock-in another. Probably need to have a team that can lead the transition towards Grafana, educate and speak for it. Grafana has an enterprise plugin for datadog that potentially could ease the migration
We just moved everything from datadog to Grafana, Prometheus and Loki, mainly due to cost. Honestly though, I feel like we get more out of the Grafana stack. Its powerful but simple enough to keep it pleasant to work with.
Haha, my team is going the opposite direction: we're leaving prometheus + grafana to datadog, and we're loving it. having every AWS monitoring default dashboard work right out of the bat is a bliss..
How about don't? Datadog offers everything that grafana does plus alerting, as well as more and better integrations (databases, AWS, middleware, etc...).
have you seen any implementations of internal tracing like APM being used with Grafana? I think lack of a good open source alternative to New Relic or Datadog APM kinda killed OSS monitoring for my company.
The Jaeger integration is a good step in the right direction though, kudos!
AFAIK, You can have internal trace markers, not just between apps. I've used that (for Jaegger) in one of our Windows Application to see the call-graph (it's a level editor for our game), and it was tracking the loading of various resources across multiple threads, then it was visualized with jaeger and even zipkin.
I'm also colorblind and a heavy Grafana user but I haven't had much difficulty reading dashboards. What are you having trouble with? The most trouble I've had is colors of multiple lines on a chart being too similar but being able to use the mouse to select a line and see what the line is and its value helps there.
The default palette is definitely a challenge, since it has pretty low contrast and Grafana's defaults use a lot of thin lines.
I especially have trouble matching colors in a legend with the drawn objects in the graph. The best help has been the hover panel, but it's unsorted by default and the highlight contrast on the panel for hovering over a line is pretty dim.
Upvote: https://github.com/grafana/grafana/issues/18041 and add any additional comments and context there. In addition to making the light/dark themes more accessible, I think we should add an additional high contrast one.
Hi Torkel,
Just finished watching the GrafanaCon 7.0 Release talk. There are some great Usability improvements and new functionality which will be definitely useful for my upcoming project.
We have all sorts of data flowing into Prometheus and charted in Grafana. Grafana lets me alert based on thresholds and that works fine for many things.
EXAMPLE 1:
Imagine a service. Load avg of 2 is unacceptable and means things will be slowing down. However, a heavy job runs every night and pushes load averages between 1.5 and 2.5 during a 30 minute window.. I wanted to set a load avg alert threshold @ 2.0 but can't since it would fire false alarms too often. So i end up setting it at 3, and end up getting a late alert when things get really bad.
Instead, an anomaly detection would look for a number of standard deviations away from a mean, and ideally account for seasonality (i.e. compare the data with same data last few weeks). Now, things will behave better. If loads hit 2 during the day, it'll fire. But during that 30 minute window at night, loads would need to exceed 3 before it fires.
EXAMPLE 2:
Let's say I'm tracking 30 metrics for a service. I understand the service well enough to set thresholds in 5 metrics. However, I'm sure a failure won't happen alone and the other 25 metrics may show some oddities and give me an early warning of "something wrong happening" .. Here, some kind of loose "anomaly alerts" could help as an early warning. If a certain metric throws too many false alerts, we find a better way to monitor it.
EXAMPLE 3:
Lets say you have a service that auto scales. There's no limit to how many requests the service handles. So I can't really thresholds on requests/5min. Here, some kind of "anomaly alert" would be useful to let the team know that something is going on that's causing a high level of requests. Again, accounting for seasonality because things tend to change on weekends.
At the end of the day, an anomaly alert is not some "magic solution" but just and extra tool in the toolkit to use when it makes sense. Datadog and AWS both have built in easy to use anomaly alerting. Would be nice for grafana/prometheus to have something similar.
At work we have to compile everything from source because compliance, and Grafana is one of the harder ones to get right. There used to be good documentation for it a few versions back but I can't seem to find it lately. Is there any good documentation on how to get Grafana compiled from source?
I've got a complicated proxy setup to contend with and have to build via Jenkins on some older Linux boxes, so it's just trial and error. You have a build script you'd be willing to share?
Just wanted to say thanks for an awesome piece of software. It's one of those "I can't believe it's doing it and doing it this well, and it's free" kind of projects. For me, Grafana together with ElasticSearch and InfluxDB+Telegraf is pure monitoring/analysis/alerting bliss. Stay true to your roots and don't get tempted to turn into Kibana ;)
Is it now possible to create a horizontal lined based on a single value (either a fixed or a computed number)?
For example when I'm tracking the power grid frequency, I would like to add two lines at 50.05 and 49.95 Hz, as well as one at 50 Hz (with different color).
Others have voiced their desire to display a horizontal line at the 95th percentile in a bandwidth usage graph.
when you post release announcements that possibly get posted snd forwarded to other places, please include a short description about what grafana is.
and maybe also add one to your home page. (make it link to the features)
when i see any release anouncement about a product i am unfamiliar with, the first question i have is what kind of product it is, so i can decide if i want to check it out
I have a mongodb database with several collections. All collections share a set of fields: "createdAt", "channelId", "status", etc.
How easy would it be to dump all this data into Grafana, and then be able to slice and dice these collections by any combination of the shared fields.
Basically, I'm tired of writing mongo queries whenever I want to inspect my data. I want to point and click around a beautiful UI, with data visualizations and easy filtering/aggregation. Is this easy to do with Grafana?
If you have mongodb enterprise you can use Mongo bi connector and connect it with any bi tool. I recommend metabase since it is open source and very good for slice and dice and reports.
The mongodb bi connector is a postgres with foreign data wrapper to mongodb. So even if you don't have a mongodb subscription, you may be able to pull this off using the OS version https://github.com/EnterpriseDB/mongo_fdw
Hi Torkel, this issue has been open since 2016[1]. There is incredible demand for it yet complete silence from Grafana on committing to the effort. Can I get official word on this please?
Have there been no efforts to add this with a PR? I mean Grafana is open source and if the project leads wants to focus on other things that's his choice.
It has no support for rrdcached flush, AVERAGE CF is hardcoded , and I cannot seem to get it to not connect lines through gaps where there were no measurements.
How about adding support for configuring Grafana with files?
This would allow us to deploy immutable Grafana instances using infrastructure-as-code tools like Terraform and Docker. Specifically, I want a way to eliminate post-install steps such as adding users and organizations and setting user passwords.
Wouldn't a better way to configure users and passwords be to use an external auth provider such as OAuth or LDAP? If you don't already have a auth provider, you could deploy Keycloak as your user DB.
OAuth/LDAP are great for large organizations. Small organizations don't need such complexity.
Even if you use OAuth/LDAP for accounts and passwords, Grafana still requires manual steps to create the organization, install plugins, and set the admin password. I wrote a Python script to do this on Docker container startup. That script is a workaround for Grafana server's poor ops design. It's extra code that we must maintain.
Do you plan to make debug logging useful? I had difficulty troubleshooting Grafana & InfluxDB auth problems and found that debug logging did not provide any extra detail.
Thanks for adding more support for logs in Grafana.
I'm not using AWS CloudWatch Logs because of the financial risk of a runaway server running up a huge AWS bill. I had difficulty getting log display and log search working well in Grafana 6.6.2 and InfluxDB. Do you plan to improve log support for other databases?
Custom Themes seems to be one of the most wanted features of the Community.
Is there any plan implementing it in the near future?
Recently I digged into the source code to find out how easy it is to create my own custom theme. Unfortunately I found the dark and light theme variables hard coded all over the place. :(
I have this set up to compare my last three days of solar output. I don't remember off the top of my head how I did it but I might be able to help out. What DB are you running on top of?
We are also working on an expression language to backfill other data sources. You can see the frontend version of this - transformations in 7.0. The backend version which will support functions like timeshift is in alpha for now. Hopefully it will be officially released in the near future.
I'm not sure you need window functions for your query.
Have you tried something like this?
SELECT time, '2018' as metric, value FROM metrics
WHERE time >= '2018-01-01' AND time < '2019-01-01'
UNION ALL
SELECT time - '1 year'::interval, '2019' as metric, value FROM metrics
WHERE time >= '2019-01-01' AND time < '2020-01-01'
ORDER by time;
You will have a 2018 and a 2019 line if you plot this. (And you can of course make the time-periods dynamic.)
Note: One of our (TimescaleDB) engineers authored and contributed the PostgreSQL data source to Grafana. We happen to see quite a few Grafana dashboards built with SQL. If you'd like to debug in real-time then I'd suggest joining our Slack: https://slack.timescale.com/.
To get graphs for this week and the last 3 weeks in the same panel you could use something like this:
SELECT i::text as metric, l1.*
FROM
(SELECT
g1 AS i,
g1 * '1week'::interval AS interval,
'1 week'::interval AS span,
now() AS start
FROM generate_series(0,3) g1) g1
JOIN LATERAL (
SELECT
time + interval AS time,
value
FROM metrics
WHERE time > start - (span+interval) AND time < start -
interval
) l1 ON TRUE
ORDER BY time;
I'd suggest they got a director/manager who was BSed into "fascinating" presentation from Datadog salesmen. A lot of great opensource projects are killed pretty much the same way.
Disclaimer: datadog may (or may not) be a good product, it's irrelevant in the case.
Kinda the same situation here, but we managed to keep our dear Grafanas, by telling: "ok, give us budget to migrate all the dashboards". I can confirm that DD marketing is aggressive and full of BS and they will try to sell you missing features, since I attended a DD "workshop" (aka sales presentation) once.
I have a Grafana dashboard set up for all of my power/hvac/internet infrastructure (I live off grid) and it's an amazing tool. I'm excited to try out some of the new features. Without Grafana I wouldn't have been able to diagnose a charge issue my solar controller has on partially cloudy days. Very grateful to the devs, thank you all for the hard work.
I do exactly the same! I love it, it is so useful. I keep my solar grafana dashboard as a pinned tab and it is amazing.
I actually developed a python library for reading data from my Outback Mate3[1]. I have ambitions to manage and automate a lot more this way, but I need to improve my electronics skills first.
I like grafana for graphing but it still lacks essential features for alerting. Most important would be per-series state tracking (https://github.com/grafana/grafana/issues/6041) - the issue is open since 2016 and it does not look like it will ever be fixed :-(
What datasource are you using? I've only used Grafana as a visualizer and use other alerting/monitoring against that datasources. At my current employer, Prometheus is the data source and we use Alert Manager which allows us to write alerts in PromQL which is very flexible.
I really hope the new plugin architecture does not change for a while. The biggest reason we are still on an older grafana release at work is because we don't have resources to update our plugins to the later versions. And now we have to migrate to yet another new architecture?
My biggest thing I wish was fixed was being able to use template variables in alerts. https://github.com/grafana/grafana/issues/6557 Right now you have to make a chart that has every server on it or a graph per server. I would love to not have to do that
Congrats! I used Grafana for two projects and I really liked it. I am thinking of providing an easy way to use Grafana as a "custom UI" for by analytics platform, so you could either view the normal charts in the analytics app itself or create your own dashboards in Grafana using the data already collected. I assume this could be easily done my connecting the MySQL database as a datasource and providing the users with some example queries to get started, right?
Congratulations for this great Project! Talking about application metrics (like business transaction timings) are there any plans to release this kind of feature in the roadmap? What about user frontend to backends metrics?
100 comments
[ 3.2 ms ] story [ 151 ms ] threadGrafana project creator & lead. Here to answer any questions.
Are there plans for letting power-users style their own custom panels using html, css and js using just the web interface? Meaning without having to develop a plugin.
There are some external plugins that offer more advanced support:
- https://grafana.com/grafana/plugins/aidanmountford-html-pane... - https://grafana.com/grafana/plugins/ryantxu-ajax-panel
this is powerful, but long term tends to lead to a maintenance nightmare
A few things have been added that may help: 1. only visible data is queried and loaded 2. If you have multiple panels that use the same data, you can now just reuse the results (or a subset of the results) in a new panel. This will cut down on additional queries that may not be necessary
https://github.com/grafana/grafana/pull/21835
First off, Grafana is amazing! So keep up the good work.
Question: Currently alerts are fairly limited both in applicable panels and how they export/share relevant data to the chosen notification channel, as well as more advanced conditions. Is there any work going on with improving alerts?
[1] https://community.grafana.com/t/template-variables-are-not-s...
[2] https://github.com/grafana/grafana/issues/6557
How do I convince my current company to get the hell off Datadog?
The Jaeger integration is a good step in the right direction though, kudos!
Traditional APM products are internal traces
As a color blind user, I find it difficult to read the dashboards created by others. How can I help any efforts to increase Grafana's accessibility?
I especially have trouble matching colors in a legend with the drawn objects in the graph. The best help has been the hover panel, but it's unsorted by default and the highlight contrast on the panel for hovering over a line is pretty dim.
Thanks for the great work!
EXAMPLE 1:
Imagine a service. Load avg of 2 is unacceptable and means things will be slowing down. However, a heavy job runs every night and pushes load averages between 1.5 and 2.5 during a 30 minute window.. I wanted to set a load avg alert threshold @ 2.0 but can't since it would fire false alarms too often. So i end up setting it at 3, and end up getting a late alert when things get really bad.
Instead, an anomaly detection would look for a number of standard deviations away from a mean, and ideally account for seasonality (i.e. compare the data with same data last few weeks). Now, things will behave better. If loads hit 2 during the day, it'll fire. But during that 30 minute window at night, loads would need to exceed 3 before it fires.
EXAMPLE 2:
Let's say I'm tracking 30 metrics for a service. I understand the service well enough to set thresholds in 5 metrics. However, I'm sure a failure won't happen alone and the other 25 metrics may show some oddities and give me an early warning of "something wrong happening" .. Here, some kind of loose "anomaly alerts" could help as an early warning. If a certain metric throws too many false alerts, we find a better way to monitor it.
EXAMPLE 3:
Lets say you have a service that auto scales. There's no limit to how many requests the service handles. So I can't really thresholds on requests/5min. Here, some kind of "anomaly alert" would be useful to let the team know that something is going on that's causing a high level of requests. Again, accounting for seasonality because things tend to change on weekends.
At the end of the day, an anomaly alert is not some "magic solution" but just and extra tool in the toolkit to use when it makes sense. Datadog and AWS both have built in easy to use anomaly alerting. Would be nice for grafana/prometheus to have something similar.
https://about.gitlab.com/blog/2019/07/23/anomaly-detection-u...
https://github.com/grafana/grafana/blob/master/contribute/de...
For example when I'm tracking the power grid frequency, I would like to add two lines at 50.05 and 49.95 Hz, as well as one at 50 Hz (with different color).
Others have voiced their desire to display a horizontal line at the 95th percentile in a bandwidth usage graph.
Otherwise depending on the database, they have functions for drawing lines or you can do a hack like this one for Prometheus:
https://play.grafana.org/d/h1lglb7ik/threshold-hack?editPane...
Now if only you could persuade kibana to do the same :-)
and maybe also add one to your home page. (make it link to the features)
when i see any release anouncement about a product i am unfamiliar with, the first question i have is what kind of product it is, so i can decide if i want to check it out
I have a mongodb database with several collections. All collections share a set of fields: "createdAt", "channelId", "status", etc.
How easy would it be to dump all this data into Grafana, and then be able to slice and dice these collections by any combination of the shared fields.
Basically, I'm tired of writing mongo queries whenever I want to inspect my data. I want to point and click around a beautiful UI, with data visualizations and easy filtering/aggregation. Is this easy to do with Grafana?
If you have mongodb enterprise you can use Mongo bi connector and connect it with any bi tool. I recommend metabase since it is open source and very good for slice and dice and reports.
The mongodb bi connector is a postgres with foreign data wrapper to mongodb. So even if you don't have a mongodb subscription, you may be able to pull this off using the OS version https://github.com/EnterpriseDB/mongo_fdw
Plugins are used for showing pie charts, etc.
Maybe you can use it directly this way? (if you don't mind the BI-equivalent hammering your primary data source)
[1] https://github.com/grafana/grafana/issues/6983
Once that is done, we plan to add the ability to link an alert rule to 1 or more dashboard panels.
https://github.com/doublemarket/grafana-rrd-server
It has no support for rrdcached flush, AVERAGE CF is hardcoded , and I cannot seem to get it to not connect lines through gaps where there were no measurements.
Example not related to rrd, but same problem different db: https://github.com/grafana/grafana/issues/2125.
This would allow us to deploy immutable Grafana instances using infrastructure-as-code tools like Terraform and Docker. Specifically, I want a way to eliminate post-install steps such as adding users and organizations and setting user passwords.
Even if you use OAuth/LDAP for accounts and passwords, Grafana still requires manual steps to create the organization, install plugins, and set the admin password. I wrote a Python script to do this on Docker container startup. That script is a workaround for Grafana server's poor ops design. It's extra code that we must maintain.
I'm not using AWS CloudWatch Logs because of the financial risk of a runaway server running up a huge AWS bill. I had difficulty getting log display and log search working well in Grafana 6.6.2 and InfluxDB. Do you plan to improve log support for other databases?
Never had any issues with it (and the ones I had were promptly fixed by the devs and the community).
Recently I digged into the source code to find out how easy it is to create my own custom theme. Unfortunately I found the dark and light theme variables hard coded all over the place. :(
https://community.grafana.com/t/yearly-monthly-weekly-and-ot...
https://imgur.com/a/UXQ4bfY
It should be possible with SQL but it is trickier. TSDB's like Prometheus, Graphite and Influxdb (Flux) have the timeshift function built into their query languages. (https://community.grafana.com/t/advanced-graphing-part2-visu...)
Have you tried something like this?
SELECT time, '2018' as metric, value FROM metrics
WHERE time >= '2018-01-01' AND time < '2019-01-01'
UNION ALL
SELECT time - '1 year'::interval, '2019' as metric, value FROM metrics
WHERE time >= '2019-01-01' AND time < '2020-01-01'
ORDER by time;
You will have a 2018 and a 2019 line if you plot this. (And you can of course make the time-periods dynamic.)
Note: One of our (TimescaleDB) engineers authored and contributed the PostgreSQL data source to Grafana. We happen to see quite a few Grafana dashboards built with SQL. If you'd like to debug in real-time then I'd suggest joining our Slack: https://slack.timescale.com/.
Disclaimer: datadog may (or may not) be a good product, it's irrelevant in the case.
Grafana has equally "fascinating" presentations for management. Management doesn't take much but a few gauges chart to be impressed.
Try to register your email for Grafana and Datadog. Last one will bombard you with endless pitching emails until you ask them to stop doing it.
I actually developed a python library for reading data from my Outback Mate3[1]. I have ambitions to manage and automate a lot more this way, but I need to improve my electronics skills first.
https://github.com/adamcharnock/mate3/