14 comments

[ 1.8 ms ] story [ 38.5 ms ] thread
(comment deleted)
> The irony is that this may be a $0 revenue user for Grafana Labs.

Why is that ironic? Since Mimir is open-source, $0 revenue users are expected. AFAIK, Grafana Labs relies heavily on go, typescript, and linux, without necessarily being their top financial contributor. They could have kept Mimir proprietary like Splunk, but whether that would have attracted the same level of adoption or community contribution is another matter.

Grafana knows their open source products are eating into revenue. Expect corresponding strategy to offset that.
Directly emitting metrics using OTLP instead of having the OTel receiver scrape the metrics endpoint is interesting. I never made that move because the Prometheus metrics endpoint works and is so simple, and it's what most projects (eg kubernetes) use.
A long time ago, I introduced dogstatsd at Airbnb. We had already been using vanilla statsd (with no tag support -- cardinality lived in the metric name!) and this was a low cost migration. More than a decade later, I'm assuming it was difficult to track down and refactor all the places that statsd calls were emitted and using OTLP was an easier route. This is a great example of how technical decisions compound over time.
Curious why the team choose Grafana Mirmir over VM cluster?
How are these substitutes? Mimir is a time series database.

edit: I understood virtual machine :)

> given Prometheus’s widespread adoption and proven reliability in diverse environments.

I have used Prometheus a lot. Reliable is not a word I would associate with it.

the zero injection fix for sparse counters is the most underrated part of this writeup -- injecting a synthetic zero on first flush to anchor the cumulative baseline is actaully a pretty elegant solution to a problem that bites almost every team migrating from delta-based systems to prometheus, and the fact that they centralized it in the aggregation tier rather than pushing the fix to every instrumentation callsite is exactly the right call.
> Initially, we anticipated that the edge case would have minimal impact, given Prometheus’s widespread adoption and proven reliability in diverse environments. However, as we migrated more users, we started seeing this issue more frequently, and it stalled migration.

That's a very professional way of saying "Wait, everyone just lives with this? What the fuck?!"

Many such cases in the Prometheus ecosystem.

Interesting choice to go with Prometheus directly, especially when other TSDBs have "native" support for OTLP ingestion support.