What the author doesn't realize is that OpenTelemetry has fundamental problems. I experienced this firsthand two years ago working with Otel in Rust, and just today I spent an entire afternoon debugging what turned out to be an otel package update breaking react-router links. Since the bug showed up with several other package updates at once, otel was in the bottom of my suspicion list.
The core issue is that, with otel, observability platforms become just a UI layer over a database. No one wants to invest in proper instrumentation, which is a difficult problem, so we end up with a tragedy of the commons where the instrumentation layer itself gets neglected as there is no money to be made there.
OpenTelemetry won observability mindshare, but it is entirely the wrong architectural choice: by buying into its ethos your code is held hostage by the least stable otel monitoring library for your dependencies.
Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere hope is that the renewed interest in ebpf makes this a reality soon: I never want to have to do another from opentelemetry.<whatever> import <whatever> ever again.
Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog?
Honeycomb is decent at what it has, but very limited offerings for dashboard.
Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation, even time shifting is like pulling teeth. Plus they just totally broke all our graphs with formulas for like 2 days.
OTel has been slow as hell since release. we tried using it for tracing last week and the app was significantly slower.
We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow.
I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's all you need.
Edit: I now read in the comments the JS ver is a bad impl, I guess this might be part of the reason.
What is the preferred setup for deploying OpenTelemetry on a Kubernetes cluster? Is OpenTelemetry the choice today?
I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg.
Languages used are Rust, Javascript and Python mostly.
There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.
I guess I'm the contrarian but I've had good success with otel. It's especially powerful being able to plug and play components like take ddtrace Python instrumentation library and hook it up to otel collector and dump into Jaeger.
I can run a full tracing stack locally for dev use with minimal config.
OTel's real bottleneck isn't the spec. It's the fact that it requires you to instrument the app itself. That couples your performance to the maturity of the least stable SDK you depend on.
eBPF solves this by reversing the model: instrument the system, not the application. Turn it on / off dynamically, zero redeploys, minimal overhead.
The missing piece is accessibility. Kernel-level observability exists; "normal engineers can use it" and good DX does not.
12 comments
[ 2.6 ms ] story [ 40.6 ms ] threadThe core issue is that, with otel, observability platforms become just a UI layer over a database. No one wants to invest in proper instrumentation, which is a difficult problem, so we end up with a tragedy of the commons where the instrumentation layer itself gets neglected as there is no money to be made there.
Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere hope is that the renewed interest in ebpf makes this a reality soon: I never want to have to do another from opentelemetry.<whatever> import <whatever> ever again.
Honeycomb is decent at what it has, but very limited offerings for dashboard.
Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation, even time shifting is like pulling teeth. Plus they just totally broke all our graphs with formulas for like 2 days.
Grafana is to Datadog what Bugzilla is to Linear.
Am I wrong?
We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow.
I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's all you need.
Edit: I now read in the comments the JS ver is a bad impl, I guess this might be part of the reason.
I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg.
Languages used are Rust, Javascript and Python mostly.
I can run a full tracing stack locally for dev use with minimal config.
eBPF solves this by reversing the model: instrument the system, not the application. Turn it on / off dynamically, zero redeploys, minimal overhead.
The missing piece is accessibility. Kernel-level observability exists; "normal engineers can use it" and good DX does not.