HN Explorer

Ask HN: Distributed tracing service with critical path analysis?

10 points by pseudonom- ↗ HN
The Dapper paper mentions "An engineer working on long tail latency debugging built a small library which infers hierarchical critical paths from DAPI Trace objects. These critical path structures were then used to diagnose problems and prioritize prospective performance improvements for universal search."

Do any of the distributed tracing services (e.g. Datadog APM, AWS X-Ray) offer this? I don't see any services which tout this functionality.

5 comments

[ 2.8 ms ] story [ 24.7 ms ] thread
programd
I think what you're looking for is Zipkin [0]. We use it in our microservers at Nulladmin.com via the opentracing Go library [1] and the zipkin-go-opentracing library [2]

[0] https://github.com/openzipkin/zipkin

[1] https://github.com/opentracing/opentracing-go

[2] https://github.com/openzipkin/zipkin-go-opentracing

pseudonom-
My company actually has and uses Zipkin. The thing that I'm looking for that I think is lacking is a way to identify critical paths without manually clicking through a bunch of traces and eyeballing them.
jjirsa
+1 for zipkin

As a Cassandra fan, bonus points that it integrates all the way down into the Cassandra internal request tracing features, afaik the only tracing app/library that does so.

albertwang
Disclosure: I work at Datadog --

Tracing requests across services and hosts to identify bottlenecks is one reason why Datadog decided to build a tracing component in APM! A flamegraph is one such method to visually infer critical path analysis that Datadog APM supports. Some obvious things to consider are ease of setup and whether the libraries and languages you use are supported. Some not-so-obvious things to consider are:

  (1) whether you wish to infer this from a single request, or across multiple requests, 
  (2) how to define "interesting" traces -- as you could be collecting millions or billions of traces in production 
  (3) how to create a bigger, "aggregate" picture across individual traces.
We're working on additional ways to easily identify critical paths as well.