Logs and tracing: not just for production, local development too (incident.io)

I was wondering, how should one get started with observability and implementing it? Are there specific books/courses/talks you'd recommend?

The reason I ask is because I've never been directly exposed to good observability at any of the companies I've worked at for a handful of reasons. It mostly boils down to the fact that I'm a DevOps engineer, so building observability is a set-up-and-keep-running sort of deal for other teams, not a useful-for-my-applications thing that I'm going to be working with often. Teams let us know "Splunk is down", "I can't reach Kibana", or "Looks like disk space is filling up" and that's about it after it's been initially set up.

There's a whole host of questions I'd ideally like to answer, but a lot of it boils down to the fact that I don't know what I don't know and I'd suggest assuming I know nothing over assuming I know something because I know the word. Questions I'd like to be able to answer are:

- What makes a good log? - What is a trace? Why is it useful? How does it help me debug issues faster? - How do you increase observability for loosely-coupled microservice systems? - How do you observe multi-threaded applications? - ... and I'm sure there are a whole bunch more.

The bit that surprised me is more likely due to my background being mostly Ruby/similar based, which is around debugging highly concurrent apps.

Our app spins up goroutines all the time. One request will fan out to a number of concurrent threads, doing things like hydrating responses from Slack to syncing issue tracker tickets.

With your STDOUT logs, that's so difficult to understand as it's all interwoven. And picturing all the threads and how they interact is fairly difficult.

I think that's why this was such a "ah-ha!" moment for us. Suddenly we could find the "error" log in our terminal, then click the trace URL and see everything in context.

That's probably the key difference to our past experience, that the app we're working with is so concurrent.

In terms of incident.io, we're a team of 3 engineers (looking for more!) and a company of 6 in total, so our priority was on buy and not build.

But in my most recent experience at a larger company (fintech unicorn, ~1k employees) we:

- Used Sentry to capture exceptions from all over the app. Our language was Ruby, which meant the stacktraces were usually explanatory, and attached debugging information to the event whenever we sent it - We didn't use tracing much, though when we did it would be at a request level. Our Sentries would link to the trace, and you'd see what happened during your request - All logs were associated with the trace

So our error report would be a Sentry, which could be joined against the logs and trace information.

In the situation where error reporting (probably better phrased as crash report?) was triggered by a user, it would normally come from a frontend site and trigger a Sentry.

The Sentry plugin will have tracked the console.log/debug calls up until that moment, and any requests that took place. So it was quite easy to piece together what had happened, and why it went wrong.

Not ever seen anything so custom as what you describe, though. Interesting to know that those systems are out there, and that you feel it worth building over buying!

I've used this in the past quite a bit, and found it really useful.

And in terms of Cloud Trace vs other tracing providers, I think it was the ease of setup and the simple log + trace inline view that sold us on it. Only took a couple of hours to get working, and it's great that it works for local dev too.

https://github.com/opentracing/opentracing-javascript

There are a relatively large number of packages to install to get a basic setup (see the examples [2]). However, it's pretty straightforward. In short, you'll need to:

1. Instantiate the WebTracerProvider which allows traces to be created

2. Instantiate a span processor (such as BatchSpanProcessor) and a span exporter into the tracer provider. In the browser, you'll likely use @opentelemetry/exporter-otlp-http for the span exporter, which can be configured to send traces to any backend (e.g. Jaegar, Honeycomb, Lightstep, or DataDog).

3. Configure a context manager for tracing across async callstacks. This is the most complex part of tracing in my opinion, but it makes the trace data — specifically for JS — significantly more powerful. Since JS does not have thread local storage, they use a package called zone.js to make this possible. It is currently the *only* solution for this problem in browser JS. Node has the luxury of the standard library async_hooks. This was attempted to be standardized, but the effort has stalled [3].

4. Register automatic instrumentations. For example, you can automatically trace all XHR requests, fetch requests, page loads, and user interactions. With distributed tracing, you should always prefer automatic instrumentation whenever possible to reduce maintenance and leverage existing conventions. The document load instrumentation allows you to treat the server as the parent span to a given page load, from which the client will then be the root span to everything in the server. This is an extremely powerful concept that allows traces to viewed from the perspective of the user, encapsulating all requests and user interactions in a single trace with no manual instrumentation!

We also trialed Sentry's APM tool (also marketed as a distributed tracing tool). While it had pretty charts, dashboards, niceties like core web vitals, and integrated well with Sentry's error product, its utility as a distributed tracing tool is significantly less powerful than tools like Honeycomb. You will end up using a large amount of your user's bandwidth sending telemetry data that can't fully be leveraged in the Sentry UI. When I last used (in April 2021), the spans of a given trace could only be viewed in a specific part the UI and they couldn't be searched for in queries or used in charts. I'm unsure if this has been updated. But this is not the worse part. Because Sentry uses its own data model for traces, it is not compatible with open source standards such as OpenTelemetry or OpenTracing! The sales team will not tell you this during the trial. This means our entire backend, which was already instrumented with OpenTracing, would now also need to instrument Sentry's tracing (...if they supported the language) in order to connect frontend traces to backend traces. If your company uses Sentry everywhere, then this may be less of a problem. Each team I met with their sales team, I said the same thing: support OpenTelemetry, otherwise you are asking for us to further isolate our backend and frontend teams.

It looks like they have heard this opinion, as they have recently published a blog post about the evolution of the distributed tracing API, citing incompatibility with OpenTelemetry due to their data model. It will require a very large change for them to support this. Meanwhile, OpenTelemetry can be used with any tracing vendor, a large number of languages, the other major instrumentation standards (OpenTracing, OpenCensus), and any trace propagation format.

- [1]: https://github.com/o...

Another commenter has pointed out things have moved on quite a bit since then- that's really good to hear, and we'll revisit this when we get some time to make the shift :)

It looks something like this:

    module "incident_io" {
      for_each = {
        "staging" = {
          project = "incident-io-staging"
        }
        "production" = {
          project = "incident-io-production"
        }
        "dev-lawrence" = {
          project    = "incident-io-dev-lawrence"
          autodeploy = true
        }
        "dev-lisa" = {
          project    = "incident-io-dev-lisa"
          autodeploy = true
        }
        # ...
      }

      source = "./modules/stack"

      application       = "incident-io"
      instance          = each.key
      google_project_id = each.value.project
      autodeploy        = lookup(each.value, "autodeploy", false)
    }