I happen to work with otel a lot so I'll offer a few of my thoughts:
- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
I'm evaluating Greptimedb in prod and while I hate to have a needless component like OTEL-Collector in general, it serves as a read-only gate between the database and the user, so that greptime keeps listening to localhost only, and OTEL-Collector guarantees nobody will write to the database directly.
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.
I know you can export directly to the backend, but the collector typically uses less than 50MB of RAM in my experience (even when handling lots of traces) and it's pretty easy to add sidecars to however you deploy your backends nowadays. Using Grafana SaaS metrics could look a little spiky or generally weirder without the collector, but normal with it, so I just default to using it now.
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
Every provider seems to produce their own soft fork of the collector for branding (eg Alloy, ADOT, etc) and slightly changes the configuration, which doesn't help.
Otel stuff always seems overly complicated to me, but it must just be the types of projects I generally work with. Feels like observability meets java.
I've dabbled in building a project that collects metrics from the logs for smaller projects. Everyone tells me it's a bad idea, but it seems to work well for me.
This collector is one of my favorites to ask Copilot Agent to use for validation when the stack is missing tests. You give the agent a couple well written prompts of what you expect to happen and since the app has distributed tracing enabled. all logs flow to text and are consumable by the agent.
Just always use a collector. It really simplifies your life. Your app then just always talks with localhost without any authentication. And during the development, you can just set up local Uptrace to help you with debugging with just a different collector config.
And while all the tracing providers speak the OTEL protocol, the way your do auth is not the same. Sometimes you need to specify it in a header, sometimes it's a part of the URL.
9 comments
[ 193 ms ] story [ 896 ms ] thread- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
- https://www.otelbin.io is a nice little tool to help with collector configuration
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
Every provider seems to produce their own soft fork of the collector for branding (eg Alloy, ADOT, etc) and slightly changes the configuration, which doesn't help.
I've dabbled in building a project that collects metrics from the logs for smaller projects. Everyone tells me it's a bad idea, but it seems to work well for me.
And while all the tracing providers speak the OTEL protocol, the way your do auth is not the same. Sometimes you need to specify it in a header, sometimes it's a part of the URL.