Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs (github.com)

We definitely have pgvector on our roadmap (which tbh I think we better publish in the repo). For Graphana tempo, it's just a matter of making sure that it works as a destination - we'll do it today/tomorrow.

1. You don't have instrumentations for libraries like OpenAI, LangChain, etc. so you need to manually open spans

2. As you said, there are no semantic conventions for logging things like prompts and chains.

What we did is just defined the new set of semantic conventions, and built the instrumentations. But we're using vanilla OpenTelemetry so it's fully compatible with standard OpenTelemetry.

Here are all the semantic conventions we've defined so far - https://github.com/traceloop/openllmetry/tree/main/packages/...

And for that my answer would be that I think having a separate repo would allow this to evolve in a more natural way, and faster (whereas OpenTelemetry, given it's massive adoption already, evolves much slower, with committees etc.).

Then, at some point when this is stabilized and useful - we can merge.

Kind of like Tesla's NACS vs. CCS

I do think vendor locking is a key differentiator, which some of the reasons why OpenTelemetry succeeded in the first place. I know that my previous company switched to OpenTelemetry for exactly this reason. You get the flexibility of using any platform you'd want (since we're compatible with OpenTelemetry), so it's not vendor-locking you to a specific platform with specific capabilities. Why use any of the ones you mention - maybe Datadog is enough if your use case is simple?

But there are more advantages - you get much more than just observability to the LLM itself - you can see calls to vector DBs, network calls, DB queries, etc. - this can be extremely useful IMO for RAG and autonomous agents for example

Let's say your application takes several steps to build up a prompt dynamically, such as a RAG pipeline. You'll end up with a different prompt for potentially each user, depending on the application.

The result is you've likely increased the accuracy of the LLM, but at the expense of understanding the whole system's behavior by introducing more steps upstream of the LLM call. Those steps could be super simple, or they could be (like in our case) dozens of steps that could all potentially fail or have a bug or whatever.

And so how do you wrangle all of this in context? You need something like OpenLLMetry that treats a request to an LLM as one of several components that make up a request and/or user experience. Otherwise you're just throwing stuff at the wall, guessing at what could improve stuff (or guessing at what could make an eval score better).

We've already started developing the typescript SDK. Would love to see exactly what your use case is, so we can prioritize specific instrumentation and collaborate on it. We'll ping you.

That's a good tbh. I wonder whether we should implement instrumentations for LLMs "hosting solutions" or for specific LLMs (E.g. LLaMa/Falcon) and ignore the hosting solution (not sure if that's even possible though as it sort of dictates the inference api).

wdyt?