HN Explorer

Show HN: Phoenix OSS – Applying LLM Spans, Traces, and Evals for AI Insights (github.com)

23 points by jlopes2 ↗ HN
We are the developers of Phoenix, which we released in April of this year with a goal of bringing LLM observability to the notebook. In the time since, the growth of LLM frameworks and complex agent workflows led us to add support for LLM spans and traces and introduce a simple Eval harness for testing the data from those spans.

The latest Traces & Spans release of Phoenix offers: -Out of the box tracing for LlamaIndex and LangChain -Fully local execution, no data sent anywhere, outside of your own LLM calls -Ability to get a common dataframe format across frameworks back to a notebook for Evals -Code based LLM Eval harness: light, simple, and fast -Benchmarking scripts for retrieval setup - chunk size, K, and retrieval approach

https://github.com/Arize-ai/phoenix

We love to hear more from the community about what kind of LLM applications you are building, whether you are using a framework or building from scratch, and how are you running/measuring LLM Evals today. Thinking a lot about what “non-framework” integrations look like.

Also genuinely interested in people's opinions of LLM spans and traces versus OTEL. Is the divergence because of something intrinsic to the application or as a community are we reinventing the wheel?

Colab if you want to test it out: https://colab.research.google.com/github/Arize-ai/phoenix/bl...

3 comments

[ 3.5 ms ] story [ 41.1 ms ] thread
michaelschiff
Hey, Michael here, CTO of Arize and minor contributor to Phoenix.

The ability to trace a piece of software is a night and day difference for understanding. Once you know it's possible to have this level of visibility into what your code is doing, its impossible to go back. Im really excited to see what people learn by applying Phoenix Traces to their applications.

I've spent ton of time debating the data model of the span information we collect, as well as many different instrumentation options. We started with LlamaIndex and LangChain callback systems as the hook for instrumentation, since those frameworks are a common way for developers to get started with LLMs. We will add support for custom instrumentation, allowing users to manage the creation of spans themselves and avoid framework lock-in. When it comes to custom instrumentation, Im curious where the community lies - are people planning to use OTEL for this, or is the expectation that LLM spans are different enough that they warrant a different approach?

VibhuSapra
Been using it for a while, def recommend anyone building with LLMs to try it out
mikeldking
Thanks Vibhu! We will keep cranking on our end.