HN Explorer

Show HN: Using LLMs and Embeddings to classify application errors (github.com)

11 points by vadman97 ↗ HN
Hi Hacker News! We’re Vadim and Chris from Highlight.io [1]. We do web app monitoring and recently worked on using LLMs/embeddings to improve our error product. Fair warning that we don’t have too much experience with LLMs, but since we’re seeing some interesting results, we figured we’d share our learnings with the HN community.

Our goal was to build two features: (1) tagging errors (e.g. deeming an error as “authentication error” or a “database error”); and (2) grouping similar errors together (e.g. two errors that have a different stacktrace and body, but are semantically not very different).

Each of these rely heavily on comparing text across our application. After some experimentation with the OpenAI embeddings API [3], we went ahead and hosted a private model instance of thenlper/gte-large (an open-source MIT licensed model), which is a 1024-dimension model running on a Nvidia Tesla T4 on Hugging face [4].

Our general approach for classifying/comparing text is as follows. As each set of tokens (i.e a string) comes in, our backend makes a request to an inference endpoint and receives a 1024-dimension float vector as a response (see the code here [5]). We then store that vector using pgvector [6]. To compare any two sets for similarity, we simply look at the Euclidian distance between their respective embeddings using the ivfflat index implemented by pgvector (example code here [7]).

To tag errors, we assign an error its most relevant tag from a predetermined set decided by us. For example, if we tag an error as an "authentication error" or a "database error", we can allow developers to have a starting point before inspecting an issue (see the logic here [8]).

Anecdotally, this approach seems to work very well. For example, here are two authentication errors that got tagged as “Authentication Error”:

* Firebase: A network AuthError has occurred

* Error retrieving user from firebase api for email verification: cannot find user from uid.

We also use these error embeddings to group similar errors. To decide whether an error joins a group or starts a new one, we decide on a distance threshold (using the euclidean distance) ahead of time [9]. An interesting thing about this approach, compared to using a text-based heuristic, is that two errors with different stack traces can still be grouped together. Here’s an example:

* github.com/highlight-run/highlight/backend/worker.(*Worker).ReportStripeUsage

* github.com/highlight-run/highlight/backend/private-graph/graph.(*Resolver).GetSlackChannelsFromSlack.func1

Both reported as `integration api error` as they involve the Stripe and Slack integrations respectively. The neat thing is that the LLM can use the full context of an error and match based on the most relevant details about the error.

We haven’t rolled this out to users yet, but feel free to give the logic a try at [2]. Long-term, we have lots of other ideas of what we could build with LLM tooling in observability, and if the HN community has any ideas, we’re all ears. Let us know what you think!

Links:

[1] https://news.ycombinator.com/item?id=36774611

[2] https://app.highlight.io/error-tags

[3] https://platform.openai.com/docs/guides/embeddings

[4] https://huggingface.co/thenlper/gte-large

[5] https://github.com/highlight/highlight/blob/main/backend/emb...

[6] <...

6 comments

[ 3.3 ms ] story [ 23.9 ms ] thread
hellovai
oh interesting, this is gonna be pretty helpful if it could actually recognize some pattern of errors even if the stack trace is different. (i.e. some shared function call fails, but that function is used within a bunch of different functions). have you had any thoughts on if fine tuned models would work better for your specific use case?
podoman
(I'm one of the Highlight.io founders). Yep, that's the intention. And anecdotally, it works pretty well, even without fine-tuning.

We started working on this before any of the fine-tuning APIs for OpenAI came out, but we are keeping an eye out for support for fine-tuning on more OSS models. I suspect it would result in better results, especially if it were trained on only exceptions, for example.

alexdanilowicz
Interesting! What do you think the main value add is: 1) saving the developer time from tagging known errors themselves? Or 2) is the idea that this would be applied to complete unknown errors, and give a sense of what's going on by labeling it?

Also, how hard was it to set up your own private model instance? I have yet to try that, I've just been playing around with OpenAI's API. Any good tutorials you recommend?

vadman97
Both - tagging errors into broad buckets can help with filtering and searching for categories of errors, while finding similar errors surfaces ones that might be more familiar to an unknown one. We're learning as we go though and also want to surface solutions to errors long term.

As for hosting the model, Hugging Face has a number of guides (ie. [1]) for this but overall the process was quite simple. All you need to do is find a model compatible with the kind of inference you want to do (ie. sentence embeddings vs. text generation). Once it's deployed, Hugging Face gives you an API endpoint similar to OpenAI's embeddings API that returns the embedding vector for a given input. For picking the model, there are a bunch of good resources with benchmarks comparing them depending on the use case (ie. [2]).

[1] https://huggingface.co/blog/getting-started-with-embeddings [2] https://supabase.com/blog/fewer-dimensions-are-better-pgvect...

teddarific
This is an interesting use case for AI — I spent a lot of time at my previous company messing around with Sentry's grouping interface that was a major PITA. Would have loved to just use natural language instead of weird regex-ish matching.

Curious from y'alls testing, what type of errors do NOT seem to work well / AI has trouble tagging?

vadman97
The hardest errors to group / tag are ones that are further removed from the actual error cause (think frontend bug that causes a bad request that then causes a backend error, or an OOM because of a gradual memory leak). For these, we're looking at other ways to give more context to the LLM about an error (for example, associating frontend errors with backend ones and providing both as part of the request) but we also have limits on the LLM input size (512 tokens for the mdoel we're using, gte-large).