Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding search
Eh, maybe you should keep both paths. When LLMs eventually crawl the site to feed back to agentic shoppers, maybe they logically follow the more truncated less decorated path.
I was in a project where we sent the whole taxonomy every request, 40k tokens + one article, ”plz classify”. This was before structured outputs. It was extremely expensive and still hallucinated. Good ol’ days.
I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON.
There you go, you just drastically reduced the output pricing.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
This is another riff on not embedding a full document, but doing a summarization of the document and embedding the summary for RAG. Nice usecase for high cardinality data!
Just this week I tried doing something similar with a nasty vibe-coded codebase I was trying to organize. I had Gemini Flash 3.6 classify each function/method in a similar way, giving a few plausible classifications for each (one agent per method).
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.
> In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
In the past, people would post advice on how to do something clever and useful yourself. Now, people post suggestions on how to talk out the side of their mouth to coax ther magic-8-ball slop generator to say something useful.
Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?
Smart trick, but assumes the “dumb” llm is smart enough not to derail into an article about the lives of South American red ants. Obvious exaggeration, the point being outcomes should stay strictly within topic, avoid unrelated bloat and hit the target.
Prompt expansion of input to extra categories makes sense if your embedding isn’t working well. But on its own, why use the LLM at all? I think you could have demonstrated the original step first and then shown that it’s useful.
I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.
If information is totally wrong then all you have to do is invert it to get the truth. What was it that Sherlock Holmes said? The problem ends up being that it often takes a tremendous number of counterexamples to eliminate everything that is impossible.
Worse is when you don't know whether the answers you have are totally wrong.
“Hallucinate” is misleading here. In the given example, a classification is being done very successfully - it’s just that it requires an extra step to map it to an arbitrary predefined list of classifications.
If you can articulate why you think this isn’t a good approach, I’d be interested to hear it.
Interesting technique, but even if you're getting rid of hallucinations it seems there's still no guarantee of consistent classifications. If you need to do a semantic (embedding) search anyways, then how does this really help?
A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LLM to take a random sample from each clustered group and create a classification for that group.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
The idea of distance thresholding models that are trained to satisfy an ordering constraint is a bit strange. The reason it's hard is that there isn't a threshold!
You can slice and dice it a ton of different ways, but the significance of groups is incidental.
It's a good starting point, but having done this a few times for a few companies it always seems like it needs substantial human review.
43 comments
[ 0.21 ms ] story [ 22.3 ms ] threadBut if accuracy matters, you can't rely on embedding sort to get a closet match. With a real test set they usually don't hold up under scrutiny.
Everything in AI is like this. You get an idea, try it once or twice, "LGTM" and you ship. Then it never survives contact reality.
Embedding sort gives you a better shortlist than the whole list, but you will probably want a heavier model to vet candidates.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
You'd think they would have solved it by now.
https://github.com/aurelio-labs/semantic-router
I guess it is based on the same fundamentals as well.
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
Worse is when you don't know whether the answers you have are totally wrong.
“Hallucinate” is misleading here. In the given example, a classification is being done very successfully - it’s just that it requires an extra step to map it to an arbitrary predefined list of classifications.
If you can articulate why you think this isn’t a good approach, I’d be interested to hear it.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
You can slice and dice it a ton of different ways, but the significance of groups is incidental.
It's a good starting point, but having done this a few times for a few companies it always seems like it needs substantial human review.