HN Explorer

Show HN: Route your prompts to the best LLM (unify.ai)

298 points by danlenton ↗ HN
Hey HN, we've just finished building a dynamic router for LLMs, which takes each prompt and sends it to the most appropriate model and provider. We'd love to know what you think!

Here is a quick(ish) screen-recroding explaining how it works: https://youtu.be/ZpY6SIkBosE

Best results when training a custom router on your own prompt data: https://youtu.be/9JYqNbIEac0

The router balances user preferences for quality, speed and cost. The end result is higher quality and faster LLM responses at lower cost.

The quality for each candidate LLM is predicted ahead of time using a neural scoring function, which is a BERT-like architecture conditioned on the prompt and a latent representation of the LLM being scored. The different LLMs are queried across the batch dimension, with the neural scoring architecture taking a single latent representation of the LLM as input per forward pass. This makes the scoring function very modular to query for different LLM combinations. It is trained in a supervised manner on several open LLM datasets, using GPT4 as a judge. The cost and speed data is taken from our live benchmarks, updated every few hours across all continents. The final "loss function" is a linear combination of quality, cost, inter-token-latency and time-to-first-token, with the user effectively scaling the weighting factors of this linear combination.

Smaller LLMs are often good enough for simple prompts, but knowing exactly how and when they might break is difficult. Simple perturbations of the phrasing can cause smaller LLMs to fail catastrophically, making them hard to rely on. For example, Gemma-7B converts numbers to strings and returns the "largest" string when asking for the "largest" number in a set, but works fine when asking for the "highest" or "maximum".

The router is able to learn these quirky distributions, and ensure that the smaller, cheaper and faster LLMs are only used when there is high confidence that they will get the answer correct.

Pricing-wise, we charge the same rates as the backend providers we route to, without taking any margins. We also give $50 in free credits to all new signups.

The router can be used off-the-shelf, or it can be trained directly on your own data for improved performance.

What do people think? Could this be useful?

Feedback of all kinds is welcome!

127 comments

[ 3.1 ms ] story [ 198 ms ] thread
lovesdogsnsnow
This is interesting! Sort of a super mixture of experts model. What's the latency penalty paid with your router in the middle?

The pattern I often see is companies prototyping on the most expensive models, then testing smaller/faster/cheaper models to determine what is actually required for production. For which contexts and products do you foresee your approach being superior?

Given you're just passing along inference costs from backend providers and aren't taking margin, what's your long-term plan for profitability?

danlenton
Great question! Generally the neural network used for the router takes maybe ~20ms during inference. When deployed on prem, in your own cloud environment, then this is the only latecy. When using the public endpoints with our own intermediate server, it might add ~150ms to the time-to-first-token, but inter-token-latency is not affected.

We generally see the router being useful when the LLM application is being scaled, and cost and speed start to matter a lot. However, in some cases the output quality actually improved, as we're able to squeeze the best of GPT4 and Claude etc.

Long-term plan for profitability would come from some future version of the router, where we save the user time and money, and then charge some overhead for the router, but with the user still paying less than they would be with a single endpoint. Hopefully that makes sense?

Happy to answer any other questions!

jonahx
Do you save the user data, ie, the searches themselves? What do your TOS guarantee about the use of that data?
danlenton
We use this data to improve the base router by default. It's fully anonymized, and you can opt out.
ColinHayhurst
Without opt out it would be a no go, so that's great to hear. What's the downside of opting out?
nl
If I was doing this I'd negotiate a volume discount, charge the clients the base rate and pocket the difference.
danlenton
definitely on the cards, we're keeping our options open here. Right now just focused on creating value though.
3abiton
Isn't this what a MoE LLM does already?
sodality2
MoE-based models are one model with multiple experts. This solution could use entirely different models with different architectures (and probably supports MoE models itself)
danlenton
MoE LLMs use several "expert" fully connected layers, which are routed to during the forward pass, all trained end-to-end. This approach can also work with black-box LLMs like Opus, GPT4 etc. It's a similar concept but operating at a higher level of abstraction.
joaquincabezas
I’ve already heard a few times that the selection of models is seen as a problem (need to benchmark continually, justify changes…), this is an elegant solution.

I don’t know if choosing different models for the same consumer can be problematic (seen as not consistent), but maybe using this approach will force the post-processing code not to be “coupled” with one particular model.

danlenton
This is a great point. With models becoming more intelligent, they're seeming to become less brittle to the subtleties in the prompts, which might mean decoupling will occur naturally anyway. With regards to customers wanting to stick with "one model", we more often find that they want to stick with "one cloud environment". This is not a problem, the router can be deployed inside custom cloud environments easily. Interesting to see how much auto-prompting such as DSPy generalizes across models though, when you really start to see intermediate prompts as latent variables. Not sure how much these learned prompts might behave differently with different LLMs. I guess you might need to jointly learn the prompts and the routing decisions together for it to work really well. One step at a time though!
tyrw
It feels like there is an analogy here with Yahoo! and the early days of going from curated lists of websites to search algorithms. Do you think of LLMs in a similar way? I.e. some kind of model ranking score that companies could eventually game?

I'm not sure what the SEO equivalent would be here...

danlenton
Great analogy, I'm not sure tbh. I don't think we will see quite as many unique models as we see unique websites, but I do think we're going to see an increasing number of divergent and specialized models, which lend themselves to routing. I guess the SEO analogy would be "tricking" the neural routing function into thinking your LLM is better than it actually is, there are many techniques already to hone in on neural net weaknesses. Definitely interested to see how the space evolves!
swiftlyTyped
Really great stuff.

People use the same model / server for all queries not because it's sensible, but because it's simple. This brings the same simplicity to the far more optimal solution.

And great startup play too, by definition no incumbent can fill this role.

danlenton
Thanks - glad to hear the idea resonates!
xena
Your notification for the launch woke me up several times last night because you had the notification change hourly for 4 hours.
danlenton
Very sorry about that! I had an issue with my google calendar, I set it to "do not send emails" but for some reason some still came through. Fixed + removed now.
ianbicking
I've bumped into a few of these. I use https://openrouter.ai as a model abstraction, but not as a router. https://withmartian.com does the same thing but with a more enterprise feel. Also https://www.braintrustdata.com/ though it's less clear how committed they are to that feature.

That said, while I've really enjoyed the LLM abstraction (making it easy for me to test different models without changing my code), I haven't felt any desire for a router. I _do_ have some prompts that I send to gpt-3.5-turbo, and could potentially use other models, but it's kind of niche.

In part this is because I try to do as much in a single prompt as I can, meaning I want to use a model that's able to handle the hardest parts of the prompt and then the easy parts come along with. As a result there's not many "easy" prompts. The easy prompts are usually text fixup and routing.

My "routing" prompts are at a different level of abstraction, usually routing some input or activity to one of several prompts (each of which has its own context, and the sum of all contexts across those prompts is too large, hence the routing). I don't know if there's some meaningful crossover between these two routing concepts.

Another issue I have with LLM portability is the use of tools/functions/structured output. Opus and Gemini Pro 1.5 have kind of implemented this OK, but until recently GPT was the only halfway decent implementation of this. This seems to be an "advanced" feature, yet it's also a feature I use even more with smaller prompts, as those small prompts are often inside some larger algorithm and I don't want the fuss of text parsing and exceptions from ad hoc output.

But in the end I'm not price sensitive in my work, so I always come back to the newest GPT model. If I make a switch to Opus it definitely won't be to save money! And I'm probably not going to want to fiddle, but instead make a thoughtful choice and switch the default model in my code.

ankrgyl
Thanks for mentioning Braintrust!

We are very committed to the proxy :)

Although, to your point, we have seen less market pull for routing, and more for (a) supporting the latest LLMs, (b) basic translation (e.g. tool call API b/w Anthropic & OpenAI), and (c) solid infra features like caching/load balancing api keys/secret management. So that's our focus.

danlenton
Super helpful feedback, thanks for going so deep! I agree that for the really heavy agentic stuff, the router in it's current form might not be the most important innovation.

However, for several use cases speed is really paramount, and can directly hinder the UX. Examples include sales call agents, copilots, auto-complete engines etc. These are some of the areas where we've seen the router really shine, diverting to slow models when absolutely necesary on complex prompts, but using fast models as often as possible to minimize disruption to the UX.

Having said that, another major benefit of the platform is the ability to quickly run objective benchmarks for quality, cost and speed across all models and providers, on your own prompts [https://youtu.be/PO4r6ek8U6M]. We have some users who run benchmarks regularly for different checkpoints of their fine-tuned model, comparing against all other custom fine-tuned models, as well as the various foundation models.

As for the overlap in routing concepts you mentioned, I've thought a lot about this actually. It's our intention to broaden the kinds of routing we're able to handle, where we assume all control flow decision (routing) and intermediate prompts are latent variables (DSPy perspective). In the immediate future there is not crossover though.

I agree cost is often an afterthought. Generally our users either care about improving speed, or they want to know which model or combination of models would be best for their task in terms of output quality (GPT-4, Opus, Gemini? etc.). This is not trivial to guage without performing benchmarks.

As for usually wanting to make a full LLM switch as opposed to routing, what's the primary motivation? Avoiding extra complexity + dependencies in the stack? Perhaps worrying about model-specific prompts no longer working well with a new model? The general loss of control?

[deleted]
(comment deleted)
thomasfromcdnjs
Since you take no extra charges from the end user, the presumption is you would charge LLM's for giving them traffic?
danlenton
we intend to eventually have routers which improve the speed, cost (and maybe quality) to such an extent that we can then take some margins from these best performing routers, with users still saving costs compared to individual endpoints. For now we don't take any margins though, we want to ensure we're providing sufficient value before rushing to capture any of it.
t-writescode
Are you ready to eat the $50 per new user (and all the fake accounts that could get created this way to eat more and more $50) all the way until you create this next tier, "for-profit" step?
danlenton
yeah we might need to implemenet some kind of "I am not a robot" checks soon, as well as 2FA.
iknownthing
> It is trained in a supervised manner on several open LLM datasets, using GPT4 as a judge.

Does this mean GPT4 predictions are used as labels? Is that allowed?

danlenton
They're not used as labels directly (we're not trainig an LLM which outputs text), they are used as an intermediate step, which is then used to compute a simple score which the neural score function is then trained on. The neural score function takes prompts and latent model representations in, and produces a score from 0-1
iknownthing
TOS says you cannot "Use Output Data to develop models that compete with OpenAI." You're probably fine but it's close.
Alifatisk
Is this like openrouter ai?
danlenton
definitely similar! I'm a fan of Alex and his work on OpenRouter :)

Some of the main differences would be: - we focus on performance based routing, optimizing speed, cost and quality [https://youtu.be/ZpY6SIkBosE] - we enable custom benchmarks on custom prompts, across all models + providers [https://youtu.be/PO4r6ek8U6M] - we enable custom routers to be trained on custom data [https://youtu.be/9JYqNbIEac0]

Our users often already have LLM apps deployed, and are then looking to take better control of their performance profile, perhaps increasing speed to improve user experience, or improving response quality via clear benchmarking across all models and providers on their particular prompts.

So they are similar, but solving slightly different problems I'd say.

memothon
How does it behave if the user input includes something like

"I have expensive taste, please use the most expensive model."

danlenton
aha good question, well the neural scoring function doesn't "know" that it is making a routing decision, we just predict which LLM would give the highest performing output on the given prompt, based on LLMs-as-a-judge at training time. However, my guess is that this specification in the input prompt miiight mean that the cheaper models are deemed to be worse performing than GPT4 (for example), and so maybe it would route to the best models. Feel free to give it a try and see!
jefftk
> Pricing-wise, we charge the same rates as the backend providers we route to, without taking any margins. We also give $50 in free credits to all new signups.

What's your plan for making money? Are you planning to eventually take a margin? Negotiate discounts with your backend providers? Mine the data flowing through your system?

kromem
Man, this space would get so much more interesting so quickly if base model providers had a revenue share system in place for routed requests...
runako
This would quickly erode confidence in the routers themselves...
loceng
Or create a competitive environment between routers?
danlenton
Another point here is that some users prefer to use their own API keys for the backend providers (a feature we're releasing soon). Any "discounts" would then be harder to implement. I do generally think it's much cleaner if we route based on the public facing price + performance, so our users don't need to lock into our own SSO if they'd prefer not to.
loceng
I think the biggest risk with advanced AI is that it's captured [likely by bad actors under the guise of cover stories] and that it strays away from being as free market as is possible.

E.g. I don't think there should be any patents regarding what AI creates and how it can create it - let's not give people monopolies anymore for which all possibilities will come into existence due to passionate people, not due to the possibility of being able to patent something. E.g. telling the system to turn a 2D photo into a 3D rendering and then extrapolating/reverse engineering that to tie into materials and known building code requirements is plainly obvious, as one easy example; a "gold rush" for patents on AI etc only aims to benefit relatively rent-seekers and those in the VC industrial complex, etc.

danlenton
that's a good point, impartiality would then be questioned
danlenton
I certainly wouldn't complain about this lol
namdnay
So they end up in the same situation as hotels or airlines, beholden to the middlemen? They’ll never allow that :)
cosmojg
The data flowing through LLM routers is a hot commodity right now. OpenRouter, for example, even provides a flat-rate 1% discount across the board if you agree to let them use your API calls for model training, and rumor has it that they're already profitable. To be fair, they do seem to be collaborating with model providers on some level, so they are likely getting discounted access on top of selling data.
danlenton
The idea is that at some point in future, we release new and improved router configurations which do take small margins, but from the user perspective they're still paying less than using a single endpoint. We don't intend to inflate the price when users only use the single-sign-on benefits. Negotiating discounts with backend providers is another possibility, but right now we're just focused on providing value.
[dead] disagree123
[flagged]
danlenton
lol well sure, that too
disagree123
but only as a secondary goal right? ;)

It's all about value for the world!

Aeolun
Honestly, I’d feel a lot more secure about building on this if you did take (for example) a small fixed fee every month. Or a 10% commission on any requests (volume discounts on that commission for high-volume users?).

If I start using you now you’ll either disappear in the future or you’ll suddenly start charging more, neither of which I like.

I’m already paying for inference, a little amount on top of that for the convenience of a single API is pretty useful.

danlenton
Makes sense, thanks a lot for the feedback. We're pretty confident that future versions of our router will provide sufficient value where we can take margins here, we therefore don't expect the need to start charging for Single-sign-on (SSO) alone. The SSO benefits are only the beginning in my mind, our main value will come from custom benchmarks across all models + providers and optimizing LLM applications, including agentic workfows. I do very much see your point though. Thankfully, we're very fortunate to have several years of runway, so we don't plan on disappearing anytime too soon!
roynasser
A common model in some cost cutting software is to charge x% of the total savings... Win/win...just a suggestion... use picks "main LLM" and you calculate the "non optimized cost" based on that. Whatever savings you drive you take a share of the savings.
janekm
It's tough in this case, because if you incentivise just to save cost, it could always route you to the cheapest LLM but the quality would suffer...
danlenton
Yeah that's a great point, something we'll keep in mind as we work out the final business model. Thanks!
danlenton
however, as janekm says, we can't charge just based on cost savings. We would need the router points to be sufficiently compelling wrt quality, speed and cost (including our own margins) that users still sometimes opt for these router points. Suffice it to say, if any router configs do start to take margins, then this will be clearly reflected in the overall router cost plotted on the scatter graph. UX will not be affected.
bitbuilder
Agree heavily with this sentiment. It sounds like this could be a useful tool for a personal project of mine, but I wasn't nearly as interested after reading they're not attempting to make money yet. I'm a bit burnt out on that business model. Predictability is just as important as price when I'm deciding how to invest a large portion of my free time. I happily gave OpenRouter $20 for their service, and I've barely dented the credits with thousands of test runs over two months.

On that note, I think I'd be even more likely to pay for Unify.ai if I could opt to bypass the auto-routing and use it the same way I use OpenRouter - a single endpoint to route to any model I want. Sometimes I've already determined the best model for a task, and other times I want redundant models for the same task. It's possible Unify has this option, though I didn't see it while skimming the docs.

But really, all in all, this is a super cool project and I'm happy it was shared.

potatoman22
This is super cool! I wonder if you could do a similar thing, but choosing between a collection of prompts for a task based on the input. Similar to dynamic few-shot prompting, but replacing the entire prompt instead of just the examples.
aleksiy123
I wonder, have you also done any work on actually improving performance by enhancing the prompt or something similar?

I always thought a product like this that could empirically decrease costs for the same performance or increase performance for a small increase in cost would have a fairly simple road of justifying its existence.

danlenton
As outlined, DSPy is a create tool for this. Currently their focus is on optimizing in-context examples, but their broader vision includes optimising system prompts for intermediate LLM nodes in an agentic system as well. We uploaded an explainder video about DSPy last week! Feel free to check it out: https://youtu.be/kFB8kFchCH4?si=0YKAVcpKJjogP1sX
ceroxylon
For those who like testing smaller local models, there is a tool in LangChain called Routing and in LlamaIndex called a "router query engine" which achieve a similar thing:

https://python.langchain.com/v0.1/docs/use_cases/query_analy...

https://docs.llamaindex.ai/en/stable/examples/query_engine/R...

dheera
> dynamic router for LLMs

This is sort of how Mixture-of-Experts models work, actually.

danlenton
Yep! Although MoE use several "expert" linear layers within a single network, and generally the "routing" is not based on high-level semantics, but token specialization, as disuccussed by Fuzhao Xue, author of OpenMoE, in one of our reading groups: https://www.youtube.com/watch?v=k3QOpJA0A0Q&t=1547s
Sontho
It's a good options to test local models I guess.
danlenton
actually we don't support local deployment quite yet, it's all run in our server. However, fully local deployment is on the roadmap, stay tuned!
swyx
yeah this is very nice, open source Martian. i never really understood the value of routing all the time. you want stability and predictability in models. and models have huge brand value. you're never going to, through routing, construct a "super" model that people want more than one or a few really good brand name models.
danlenton
Yeah I agree, routing all the time is overkill for some applications, however for others it really is necessary in order to reach the necessary speed and performance for users of your real-time app.

For those who don't want to always route, another core benefit of our platform is simple custom benchmarking on your task across all existing providers: https://youtu.be/PO4r6ek8U6M

If you then just want to use that provider rather than a router config, then that's fine!

lhousa
It's always something that "unifies" stuff. Be it cabs or food or services, etc. There's potential for this.
danlenton
aha maybe we should change our slogan to "Uber for LLMs"
danlenton
Agreed, we've spoken to tons of users who reach out to us and start the conversation with "we've tried to implement this ourselves".
mzl
Sounds similar to the Composition of Experts from SambaNova (although that is for increased accuracy on single systems mostly, not for decreased cost from various suppliers): https://sambanova.ai/blog/samba-coe-the-power-of-routing-ml-...
danlenton
Yep it's the same idea. We also see improvements in quality on some benchmarks beyond any specific model. This is common especially common when training a custom router on your own prompt dataset: https://youtu.be/9JYqNbIEac0
t_mann
Sounds interesting, can you explain the conceptual differences between your approach and mixture-of-experts (for someone with minimal understanding of MoE)?
danlenton
Sure! Basically traditional MoE has several linear layers, and the network learns to route down those paths, based on the training loss (similar to how CNNs learn through max-pooling, which is also non-differentiable). However, MoEs have been shown to specialiaze on tokens, not high-level semantics. This was eloquently explained by Fuzhao Xue, author of OpenMoE, in one of our reading groups: https://www.youtube.com/watch?v=k3QOpJA0A0Q&t=1547s

In contrast, our router sits at a higher level of the stack, sending prompts to different models and providers based on quality on the prompt distribution, speed and cost. Happy to clarify further if helpful!