Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
One of the best features I built into our game studio's AI router (IMO) was to take a random sampling of generations and have them generate across ALL models (that we support, at least) so we, the developers, can browse outputs and get a sense each model's output. Seeing them all side by side for generations you're already familiar with makes it feel like significantly less cognitive load.
The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.
It's hard to keep track of all those launches, I agree. I would separate 2 usages of LLMs:
- LLM is integrated into a workflow/app for a classification or whatever job: scoring outputs (evals, KPIs) and choosing a fit between quality and cost is enough
- LLM is used as a tool you interact with (like coding or research): here is where you have to know your brushes. As humans we get used very quickly by the model personality, the format of output, the way it "thinks" and interacts.
A programmer is someone who declares "I understand the problem so well that I can teach a computer to solve the problem, even for examples I have never seen before".
If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.
Yeah I'm growing increasingly tired of those stupid metaphors about AI.
LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.
I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.
I'm all for dropping priority on vendor neutrality and tightly coupling with vendors.
But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.
I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.
There’s a heuristic I’ve been using to pick which model to use. The provider is already chosen by my org so what I do is activate the model picker in my harness, select the most expensive model’s latest version and set it on the highest effort setting.
This has worked quite well for me and saves a lot of time.
> Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
How does one spend a lot of time researching model routing? Did you evaluate different routing providers? Maybe you could build a rorouterter that lets you easily switch to any router, ideally if there's a vuln in any of them you can get infected even if you don't use it.
Their router classified prompts into difficulty buckets. This obviously won't work. Consider a senior developer routing work based only on the task description. Clearly you need to dig a bit deeper into the task.
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
I didn't know about Tokenless, the approach seems really innovative, if it works it fixes the "Complexity cannot be deduced from the prompt alone" problem. However you still have the other hidden costs: cache, breaking behavior consistency, and unpredictability.
Your point is interesting, you say that task specificity is easier to classify than task complexity, which I agree - I didn't mention it but we had task specificity routing too :). My opinion is that in many cases task specificity calls are easy to distinguish at build time, and therefore you can isolate them and attach the right model/settings beforehand, so you have less need for a smart routing on the fly.
I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.
I recently wrote about first principles of model routing that I've learnt building a model router.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
From a big-O notation perspective, if you're serving N queries, working the kinks out of your routing system costs perhaps O(logN) in developer time, whereas LLM provider savings grow as O(N).
In other words, the more queries you're serving, the more worthwhile it looks to figure out a viable method of model routing.
I agree with one distinction - coding agent workflows can use defined subagent roles that are pinned to specific models and I have found this very effective. The orchestrator is building all the context to make these assignments - it’s not a dumb router. Using Minimax M3 for exploration and librarian tasks for example is fast and cheap - my $10 plan lasts all month and saves a lot of tokens for my main coding plan.
Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers.
The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.
E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.
Now that even the smaller models from labs (Luna, deepseek v4 flash) are getting powerful, I think the orchestrator pattern of a smart model coordinating smaller models for work will end up being the way to go.
What was the architecture of your router? If it was based on GRPO/RL, it would be interesting to hear why your router performance capped.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.
We tried different things: heuristic, TF-IDF and LLMs. I voluntarily didn't talk about the tech because it doesn't fix the problems mentioned by switching models on the fly.
To add: I doubt frontier labs will build routers - they are not financially incentivized to optimize token usage (though in the short term they may be incentivized by constrained GPU capacity to reduce load).
All inference providers (labs and neoclouds like TogetherAI or Fireworks) are incentivized to be efficient to be more competitive. For example MoE reduces compute without reducing output quality. I would not be surprised if they end up implementing some kind of internal routing at some point.
I think the author need to rethink out of the box what's LLM routers in a traditional sense ( input in, route, output ) and move to think how a router would work in agentic workflow. See cognition Devin Fusion design.
55 comments
[ 2.3 ms ] story [ 46.9 ms ] thread> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.
If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.
LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.
I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.
But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.
I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.
This has worked quite well for me and saves a lot of time.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
Your point is interesting, you say that task specificity is easier to classify than task complexity, which I agree - I didn't mention it but we had task specificity routing too :). My opinion is that in many cases task specificity calls are easy to distinguish at build time, and therefore you can isolate them and attach the right model/settings beforehand, so you have less need for a smart routing on the fly.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
https://try.works/first-principles-of-model-routing
Seems like a naive classification model lacking context?
In other words, the more queries you're serving, the more worthwhile it looks to figure out a viable method of model routing.
The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.
E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.