Really? Because removing it from my company has saved us over 2 million a year and we were able to speed up processing. The chargeback model for databricks is predatory at best.
That is a wild article! Hopefully the recent US open source model initiatives can pan out to cut costs, because I’m not a fan of dependence on Chinese models either, but I’m in a spot where I need to maximize intelligence on minimal cost…
There’s not even a distant comparison US lab rn which is sad
This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs.
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
These seem like the obvious tweaks akin to "using a cheaper hosting platform". I think the real savings come from careful context control for programmatic agents, careful tool awareness and usage to reduce thrashing, distilling workflows into deterministic processes and, moat importantly, adding friction and boundaries for non-technical users who tend to burn tokens making insane asks like "analyze all documents and give me a summary".
> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique.
I think the more interesting lever is the fourth they mention: token efficiency.
> By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include.
I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.
1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes
2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model
Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.
I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like:
- Spend most time prioritizing/discussing what to do.
- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)
- Use Opus 5 or Sol Med to execute
- Auto-fix bugs and CI until green + thermonuclear review skill x3.
- Manual interrogation of change/nits
- Come up with QA plan and have Codex Computer Use execute on it
- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)
I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
Interested to dive deeper on the upfront design discussion. Have you found these to more often then not translate into the real product.
In my experience at the begining of the full agentic coding loop in our company we were more hands on with the codebase and had better judgement over the plans. Now it is quite often that the inital plan after executed needs more refinement and that made the plan review somewhat obsolete for us.
There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year".
On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.
> On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering.
Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers.
The point of AI is:
1. It's bad engineering and everyone knows it.
2. Management is pushing it through anyway.
3. Deal with it, asshole.
The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.
It’s funny how different everyone’s experience is with this stuff. To me the diminishing returns are more around not going crazy with prototyping or running with xmax thinking all the time. I haven’t found it hard to stay under the usage limit of one $200/mo Claude and one $200/mo Codex subscription.
If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.
how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)
Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools.
I think building companies is going to look more generic in the future because intelligence is an API now.
So did we. I just asked my team to get personal accounts that I reimburse them for. It’s just a golden age loop though, the gravy train can’t go on forever unless we start building out thousands of data centers and associated renewable energy.
My workflow as well. If I had to gauge the direction of the wind, this produces the work of somewhere between 1-2 pre-AI engineers. My own code quality probably went down, along with my focus and the depth of understanding. Bug rates have increased. I don't have any stats to prove this.
What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant.
One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)
How can Smart Router achieve higher task completion rate compared to any of the base models if all it does is dynamically switch base models based on cost??
52 comments
[ 0.20 ms ] story [ 12.4 ms ] threadThere’s not even a distant comparison US lab rn which is sad
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
I think the more interesting lever is the fourth they mention: token efficiency.
> By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include.
I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.
1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes
2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model
Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.
- Spend most time prioritizing/discussing what to do.
- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)
- Use Opus 5 or Sol Med to execute
- Auto-fix bugs and CI until green + thermonuclear review skill x3.
- Manual interrogation of change/nits
- Come up with QA plan and have Codex Computer Use execute on it
- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)
I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.
In my experience at the begining of the full agentic coding loop in our company we were more hands on with the codebase and had better judgement over the plans. Now it is quite often that the inital plan after executed needs more refinement and that made the plan review somewhat obsolete for us.
(and mid-to-long-term, often also short-term end up cheaper than weaker models)
this might change soon if we are reaching a certain capability threshold
but right now that's still the case
unless you are working on throw-away trivial stuff where iteration speed and trying many speculative things might give you an edge
On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.
Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.
The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering.
Engineering payroll needed to be slashed and the C-Suite was happy to let their managers get some lashes in on the uppity workers.
The point of AI is:
1. It's bad engineering and everyone knows it.
2. Management is pushing it through anyway.
3. Deal with it, asshole.
The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
Edit: It is just as much about affirming the exalted status of management in comparison to labor. It's a bitter reinscription of the status hierarchy in tech; Management is good and labor is bad; Management is considerate and labor is ignorant; Management is strong and labor is weak; Management is superior and labor is inferior.
If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.
An AI-edited post by the way.
Kudos to databricks, I also find it interesting that such different companies (Stripe, Ramp, Databricks) are all building the exact same internal tools.
I think building companies is going to look more generic in the future because intelligence is an API now.
My workflow is very simple:
1. develop requirements for code change
2. take manual notes for implementation, maybe use LLM for some discovery/investigation
3. present notes to frontier LLM
4. develop implementation plan (bulk of work)
5. let LLM rip
6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions
7. get automated LLM review
8. get human review
this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc.
Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself.
Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one.
I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls.
But it is very, very productive.
Have you all attempted finetuning smaller OSS models on your repos for coding?
One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)