There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
Yep. It's a real shame that the labs are incentivized not to go in this direction. They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent.
They naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth.
I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful.
Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.
isn't this also the threat to frontier AI houses? As in they want you to expend tokens in their ecosystem, but the optimization at 100x is their profit?
IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.
It is nice to have a model that can "do it all", though. And surely that's still the end goal? Like how MoE is still somewhat popular in certain areas even after its heyday.
I am wondering if models will end up being some sort of evolution of MoE where it has something internally like the model the author refers to that gets surfaced when it needs to search in some way. I guess it makes sense; our own brains have so many distinct task-specific regions.
Yes, agreed, but I think it’s going to be difficult to have a high quality model that “does it all” and have it be local. For a quality “does it all” model, you’re going to need a lot of parameters and that means you’re almost always going to be running in the cloud. But it seems like you could probably get a smaller, focused model that runs locally and is also high quality. In other words, I want the programming expert portion of a 2T model that is maybe 35B or 70B parameters or whatever but it’s running locally (and yes, I know you don’t just carve out an expert from a larger model, but conceptually that’s what I’m after).
Yeah, I always thought the future of this stuff would be hot-pluggable MoE modules or LORAs that are able to be downloaded and applied/used at will like how skills have become a thing.
Like, atm most architectures seem limited by a single context and fixed architecture with no hot loading. Especially for robotics, being able to load/unload various specialised skills on limited mobile hardware will (I hope) definitely become a thing.
Bit unrelated, I realized that z.ai gives you access to deepseek 4 flash. It's incredible how well it performs when given a detailed spec. I'm not sure I've seen a model one-shot like that, and I was already impressed by gemma 4's speed and efficiency.
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
<founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.
totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet
I'm curious about this too as I've been working with a 1300 page document on procedures for [industry]. Where every single section is primarily about [industry] with minor differences in verbs actions etc for the procedures.
I've found with traditional embeddings that obviously you're getting an average of the content of the chunk even with the semantic awareness magic. And our (or I guess the) core problem seems to be a lack of enough contrast between chunks with makes one-shot pure embedding based RAG extremely difficult and low quality.
One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.
(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.
a few ideas i have yet to validate are:
- prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others)
- actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance)
- you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers
I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.
founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.
On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
the page shows the exact trace of all the models we are comparing against and the aggregate scores
we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)
This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.
Yes, and it's only logical things move in this direction because there's massive hardware incentive to do so. If frontier models can be broken down into small models, networks of smaller-GPUs can be utilized. Right now the smaller GPUs are basically paper weights for frontier intelligence.
The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper.
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
IMHO, what is broken is retrieval, the whole blind chunking which was first generation is still the default standard in RAG, this has to be changed. I'm saying this by seeing the results when we used richer parent candidates model for LLM and child segments as search probes. Even without reranking we got solid results.
Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
I (and I imagine many others) would love to use something like this, but can't, because my data is too sensitive to be uploaded to a cloud of which I have no gaurantees of privacy/security.
Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access to my data.
People are building agentic search one of three ways:
1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction
2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some other naive retrieval system. Let the agent figure it out. But it’ll consume a lot of tokens to get good results as it iterates and loops.
3. A model trained for retrieval. Give it dumb retriever like in (2) but it is fine tuned on the task as in (1).
This article is 3. But we’ve been seeing this all year with SID.ai, Gleans Waldo model etc. if this interests you I’d check those out, particularly SID.
Thank you for the link! Super interesting and I appreciate how it was written.
It seems like a lot of the problems I have been running into with RAG on large/complex documents with generally low contrast in the information is not one that has been perfectly solved yet - here I am thinking I'd been a bit behind.
It's just unfortunate that none of the cloud providers are flexible enough to deal with the pace of change. Probably going to have to shove one of those 8b~ models into an instance to use when needed.
It's interesting you mention late interaction (retrieval), I had recently been using ChatGPT as a mirror to throw ideas back at me on this issue and had been musing about how nice it would be to have some sort of hierarchical embeddings that capture a whole chunk, then sentences and then sentence fragments or individual word and it seems that that fits the bill!
Nice things like this, get "blue-washed" by the parent org's tech. For general adoption, kakebase should have been swapped out for a more generic non-databricks tech. 'Blue-wash' is a reference to an IBM's practice.
Models without tools and harnesses are not really that useful. My observation is that the tool ux is driving most progress at this point. There are of course open source tools and harnesses but they require more effort to setup properly.
The key challenge is to pick the right model for the right task or sub task and doing that automatically rather than manually. A big part of the problem here is that everybody is picking the most expensive and resource intensive models by default just in case they hit something that is a bit more difficult to get right. It's overkill. Most work people actually do is completely routine and would not have been a challenge for most of the mainstream OSS models.
I'm starting to suffer a bit from model fatigue. There are announcements almost on a daily basis about this or that new model. I can't keep up with that and I don't have time to try them out or evaluate them. I don't want to waste brain cycles on which one to use. I just want to get shit done without micromanaging AI models.
All this marketing BS and confusing naming isn't helping either. It seems a lot of that is just about tricking people into picking the expensive model so they'll burn through more tokens.
49 comments
[ 16.0 ms ] story [ 1024 ms ] threadThey naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth.
I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful.
Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.
I am wondering if models will end up being some sort of evolution of MoE where it has something internally like the model the author refers to that gets surfaced when it needs to search in some way. I guess it makes sense; our own brains have so many distinct task-specific regions.
Like, atm most architectures seem limited by a single context and fixed architecture with no hot loading. Especially for robotics, being able to load/unload various specialised skills on limited mobile hardware will (I hope) definitely become a thing.
How? Can you give details?
totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet
the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
I've found with traditional embeddings that obviously you're getting an average of the content of the chunk even with the semantic awareness magic. And our (or I guess the) core problem seems to be a lack of enough contrast between chunks with makes one-shot pure embedding based RAG extremely difficult and low quality.
a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers
if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
- founder of castform
will share the full results soon!
the page shows the exact trace of all the models we are comparing against and the aggregate scores
we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)
Nor of how much faster their custom model performs?
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access to my data.
1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction
2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some other naive retrieval system. Let the agent figure it out. But it’ll consume a lot of tokens to get good results as it iterates and loops.
3. A model trained for retrieval. Give it dumb retriever like in (2) but it is fine tuned on the task as in (1).
This article is 3. But we’ve been seeing this all year with SID.ai, Gleans Waldo model etc. if this interests you I’d check those out, particularly SID.
I wrote about these 3 approaches here https://softwaredoug.com/blog/2026/06/08/three-kinds-of-agen...
It seems like a lot of the problems I have been running into with RAG on large/complex documents with generally low contrast in the information is not one that has been perfectly solved yet - here I am thinking I'd been a bit behind.
It's just unfortunate that none of the cloud providers are flexible enough to deal with the pace of change. Probably going to have to shove one of those 8b~ models into an instance to use when needed.
It's interesting you mention late interaction (retrieval), I had recently been using ChatGPT as a mirror to throw ideas back at me on this issue and had been musing about how nice it would be to have some sort of hierarchical embeddings that capture a whole chunk, then sentences and then sentence fragments or individual word and it seems that that fits the bill!
The key challenge is to pick the right model for the right task or sub task and doing that automatically rather than manually. A big part of the problem here is that everybody is picking the most expensive and resource intensive models by default just in case they hit something that is a bit more difficult to get right. It's overkill. Most work people actually do is completely routine and would not have been a challenge for most of the mainstream OSS models.
I'm starting to suffer a bit from model fatigue. There are announcements almost on a daily basis about this or that new model. I can't keep up with that and I don't have time to try them out or evaluate them. I don't want to waste brain cycles on which one to use. I just want to get shit done without micromanaging AI models.
All this marketing BS and confusing naming isn't helping either. It seems a lot of that is just about tricking people into picking the expensive model so they'll burn through more tokens.