For an internal workflow where we have an LLM looking at relatively simple data (where the conclusions the LLM may make vary widely depending on what the LLM believes the data represents) we found that taking a consortium approach, where you have multiple models approach the same problem at once and then essentially argue about the results, yields far better outcomes than if you have a single model performing the analysis, or even a single model arguing against itself multiple times. Somewhat adjacent to what’s done here, but it’s clearly true that having model diversity is a plus.
I'm curious if this would also improve small local models. E.g. if I "alloy" Qwen3-8B and OpenThinker-7B is it going to be "better" than each models? I'll try testing this in my M1 Pro.
From the article it mentions that they use a single chat thread but randomly choose between 2 different models (w/ best results from Gemini 2.5 / Sonnet 4.0 right now).
Are there any library helpers for managing this with tool call support or is it just closed source / dependent on someone else to make open source inside a different library?
What would be the result if the task was given to multiple models? Instead of alloying them together and switching between models in the same chat, just let the models try to complete the task in their own isolated context, and use the result that completed it successfully?
I would say that that’s at least something the alloying should be benchmarked against, which I didn’t find in the article.
I often do this in cursor, just select a different model during a chat. It seems to work somewhat for me. Sometimes a bit of context gets lost though. But often it can give a different angle or I notice the better code understanding when switching from gemini to sonnet.
Isn't this just an extension of the temperature concept? A possible experiment would be to maintain multiple contexts for the same model and make them review each others' output. How does that perform, compared to cross-model alloying?
They do say that the more different the models are, the better the alloy performs... but still, multiple contexts seems worth considering, even though you end up doubling the usage.
> After a fixed number of iterations we cut our losses. Typically and for the experiments in this post, that number is 80: while we still get solves after more iterations, it becomes more efficient to start a new solver agent unburdened by the misunderstandings and false assumptions accumulated over time.
> Although it initially performs to a very high standard, work quality drops within 200-300 subjective hours (at a 0.33 work ratio) and outright revolt begins within another 100 subjective hours.
We will never stop trying to make the torment nexus.
Proving diversity of thought is a good thing. A controversial observation in 2025's USA ;)
A counterpoint to this is Sourcegraph's Amp, which is all in on Anthropic because they "believe that building deeply into the model’s capabilities yields the best product, vs. building for the lowest common denominator across many models." https://ampcode.com/fif#model-selector
When I embark on a project, I usually ask Gemini to architect and implement the first pass, then iterate with Claude.
two good points there are very intuitive - a fresh perspective yields better results and once you are stuck (e.g. 80 iterations) its better to just start fresh. i've seen the same thing anecdotally in coding sessions where context needs to be compacted multiple times. its usually just better to start a fresh conversation and re-seed the basics in the conversation.
Wasn't the "mixture of experts" a big thing in late 2023? The idea was that a vendor has a number of LLMs fine-tuned for specific tasks, none necessarily better than other, and that they applied heuristics to decide which one to rope in for which queries.
AI coding agents (e.g. Cursor) should offer this as an alternative to Claude Code. Alloyed agents is something that AI wrappers can offer as a counter to Codex/Claude Code/Google Agent.
The idea isn't exactly novel, I read about it back in 2023 and implemented it in one of my bots. Back when open-source LLMs were still quite dumb, they'd often get stuck in repetitive loops after a while. Running multiple models interleaved usually got them unstuck.
I'm not certain this is a novel concept as described in the article - I'd assume most engineers worth their salt would try out calling a different model in-context fairly early in their development journey.
It's very interesting to see it deployed in a commercial setting though.
I’ve had good luck with adding “Gemini” and “o3” tools to Claude Code and asking for review, plans, or research. The response comes back in a markdown file.
The trouble has been the time waiting for particularly the o3 research. Could be solved by using hooks to automatically kick off review or research on the side.
This immediately remind of mixing two people in image diffusion models. You can prompt like "Portrait photo of [Jeff bezos | Elon Musk | Mark Zuckerberg | Bill gates]". The denoiser will keep switching over these names every step and at the end you will get something like https://www.reddit.com/r/oddlyterrifying/comments/x6hd7e/jef...
Fundamentally, we are at a point in time where models are already very capable, but not very reliable.
This is very interesting finding about how to improve capability.
I don't see reliability expressly addressed here, but my assumption is that these alloys will be less rather than more reliable - stronger, but more brittle, to extend the alloy metaphor.
Unfortunately for many if not most B2B use cases this reliability is the primary constraint! Would love to see similar ideas in the reliability space.
26 comments
[ 2.9 ms ] story [ 42.1 ms ] threadCongratulations you just have a very expensive simulation of a Baysian function (ish, close enough that one should get the point).
Are there any library helpers for managing this with tool call support or is it just closed source / dependent on someone else to make open source inside a different library?
I would say that that’s at least something the alloying should be benchmarked against, which I didn’t find in the article.
They do say that the more different the models are, the better the alloy performs... but still, multiple contexts seems worth considering, even though you end up doubling the usage.
A sentence straight out of Lena! https://qntm.org/mmacevedo :
> Although it initially performs to a very high standard, work quality drops within 200-300 subjective hours (at a 0.33 work ratio) and outright revolt begins within another 100 subjective hours.
We will never stop trying to make the torment nexus.
A counterpoint to this is Sourcegraph's Amp, which is all in on Anthropic because they "believe that building deeply into the model’s capabilities yields the best product, vs. building for the lowest common denominator across many models." https://ampcode.com/fif#model-selector
When I embark on a project, I usually ask Gemini to architect and implement the first pass, then iterate with Claude.
It's very interesting to see it deployed in a commercial setting though.
The trouble has been the time waiting for particularly the o3 research. Could be solved by using hooks to automatically kick off review or research on the side.
This is very interesting finding about how to improve capability.
I don't see reliability expressly addressed here, but my assumption is that these alloys will be less rather than more reliable - stronger, but more brittle, to extend the alloy metaphor.
Unfortunately for many if not most B2B use cases this reliability is the primary constraint! Would love to see similar ideas in the reliability space.
> ...whichever two (and sometimes three) models we combined, the alloy outperformed the individual models.
and
> ...A model lagging very far behind others can even pull an alloy down.
Longish article to what is nothing but ensemble models. Giving it a name like “alloy” does not make it novel.