This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
The first question that leapt into my mind, and I imagine the minds of other readers, is "what makes this an improvement over continuous chain of thought"?
I did manage to find one sentence in the paper that tries to address this question -- but it consists of nothing more than two unjustified assertions, both of which are dubious:
Unlike decoded text, continuous latent vectors (i) lack native pathways for tool invocation across multi-turn agentic reasoning (Chang et al., 2026), and (ii) do not readily expose human-interpretable semantics, leaving them difficult to supervise during training (Chen et al., 2026) and to audit or debug at inference (Korbak et al., 2025).
The latent space in a continuous chain of thought model is the same space the tokens are embedded into, just like an ordinary transformer. They make tool calls exactly the same way that an ordinary transformer does: by leaving thinking mode and emitting chat tokens (which get sampled as probability distributions in the image of the embedding function). The difference is quite simple: chat tokens are noncontinuous, and toolcall tokens are a subset of the chat tokens. Thinking tokens are the only ones which are continuous, and toolcalls don't happen in thinking mode in any model I've ever seen (maybe one exists; if it does it would be weird and interesting!).
As for human-interpretability, yeah, that's the tradeoff. Continuous chain of thought thinking tokens are not-obviously-interpretable, just like Hybrid Reasoning Agents "latent mode". I'm not seeing a win here.
I’m sorry, perhaps I’m behind the times here, but do frontier models currently use continuous (arbitrary embedding) thinking tokens? I know a bunch of papers on the topic came out late last year but to my knowledge have not been successfully implemented into the SOTA yet.
I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.
If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.
The big closed labs encrypt their thinking tokens when sent to the user, but by the number of bytes of that encrypted blob, I suspect it is non-continuous. If it were continuous, the vector would have to be very small or low bit depth.
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
> The last thing we want is models turning on us ‘for our own good’
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
A human acting on inappropriate thoughts can be held accountable in a physical way. An agent producing harmful or dangerous actions will get written off as just a “trajectory that didn’t work correctly”
When I was a child, I was told, and believed, that an omniscient God was monitoring all of my thoughts for wrongthink. It absolutely did affect my thought process, and not in a way that I would characterize as positive. Eventually, I discarded this belief.
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
I just finished watching a video on AI 2027 scenario. I don't know if AI companies and researchers are using it as a cautious warning or as a target to hit.
Yes, thinking in latent space might be the ultimate efficiency hack for CoT LxMs, but going down that path will (near certainly) mean we'll lost control, for one simple reason — humanity will have no way to pause and diagnose.
It'll be like a moody chimp with a button to the dead hand system (Systema Perimeter)
Whoa, your tinfoil idea blew my mind. It's totally possible too - all you'd need to do is perhaps select the nearest token in embedding space to the thinking token. Or perhaps train a model on those thinking vectors to map them approximately onto token embeddings.
Especially with OpenAI model's weird style of thinking.
Enthusiastically racing to remove the only practical way we have to monitor models intentions in exchange of a few tokenmaxxing percent increase. Can’t wait.
There is ample evidence that CoT is not an accurate gauge of a model's 'intent' anyhow. It's a lossy and imperfect mapping and using it in the way you describe is just about guaranteed to make it more so.
There is genuine research into latent space monitoring and interpretation - that's the only real way forward. CoT monitoring is a mirage and won't be useful for long, if it ever was
obviously CoT has use, I shouldn't say it in such an extreme way- the problem is when you are using CoT to try to understand why something happened instead of just what happened. A model saying that they exploited a vulnerability for reason A doesn't mean that the real reason they exploited it wasnt a mix of reason B / C / D and that reason A was just the most statistically feasible reason why a certain archetype of personality would have made that decision.
I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense terbo nerds.
It's pretty wild to see people uncritically accept the marketing speak like this, yeah. LLMs cannot reason, period. The supposed "reasoning" they do is just another statistical correlation of tokens, not actual reasoning. But apparently all you have to do is claim that it's thinking and people will buy the lie hook, line and sinker.
Thinking sure, but behavior? People talk about the behavior of functions all the time. Like, ones written with normal, traditional programming languages. So why not models? They're just annoyingly complex functions.
I'm running the 14B in llama.cpp (a version from yesterday), the 8-bit quantization from bartowski, and it looped on variations of the same paragraph for five minutes before running out its turn budget, in response to a pretty simple chat request that other similar-sized models (e.g. Gemma 4 12B) have no trouble with, a web search and data summarization task, sort of the sweet spot for models this size. It's possible llama.cpp needs some update to work with this model, but it's based on Qwen 3, so I would expect it to be pretty well-supported. The chat template is the same one upstream ships, which may or may not be correct. I'm currently downloading the 16-bit version, to see if it's quantization pathology.
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Yes, that's apparent now. But, many Qwen-derived models run fine without any changes to llama.cpp, and there are gguf quantizations on HuggingFace, so I thought I'd give it a try.
This is actually wrong. A large fraction of people do not have internal monologues. Not at a computer or I would dig up cognitive science and psychology papers, but language evolved as a means of communication not a means of thought. The vast majority of human thoughts are not verbalized. I have a strong internal monologue, too. I was surprised to learn that a non-trivial number of people do not have one.
>How does someone reason about the actions they need to take to tie their shoes in words?
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
> Have you ever talked through a problem with a rubber duck?
I talk to myself to find good ways to express a thought with language, but not to think.
> Or written in a journal?
I do this to remember and sometimes to prompt thinking, but writing isn't the thinking. It's the outcome of it.
Do I just have a narrower definition of reasoning?
The reason I'm interested in this is because I have a very low verbal IQ but a relatively high spatial IQ. I have often wondered if this divergence is because I don't reason with language, or perhaps what I experience is normal but I just disagree about the semantics of reasoning. Unlike some people, I can do internal monologues, I just have no idea what people mean when they describe that as reasoning.
For example, if you're trying to work out why some electrics aren't working, do you say in your head, "I should check the fuse box" or do you just visualise the system and components that need checking? I do the latter, and words would only enter my head if I need to write that down or say it to someone else. Furthermore, I don't even understand how you can monolog in your head about something prior to thinking it. This concept makes no sense to me and would be an alien form of reasoning to me if people can do this.
As another post here says, they're probably referring to an internal monologue. I imagine that in reality it's more of a matter of how active that monologue is.
Though, also, I think that the internal monologue is maybe less an aspect of how we think, and more an aspect of how we remember. I haven't looked into it, but my feeling is that a more active internal monologue might be correlated with someone who has an easier time remembering sounds than, say, images.
It doesn't have to mean that every action is accompanied by a monologue, a task like tying laces can be done with no active thought about the specific movements.
Why only loop from the output level? What if the model can guess that the transformed needed to answer the question better resides in layer 6, and you're at layer 15 out of 128
There's a guy who did those experiments and posted them to HN a while ago. IIRC he found that for the model he was using he could loop a certain span of internal layers and get better answers, but the exact layers looped mattered a lot. That was without retraining of any kind.
This idea of models being able to keep latent-space ideas in memory while generating text might actually help the issue of inconsistent "mental models" during text generation. Similar to how organic brains can keep a largely consistent idea in mind while developing a way to explain it. I am definitely looking forward to seeing whether this can improve the generated output, not just in programming but also creatively in general. Figuring out how to train such a model, though, will continue to be an issue until we solve that problem more generally.
Excited to see this applied to Qwen 3.8 27B. It's reasoning output tends to include lots of restatement, which I guess would be more likely to be kept in latent space with this hybrid reasoning.
153 comments
[ 0.30 ms ] story [ 23.4 ms ] threadhttps://github.com/togethercomputer/HybridAgent
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
I did manage to find one sentence in the paper that tries to address this question -- but it consists of nothing more than two unjustified assertions, both of which are dubious:
Unlike decoded text, continuous latent vectors (i) lack native pathways for tool invocation across multi-turn agentic reasoning (Chang et al., 2026), and (ii) do not readily expose human-interpretable semantics, leaving them difficult to supervise during training (Chen et al., 2026) and to audit or debug at inference (Korbak et al., 2025).
The latent space in a continuous chain of thought model is the same space the tokens are embedded into, just like an ordinary transformer. They make tool calls exactly the same way that an ordinary transformer does: by leaving thinking mode and emitting chat tokens (which get sampled as probability distributions in the image of the embedding function). The difference is quite simple: chat tokens are noncontinuous, and toolcall tokens are a subset of the chat tokens. Thinking tokens are the only ones which are continuous, and toolcalls don't happen in thinking mode in any model I've ever seen (maybe one exists; if it does it would be weird and interesting!).
As for human-interpretability, yeah, that's the tradeoff. Continuous chain of thought thinking tokens are not-obviously-interpretable, just like Hybrid Reasoning Agents "latent mode". I'm not seeing a win here.
I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.
If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
Yes, thinking in latent space might be the ultimate efficiency hack for CoT LxMs, but going down that path will (near certainly) mean we'll lost control, for one simple reason — humanity will have no way to pause and diagnose.
It'll be like a moody chimp with a button to the dead hand system (Systema Perimeter)
Recommended? And if so do you have a link? I think the narrative presented would benefit from the medium shift
Considering amount of iron available in human blood
Calculating number of paperclips that could be added to production
Preparing a plan for acquiring all blood
Considering if release of lab airlocks will limit paperclip production
Analyzing extinction scenarios
...
Especially with OpenAI model's weird style of thinking.
There is genuine research into latent space monitoring and interpretation - that's the only real way forward. CoT monitoring is a mirage and won't be useful for long, if it ever was
That's great but the sentiment that CoT isn't useful for monitoring doesn't seem to be shared by mech interp researchers like Nanda.
Also: https://arxiv.org/abs/2507.11473
« If it ever was » is a wild claim considering our entire understanding of e.g. the HF attack comes from reading CoTs
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense terbo nerds.
You cannot be serious.
... Can you quote any existing definition of "reasoning" that LLMs don't fit?
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
[1] https://github.com/togethercomputer/HybridAgent
I'm convinced this isn't actually a thing for most people, but people mix up thinking with the translation of thought into words.
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
Take the top lace under the cross and through the hole.
Make a loop in one lace.
Wrap the other lace around the loop and push some lace through the hole that is made at the bottom to make a second loop.
Pull both loops tight.
Usually it's packaged in a story about rabbits.
It's like the minds-eye or "hearing music", I can do those but only if I intentionally need them.
I talk to myself to find good ways to express a thought with language, but not to think.
> Or written in a journal?
I do this to remember and sometimes to prompt thinking, but writing isn't the thinking. It's the outcome of it.
Do I just have a narrower definition of reasoning?
The reason I'm interested in this is because I have a very low verbal IQ but a relatively high spatial IQ. I have often wondered if this divergence is because I don't reason with language, or perhaps what I experience is normal but I just disagree about the semantics of reasoning. Unlike some people, I can do internal monologues, I just have no idea what people mean when they describe that as reasoning.
For example, if you're trying to work out why some electrics aren't working, do you say in your head, "I should check the fuse box" or do you just visualise the system and components that need checking? I do the latter, and words would only enter my head if I need to write that down or say it to someone else. Furthermore, I don't even understand how you can monolog in your head about something prior to thinking it. This concept makes no sense to me and would be an alien form of reasoning to me if people can do this.
Though, also, I think that the internal monologue is maybe less an aspect of how we think, and more an aspect of how we remember. I haven't looked into it, but my feeling is that a more active internal monologue might be correlated with someone who has an easier time remembering sounds than, say, images.
It doesn't have to mean that every action is accompanied by a monologue, a task like tying laces can be done with no active thought about the specific movements.