Based on a quick first skim of the abstract and the introduction, the results from hierarchical reasoning (HRM) models look incredible:
> Using only 1,000 input-output examples, without pre-training or CoT supervision, HRM learns to solve problems that are intractable for even the most advanced LLMs. For example, it achieves near-perfect accuracy in complex Sudoku puzzles (Sudoku-Extreme Full) and optimal pathfinding in 30x30 mazes, where state-of-the-art CoT methods completely fail (0% accuracy). In the Abstraction and Reasoning Corpus (ARC) AGI Challenge 27,28,29 - a benchmark of inductive reasoning - HRM, trained from scratch with only the official dataset (~1000 examples), with only 27M parameters and a 30x30 grid context (900 tokens), achieves a performance
of 40.3%, which substantially surpasses leading CoT-based models like o3-mini-high (34.5%) and Claude 3.7 8K context (21.2%), despite their considerably larger parameter sizes and context lengths, as shown in Figure 1.
I'm going to read this carefully, in its entirety.
I am extremely skeptical of a 27M parameter model being trained “from scratch” on 1000 datapoints. I am likewise incredulous of the lack of comparison with any other model which is trained “from scratch” using their data preparation. Instead they strictly compare with 3rd party LLMs which are massively more general purpose and may not have any of those 1000 examples in their training set.
I hope/fear this HRM model is going to be merged with MoE very soon. Given the huge economic pressure to develop powerful LLMs I think this can be done in just a month.
The paper seems to only study problems like sudoku solving, and not question answering or other applications of LLMs. Furthermore they omit a section for future applications or fusion with current LLMs.
I think anyone working in this field can envision their applications, but the details to have a MoE with an HRM model could be their next paper.
I only skimmed the paper and I am not an expert, sure other will/can explain why they don't discuss such a new structure. Anyway, my post is just blissful ignorance over the complexity involved and the impossible task to predict change.
Edit: A more general idea is that Mixture of Expert is related to cluster of concepts and now we would have to consider a cluster of concepts related by the time they take to be grasped, so in a sense the model would have in latent space an estimation of the depth, number of layers, and time required for each concept, just like we adapt our reading style for a dense math book different to a newspaper short story.
Skimming this, there is no reason why a MoE LLM system (whether autoregressive, diffusion, energy-based or mixed) couldn't be given a nested architecture that duplicates the layout of a HRM. Combining these in different ways should allow for some novel benchmarks around efficiency and quality, which will be interesting.
> "After completing the T steps, the H-module incorporates the sub-computation’s outcome (the final state L) and performs its own update. This H update establishes a fresh context for the L-module, essentially “restarting” its computational path and initiating a new convergence phase toward a different local equilibrium."
So they let the low-level RNN bottom out, evaluate the output in the high level module, and generate a new context for the low-level RNN. Rinse, repeat. The low-level RNNs are iterating backpropagation while the high-level is periodically kicking the low-level RNNs to get better outputs. Loops within loops. Composition.
Another interesting part:
> "Neuroscientific evidence shows that these cognitive modes share overlapping neural circuits, particularly within regions such as the prefrontal cortex and the default mode network. This indicates that the brain dynamically modulates the “runtime” of these circuits according to task complexity and potential rewards.
> Inspired by the above mechanism, we incorporate an adaptive halting strategy into HRM that enables `thinking, fast and slow'"
A scheduler that dynamically balances resources based on the necessary depth of reasoning and the available data.
I love how this paper cites parallels with real brains throughout. I believe AGI will be solved as the primitives we're developing are composed to extreme complexity, utilizing many cooperating, competing, communicating, concurrent, specialized "modules." It is apparent to me that human brain must have this complexity, because it's the only feasible way evolution had to achieve cognition using slow, low power tissue.
This work does have some very interesting ideas, specifically avoiding the costs of backpropagation through time.
However, it does not appear to have been peer reviewed.
The results section is odd. It does not include include details of how they performed the assesments, and the only numerical values are in the figure on the front page. The results for ARC2 are (contrary to that figure) not top of the leaderboard (currently 19% compared to HRMs 5% https://www.kaggle.com/competitions/arc-prize-2025/leaderboa...)
In fields like AI/ML, I'll take a preprint with working code over peer-reviewed work without any code, always, even when the preprint isn't well edited.
Everyone everywhere can review a preprint and its published code, instead of a tiny number of hand-chosen reviewers who are often overworked, underpaid, and on tight schedules.
If the authors' claims hold up, the work will gain recognition. If the claims don't hold up, the work will eventually be ignored. Credentials are basically irrelevant.
Think of it as open-source, distributed, global review. It may be messy and ad-hoc, since no one is in charge, but it works much better than traditional peer review!
Skepticism is an understatement. There are tons of issues with this paper. Why are they comparing results of their expert model that was trained from scratch on a single task to general purpose reasoning models? It is well established in the literature that you can still beat general purpose LLMs in narrow domain tasks with specially trained, small models. The only comparison that would have made sense is one to vanilla transformers using the same nr of parameters and trained on the same input-output dataset. But the paper shows no such comparison. In fact, I would be surprised if it was significantly better, because such architecture improvements are usually very modest or not applicable in general. And insinuating that this is some significant development to improve general purpose AI by throwing in ARC is just straight up dishonest. I could probably cook up a neural net in pytorch in a few minutes that beats a hand-crafted single task that o3 can't solve in an hour. That doesn't mean that I made any progress towards AGI.
I think that’s too harsh a position solely for not being peer reviewed yet. Neither of yhe original mamba1 and mamba2 papers were peer reviewed. That said, strong claims warrant strong proofs, and I’m also trying to reproduce the results locally.
Scepticism is generally always a good idea with ML papers. Once you start publishing regularly in ML conferences, you understand that there is no traditional form of peer review anymore in this domain. The volume of papers has meant that 'peers' are often students coming to grips with parts of the field that rarely align with what they are asked to review. Conference peer review has become a 'vibe check' more than anything.
Real peer review is when other experts independently verify your claims in the arXiv submission through implementation and (hopefully) cite you in their followup work. This thread is real peer review.
Enough already. Please. The paper + code is here for everybody to read and test. Either it works or it doesn't. Either people will build upon it or they won't. I don't need to wait 20 months for 3 anonymous dudes to figure it out.
> However, it does not appear to have been peer reviewed.
my observation is that peer reviewers never try to reproduce results or do basic code audit to check that there is no data leak for example to training dataset.
Do you consider yourself a peer? Feel free to review it.
A peer reviewer will typically comment that some figures are unclear, that a few relevant prior works have gone uncited, or point out a followup experiment that they should do.
That's about the extent of what peer reviewers do, and basically what you did yourself.
The fact that you are expecting a paper just published to have been peer reviewed already tells me that you are likely not familiar with the process. The first step to have your work peer reviewed is to publish it.
If I understand this correctly, it learns the rules of Sudoku by looking at 1,000 examples of (puzzle, solution) pairs. It is then able to solve previously unseen puzzles with 55% accuracy. If given millions of examples, it becomes almost perfect.
This is apparently without pretraining of any sort, which is kind of amazing. In contrast, systems like AlphaZero have the rules to go or chess built-in, and only learn the strategy, not the rules.
Off to their GitHub repository [1] to see this for myself.
As a cognitive psychologist, I highly suspected that, broadly speaking, this was the needed direction for AI. See Fuzzy Trace Theory[1].
Fuzzy Trace Theory basically suggests that memory (and cognition generally) works at multiple levels spanning verbatim representations to gist-level representations, that get bound together into memories. Recalling gist, the general idea, along with specific details, allows for powerful generalization and flexible retrieval pathways.
I appreciate the connections with neurology, and the paper itself doesn't ring any alarm bells. I don't think I'd reject it if it fell to me to peer review.
However, I have extreme skepticism when it comes to the applicability of this finding. Based on what they have written, they seem to have created a universal (maybe; adaptable at the very least) constraint-satisfaction solver that learns the rules of the constraint-satisfaction problem from a small number of examples. If true (I have not yet had the leisure to replicate their examples and try them on something else), this is pretty cool, but I do not understand the comparison with CoT models.
CoT models can, in principle, solve _any_ complex task. This needs to be trained to a specific puzzle which it can then solve: it makes no pretense to universality. It isn't even clear that it is meant to be capable of adapting to any given puzzle. I suspect this is not the case, just based on what I have read in the paper and on the indicative choice of examples they tested it against.
This is kind of like claiming that Stockfish is way smarter than current state of the art LLMs because it can beat the stuffing out of them in chess.
I feel the authors have a good idea here, but that they have marketed it a bit too... generously.
I really like this usage of recurrent modules to augment attention-based models, and I think this is a really cool result and a fruitful avenue for future work
This is really interesting, but does anyone think this is something that might generalize for ambiguous reasoning situations with more development? I am no expert, but sudoku and puzzles seem like very well-defined problem spaces.
I've been keeping an eye on this one as well. based on what the paper claims this would be huge. But i think like many here, we are waiting for either confirmation or denial of the claim via 3d parties. the concept behind them sounds legit, but id like to see it in practice.
> For ARC-AGI challenge, we start with all input-output example pairs in the training and the evalua- tion sets ... At test time, we proceed as follows for each test input in the evaluation set: ...
Very often I see people misuse the ARC-AGI data when training. The input examples in the evaluation set are not intended for training your AI system. It is a downside of ARC that its data is (somehow?) complicated enough for the clever people building AI systems to miss the point, and people report and compare results as a single percentage where the data mix used for training may not make the comparison applicable.
31 comments
[ 6.6 ms ] story [ 59.9 ms ] thread> Using only 1,000 input-output examples, without pre-training or CoT supervision, HRM learns to solve problems that are intractable for even the most advanced LLMs. For example, it achieves near-perfect accuracy in complex Sudoku puzzles (Sudoku-Extreme Full) and optimal pathfinding in 30x30 mazes, where state-of-the-art CoT methods completely fail (0% accuracy). In the Abstraction and Reasoning Corpus (ARC) AGI Challenge 27,28,29 - a benchmark of inductive reasoning - HRM, trained from scratch with only the official dataset (~1000 examples), with only 27M parameters and a 30x30 grid context (900 tokens), achieves a performance of 40.3%, which substantially surpasses leading CoT-based models like o3-mini-high (34.5%) and Claude 3.7 8K context (21.2%), despite their considerably larger parameter sizes and context lengths, as shown in Figure 1.
I'm going to read this carefully, in its entirety.
Thank you for sharing it on HN!
This smells like some kind of overfit to me.
The paper seems to only study problems like sudoku solving, and not question answering or other applications of LLMs. Furthermore they omit a section for future applications or fusion with current LLMs.
I think anyone working in this field can envision their applications, but the details to have a MoE with an HRM model could be their next paper.
I only skimmed the paper and I am not an expert, sure other will/can explain why they don't discuss such a new structure. Anyway, my post is just blissful ignorance over the complexity involved and the impossible task to predict change.
Edit: A more general idea is that Mixture of Expert is related to cluster of concepts and now we would have to consider a cluster of concepts related by the time they take to be grasped, so in a sense the model would have in latent space an estimation of the depth, number of layers, and time required for each concept, just like we adapt our reading style for a dense math book different to a newspaper short story.
So they let the low-level RNN bottom out, evaluate the output in the high level module, and generate a new context for the low-level RNN. Rinse, repeat. The low-level RNNs are iterating backpropagation while the high-level is periodically kicking the low-level RNNs to get better outputs. Loops within loops. Composition.
Another interesting part:
> "Neuroscientific evidence shows that these cognitive modes share overlapping neural circuits, particularly within regions such as the prefrontal cortex and the default mode network. This indicates that the brain dynamically modulates the “runtime” of these circuits according to task complexity and potential rewards.
> Inspired by the above mechanism, we incorporate an adaptive halting strategy into HRM that enables `thinking, fast and slow'"
A scheduler that dynamically balances resources based on the necessary depth of reasoning and the available data.
I love how this paper cites parallels with real brains throughout. I believe AGI will be solved as the primitives we're developing are composed to extreme complexity, utilizing many cooperating, competing, communicating, concurrent, specialized "modules." It is apparent to me that human brain must have this complexity, because it's the only feasible way evolution had to achieve cognition using slow, low power tissue.
This work does have some very interesting ideas, specifically avoiding the costs of backpropagation through time.
However, it does not appear to have been peer reviewed.
The results section is odd. It does not include include details of how they performed the assesments, and the only numerical values are in the figure on the front page. The results for ARC2 are (contrary to that figure) not top of the leaderboard (currently 19% compared to HRMs 5% https://www.kaggle.com/competitions/arc-prize-2025/leaderboa...)
In fields like AI/ML, I'll take a preprint with working code over peer-reviewed work without any code, always, even when the preprint isn't well edited.
Everyone everywhere can review a preprint and its published code, instead of a tiny number of hand-chosen reviewers who are often overworked, underpaid, and on tight schedules.
If the authors' claims hold up, the work will gain recognition. If the claims don't hold up, the work will eventually be ignored. Credentials are basically irrelevant.
Think of it as open-source, distributed, global review. It may be messy and ad-hoc, since no one is in charge, but it works much better than traditional peer review!
Real peer review is when other experts independently verify your claims in the arXiv submission through implementation and (hopefully) cite you in their followup work. This thread is real peer review.
Which is fine, because peer review is not a good proxy for quality or validity.
Enough already. Please. The paper + code is here for everybody to read and test. Either it works or it doesn't. Either people will build upon it or they won't. I don't need to wait 20 months for 3 anonymous dudes to figure it out.
my observation is that peer reviewers never try to reproduce results or do basic code audit to check that there is no data leak for example to training dataset.
A peer reviewer will typically comment that some figures are unclear, that a few relevant prior works have gone uncited, or point out a followup experiment that they should do.
That's about the extent of what peer reviewers do, and basically what you did yourself.
This is apparently without pretraining of any sort, which is kind of amazing. In contrast, systems like AlphaZero have the rules to go or chess built-in, and only learn the strategy, not the rules.
Off to their GitHub repository [1] to see this for myself.
[1] https://github.com/sapientinc/HRM
[1] - https://arxiv.org/abs/2210.03629
Fuzzy Trace Theory basically suggests that memory (and cognition generally) works at multiple levels spanning verbatim representations to gist-level representations, that get bound together into memories. Recalling gist, the general idea, along with specific details, allows for powerful generalization and flexible retrieval pathways.
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC4979567/
However, I have extreme skepticism when it comes to the applicability of this finding. Based on what they have written, they seem to have created a universal (maybe; adaptable at the very least) constraint-satisfaction solver that learns the rules of the constraint-satisfaction problem from a small number of examples. If true (I have not yet had the leisure to replicate their examples and try them on something else), this is pretty cool, but I do not understand the comparison with CoT models.
CoT models can, in principle, solve _any_ complex task. This needs to be trained to a specific puzzle which it can then solve: it makes no pretense to universality. It isn't even clear that it is meant to be capable of adapting to any given puzzle. I suspect this is not the case, just based on what I have read in the paper and on the indicative choice of examples they tested it against.
This is kind of like claiming that Stockfish is way smarter than current state of the art LLMs because it can beat the stuffing out of them in chess.
I feel the authors have a good idea here, but that they have marketed it a bit too... generously.
Very often I see people misuse the ARC-AGI data when training. The input examples in the evaluation set are not intended for training your AI system. It is a downside of ARC that its data is (somehow?) complicated enough for the clever people building AI systems to miss the point, and people report and compare results as a single percentage where the data mix used for training may not make the comparison applicable.