CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models

Abstract: LLMs have achieved strong reasoning capabilities, though often at the cost of massive parameter counts and expensive inference. In this work, we explore a different direction: adaptive reasoning depth in compact LLMs. We present CosmicFish-HRM, a compact LLM built around a Hierarchical Reasoning Module (HRM) that dynamically allocates computational effort during inference. Instead of applying fixed computation to every input, the model iterates through high-level and low-level reasoning cycles and learns when to halt based on input complexity. CosmicFish-HRM combines this adaptive reasoning core with modern transformer components including Grouped Query Attention, RoPE, and SwiGLU activations. While the additional reasoning infrastructure introduces overhead at small scale, we hypothesize that this tradeoff becomes increasingly favorable as model size grows and the relative cost of the HRM core diminishes. Our results show that the model learns non-uniform reasoning behavior, allocating different numbers of reasoning steps across tasks and inputs. These findings suggest that adaptive reasoning depth may offer a promising alternative to relying solely on parameter scale for reasoning capability.

• The paper integrates a Hierarchical Reasoning Module with learned halting into an 82.77M-parameter decoder-only language model, producing input-dependent reasoning depth without explicit reasoning supervision.
• CosmicFish-HRM shows variable computation with an overall mean of 2.681 reasoning steps and a standard deviation of 5.949, but trails comparable fixed-depth models on HellaSwag, PIQA, and WinoGrande.
• The results suggest adaptive reasoning is feasible in compact language models, while the proposed scaling hypothesis—that HRM overhead becomes proportionally smaller in larger models—remains untested.

CosmicFish-HRM is a compact 82.77M-parameter decoder-only LLM that integrates a Hierarchical Reasoning Module (HRM) between transformer stacks, enabling inference-time adaptive reasoning depth via a learned halting mechanism. The paper's central claim is deliberately modest: rather than demonstrating that adaptive computation improves benchmark accuracy, it establishes that non-uniform, input-dependent reasoning behavior can emerge inside an autoregressive LLM without hand-crafted stopping rules or explicit reasoning supervision — while conceding that this comes at a measurable cost in raw performance at compact scale.

Motivation and positioning

The work departs from two dominant efficiency paradigms. Compression approaches such as quantization, pruning, and distillation (Hinton et al., 2015) shrink large models but preserve fixed-depth computation: every input traverses the same pipeline regardless of difficulty. Data-centric approaches to small models, exemplified by TinyLlama (Zhang et al., 2024) and the Phi series (Gunasekar et al., 2023), improve capability through corpus curation while leaving the architecture unchanged. CosmicFish-HRM instead varies how much computation each input receives.

The architecture builds directly on the Hierarchical Reasoning Model of Wang et al. (Wang et al., 26 Jun 2025), which paired a slow high-level module with a fast low-level module and achieved strong results on structured tasks such as Sudoku, mazes, and ARC-style puzzles despite a small parameter count. The original HRM, however, was a task-specific reasoning engine with no autoregressive language modeling capability. The contribution here is to transplant hierarchical recurrent reasoning into a decoder-only LM trained on open-ended text, testing whether its adaptive dynamics survive that transfer.

The paper also situates itself within the adaptive-computation lineage: Adaptive Computation Time (Graves, 2016), PonderNet (Banino et al., 2021), Universal Transformers (Dehghani et al., 2018), early-exiting in Confident Adaptive Language Modeling (Schuster et al., 2022), and token-level routing in Mixture-of-Depth (Raposo et al., 2024). Unlike these, the HRM core performs iterative cycles across two abstraction levels before generation proceeds, governed by a learned halting policy trained jointly with the language modeling objective.

Architecture

The model follows a three-stage pipeline: six pre-norm transformer layers for contextual encoding, the HRM reasoning core, and six output transformer layers feeding a weight-tied LM head. The backbone uses modern components throughout — RMSNorm, RoPE (Su et al., 2021), Grouped Query Attention with 8 query heads sharing 4 KV heads (Ainslie et al., 2023), and SwiGLU feedforward layers (Shazeer, 2020) — at embedding dimension $d = 448$, vocabulary size 50,304, and context length 512.

The HRM core maintains two recurrent states initialized identically from the encoder output. At each reasoning step, the low-level module $\mathcal{L}$ (4 transformer blocks) runs $c_L = 2$ cycles conditioned on the high-level state; the updated low-level representation then conditions the high-level module $\mathcal{H}$ (4 blocks) over $c_H = 2$ cycles. This alternating update creates a bidirectional feedback loop between abstract and fine-grained states, up to a maximum of $S_{\max} = 16$ steps.

Halting is decided by a lightweight head computing halt/continue scores from mean-pooled high-level representations. Three mechanisms shape the halting behavior:

• Exploration during training: with probability $p_{\text{explore}} = 0.1$, halting is deferred until a uniformly sampled step count, exposing the network to longer trajectories.
• Inference biasing: a halt bias $\delta = 0.35$ is added at inference time, mildly favoring earlier stopping.
• Step penalty: the training loss adds $\lambda \bar{S}$ with $\lambda = 0.01$, where $\mathcal{L}$0 is the batch-mean number of reasoning steps, applying weak pressure toward efficient halting.

Notably, the paper frames this as a learned adaptive-computation mechanism integrated into the forward pass rather than a full reinforcement learning formulation — a deliberate simplification whose consequences for optimality of the halting policy are not analyzed.

Experimental results

Training used the 10B-token CosmicSet corpus (web text, Wikipedia, code, mathematics, research papers) for 27,500 iterations at effective batch size 128, reaching a final validation loss of 3.36.

Zero-shot evaluation against comparable-scale baselines shows CosmicFish-HRM underperforming across the board:

Model	HellaSwag	PIQA	WinoGrande
GPT-2 Small (117M)	29.7	62.5	50.7
OPT-125M	30.6	62.6	52.9
Pythia-160M	29.4	62.1	52.8
CosmicFish-90M (no HRM)	27.9	59.8	50.6
CosmicFish-HRM	26.2	58.1	50.7

The authors state plainly that the model does not outperform conventional transformers of similar size, attributing this to capacity allocation: in a sub-100M model, the recurrent reasoning infrastructure, halting machinery, and dual-state interaction consume a substantial fraction of the parameter budget that would otherwise serve raw language modeling. The paper advances a scaling hypothesis — that the HRM overhead becomes proportionally negligible as the backbone grows while adaptive behavior persists — but presents no experimental evidence for it. This hypothesis is the load-bearing assumption of the entire framing, and it remains untested.

The paper also concedes that the chosen benchmarks measure shallow commonsense prediction rather than extended reasoning chains, so they are poorly suited to evaluating the architecture's intended strength. The experiments are therefore positioned as an initial study of architectural tradeoffs, not a demonstration of reasoning advantage.

Analysis of adaptive behavior

The strongest empirical content lies in the behavioral analysis. Mean reasoning steps per task reveal both early halting and high variance relative to the 16-step budget:

Benchmark	Mean Steps	Std. Dev.
HellaSwag	3.033	6.263
PIQA	1.866	5.132
WinoGrande	0.954	3.777
TriviaQA	0.804	3.017
Overall	2.681	5.949

Two observations follow. First, average depth well below the maximum indicates frequent early halting, so the recurrent loop does not impose uniform compute cost. Second, standard deviations comparable to or exceeding the means indicate strongly input-dependent policies rather than collapse to a constant depth. The authors appropriately caution that these measurements are not a compute-efficiency benchmark against optimized transformer inference systems.

Qualitative examples reinforce this pattern: simple factual completions ("What is 1 + 2?") use roughly 5–6 steps per token, while generative tasks such as greeting responses reach 15 steps per token. The bat-and-ball cognitive-reflection prompt triggers substantially deeper processing than factual recall, which the authors interpret — carefully — as sensitivity to prompt structure rather than evidence of human-like deliberation.

The appendix complicates the picture. Failure cases show that deeper reasoning does not guarantee correctness: "5 + 5" produced "25" at 10.54 steps per token, and a factual query about India hallucinated population figures at 10.46 steps per token. The appendix summary acknowledges this directly. There is also a tension worth noting: the appendix reports per-token step averages near the 16-step ceiling for trivial prompts like "Hello," which sits uneasily with the main-text narrative of efficient early halting on simple inputs, suggesting the instruction-tuned variant may exhibit different halting statistics than the base model analyzed in Section 5.

Limitations and open questions

Several limitations are conceded explicitly. The negative benchmark results are attributed to a hypothesized scale-dependent tradeoff that is never validated empirically. The evaluation suite measures shallow prediction rather than multi-step reasoning, planning, or mathematical ability — precisely the regimes where adaptive depth should matter most. The internal dynamics of the high- and low-level states remain unexamined; the paper does not establish what the two modules actually learn to represent differently. The relationship between halting decisions and prediction uncertainty or entropy is asserted only implicitly through the observed step distributions. Finally, the discrepancy between main-text halting statistics and appendix per-token statistics across model variants leaves open whether instruction tuning systematically alters halting behavior — a question the paper itself flags.

Conclusion

CosmicFish-HRM demonstrates that hierarchical recurrent reasoning with learned adaptive halting can be embedded in a compact autoregressive LLM and will produce genuinely variable-depth computation conditioned on input characteristics, without manual stopping criteria or reasoning supervision. The cost is concrete: at 82.77M parameters, the architecture trails fixed-depth baselines of similar scale on standard zero-shot benchmarks. The paper's substantive contribution is therefore behavioral rather than performance-oriented, and its central scaling hypothesis — that the HRM overhead amortizes favorably at larger parameter counts — remains the most important open question this work raises.