HRM-Text: Hierarchical Recurrent Language Model

• The paper introduces a multi-timescale HRM-Text architecture that decouples context integration and token generation to achieve competitive performance with significantly less data and compute.
• It employs MagicNorm stabilization and a warmup deep credit assignment to maintain gradient stability during deep recurrent unrolling.
• Benchmark results show HRM-Text achieving strong performance on MMLU, ARC-C, DROP, GSM8K, and MATH using only 40B tokens and approximately $1,500 of compute.

HRM-Text refers to a Hierarchical Recurrent Model (HRM) for language modeling that radically diverges from the prevailing transformer-based approach to large-scale pretraining. It introduces an architecture with explicit multi-timescale recurrence and a task-completion-centric pretraining objective, motivated by computational, sample-efficiency, and neurobiological considerations. HRM-Text demonstrates that a 1B-parameter model can be trained from scratch on as little as 40B unique tokens and approximately $1,500 of compute—orders of magnitude less than the regime required by current transformers—while achieving benchmark performance competitive with open 2–7B parameter models (Wang et al., 20 May 2026).

1. Biological and Computational Motivation

HRM-Text is motivated by the inefficiency of modern transformer pretraining, which requires exaFLOPs-scale compute and trillions of text tokens for next-token prediction. Much of this data is wasted on reconstructing irrelevant or repetitive content. By contrast, biological cognition, notably the frontoparietal loop, utilizes distinct circuits for slow, strategic planning and fast, execution-level operations, suggesting a multi-timescale architectural prior for sample-efficient learning. HRM-Text leverages this analogy to decouple context integration and stepwise token generation within a LLM, hypothesizing that such architectural co-design can drastically shift the compute/performance frontier (Wang et al., 20 May 2026).

2. HRM-Text Architecture

The model consists of two recurrent modules operating at different timescales:

• High-level (H) Module: Integrates semantic context and slow-evolving information, akin to planning circuits in cortex.
• Low-level (L) Module: Executes rapid, token-level updating, maintaining local self-attention and feedforward refinements.

The explicit recurrence is structured as follows. Denote $z_H^{(i)} \in \mathbb{R}^d$ as the state of the H module after the $i$-th cycle, and $z_L^{(i, t)}$ as the L-state at step $t$ within the $i$-th cycle:

• For each strategic cycle $i$:
  • For $t=1, ..., N_L$, update $$z_L^{(i,t)} = \mathrm{MagNorm}(z_L^{(i,t-1)} + \mathrm{Sublayer}_L(\mathrm{Norm}(z_L^{(i,t-1)}), z_H^{(i-1)}))$$
  • After $N_L$ executions, update $$z_H^{(i)} = \mathrm{MagNorm}(z_H^{(i-1)} + \mathrm{Sublayer}_H(\mathrm{Norm}(z_H^{(i-1)}), z_L^{(i,N_L)}))$$

The process proceeds for $i$0 top-level cycles. The final $i$1 is fed to a linear output head to generate logits. RoPE and standard attention/MLP sublayers are used in both H and L modules. This dual-timescale architecture decouples context and execution, capturing dependencies over longer contexts with fewer parameters and less data than standard transformers (Wang et al., 20 May 2026).

3. Stabilization: MagicNorm and Training Credit Assignment

Deep recurrence in HRM-Text presents challenges to gradient stability analogous to pre/post-norm placement in deep transformers. To mitigate exploding/vanishing gradients during recurrent unrolling, HRM-Text introduces MagicNorm: $i$2 This top-level normalization constrains the variance of the hidden state at each step (as in PostNorm), while internal PreNorm layers provide favorable backward signal paths during truncated BPTT.

In parallel, the truncated BPTT horizon is gradually increased during pretraining (“warmup deep credit assignment”): $i$3 Initially, only short-range credit is assigned, with longer dependencies becoming available as training proceeds, following evidence from developmental and dopaminergic learning in animal neurobiology (Wang et al., 20 May 2026).

4. Pretraining Objective: Task Completion via PrefixLM Masking

Departing from generic next-token prediction, HRM-Text is trained specifically on instruction–response pairs $i$4. The loss is: $i$5 A PrefixLM masking pattern is employed: instruction (“prefix”) tokens attend bidirectionally, while response (“completion”) tokens attend only leftward (causally). This compresses the encoder-decoder paradigm into a single stack and directly aligns training with conditional generation (the standard downstream use case for LMs), thus improving sample efficiency (Wang et al., 20 May 2026).

5. Training Setup and Benchmark Results

The reported model instantiates the HRM architecture with $i$6 hidden size, 16 layers each for H and L modules, and other standard architectural features (RoPE with $i$7, context $i$8, $i$91B parameters). Training uses only $z_L^{(i, t)}$040B unique tokens sampled from instruction-centric datasets. Full train details include Adam-atan2 optimization, batch of 196 k tokens, no clipping, bfloat16, with FSDP over 16 H100 GPUs for 736 GPU-hours (roughly $z_L^{(i, t)}$1 FLOPs, $z_L^{(i, t)}$2$1,500 cost).

HRM-Text 1B achieves:

• MMLU: 60.7%
• ARC-C: 81.9%
• DROP: 82.2%
• GSM8K: 84.5%
• MATH: 56.2%

This matches or exceeds open-source 2–7B models trained on up to 100–900x more data and 96–432x more compute, demonstrating the viability of efficient pretraining through architectural and objective co-design (Wang et al., 20 May 2026).

6. Comparative Performance and Design Implications

Ablation studies with matched compute and training tokens indicate that HRM-Text’s explicit two-timescale recurrence consistently outperforms standard, looped, and shared-parameter transformer architectures by 3–7 points across multiple reasoning and QA benchmarks. Notably, these gains arise despite orders-of-magnitude less pretraining data, underscoring the efficacy of the HRM architecture and instruction-centric loss.

A plausible implication is that co-designing architecture and training objective—especially with explicit recurrence and focus on conditional completion—can break the existing Pareto frontier on data/computational efficiency in LLM pretraining, making from-scratch pretraining feasible for smaller institutions (Wang et al., 20 May 2026).

7. Research Outlook and Future Directions

HRM-Text provides an existence proof for compact, task-focused LLM pretraining leveraging hierarchical recurrence. Extension of this paradigm suggests a research program of separating a “reasoning core” (compact, recurrent, sample-efficient) from large-scale retrieval and memory systems—mirroring the anatomical multitimescale separation observed in biological brains. Open questions include scaling this paradigm, combining it with retrieval-augmented memory, and broader architectural/optimization generalizations (Wang et al., 20 May 2026).