HRM-Text: Efficient Pretraining Beyond Scaling

Abstract: The current pretraining paradigm for LLMs relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2-7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community.

• The paper introduces a dual-timescale hierarchical recurrent model that interleaves a fast execution module and a slow strategic module to achieve deep computation efficiently.
• It employs MagicNorm and a warmup deep credit assignment strategy to stabilize training and mitigate gradient spikes during long-horizon backpropagation.
• The HRM-Text model, trained with limited compute and data, attains competitive reasoning benchmark scores, challenging the necessity of brute-force scaling.

HRM-Text: Efficient Pretraining via Hierarchical Recurrent Models

Motivation and Problem Setting

HRM-Text proposes a stark departure from prevailing LLM pretraining approaches, which rely on massive compute and internet-scale raw text. The paper targets the inefficiency and exclusivity of brute-force scaling and develops a model that achieves open-domain performance on standard benchmarks while utilizing drastically less compute and fewer training tokens. The core premise is that architectural and objective co-design, inspired by biological multi-timescale processing, can break conventional compute–performance tradeoffs.

Model Architecture and Stabilization

HRM-Text implements a dual-timescale Hierarchical Recurrent Model (HRM), with cycles that alternate between a fast-evolving execution module ($L$) and a slow-evolving strategic module ($H$). Each high-level $H$ update is encompassed by several low-level $L$ updates, producing deep internal computation without the need for parameter scaling in proportion to depth. To stabilize optimization under deep recurrence, HRM-Text introduces MagicNorm—a hybrid normalization strategy that balances forward activation bounding and direct gradient flow by leveraging the asymmetry of truncated backpropagation through time (TBPTT). Additional architectural features include parameterless RMSNorm, SwiGLU activations, RoPE positional encoding, and sigmoid-gated multi-head self-attention.

Figure 1: HRM-Text's dual-timescale recurrent architecture, module internals, attention, and PrefixLM mask.

Warmup deep credit assignment further enhances training stability: the optimizer’s backward-unroll length increases gradually during early training, mitigating early-stage instabilities characteristic of full-range BPTT in deeply recurrent models.

Objective Redesign: Task-Completion and PrefixLM

HRM-Text rejects standard autoregressive next-token prediction on raw text, instead pretraining exclusively on instruction–response datasets. The training objective is the conditional NLL loss over the response segment, $-\log P(x_a|x_q)$, thereby allocating modeling capacity directly to the critical generation phase. The model employs PrefixLM attention masking: instruction tokens attend bidirectionally, while response tokens conform to a causal mask. This paradigm synergistically improves prompt utilization and response modeling efficiency.

Figure 2: Comparison of causal vs. response-only vs. PrefixLM objectives, showing lower NLL and more global attention under PrefixLM.

Layerwise attention entropy analysis demonstrates that PrefixLM induces broader context usage, and attention maps confirm fundamental differences between standard and PrefixLM masking. Both the response-only objective and PrefixLM yield substantial improvements under data- and compute-limited pretraining.

Sample-Efficient Pretraining and Results

A 1B-parameter HRM-Text model, trained from scratch on 40B unique instruction–response tokens with $1,500 USD$ compute, achieves competitive performance with open LLMs in the 2–7B parameter range on multiple reasoning and factual benchmarks. Strong numerical results include: 60.7% MMLU, 81.9% ARC-C, 82.2% DROP, 84.5% GSM8K, 56.2% MATH.

Figure 3: HRM-Text 1B achieves results competitive with much larger models with up to $432\times$ less compute and $900\times$ fewer tokens.

Compared to FLOPs- and parameter-matched transformers, looped transformers, and RINS models, HRM-Text consistently outperforms all alternatives on key reasoning tasks. Crucially, the model’s deep recurrence does not lead to representational collapse or over-smoothing. Effective depth analysis, using both norm differences and blockwise cosine similarity across layers, shows that HRM-Text leverages significantly deeper computation than transformer baselines.

Figure 4: Effective depth analysis: HRM hidden states change consistently, avoiding the representational smoothing seen in other models.

Per-layer logit lens KL divergence demonstrates that HRM-Text’s prediction distributions remain adaptable much deeper into the network compared to standard architectures, underlining the utility of recurrent depth.

Figure 5: Logit lens analysis—HRM maintains higher KL in deep layers, indicating persistent computation compared to transformers.

Gradient Dynamics and Stabilization

Extensive analysis of training dynamics reveals that long-horizon BPTT can induce rare but massive gradient spikes, consistent with products-of-Jacobians producing heavy-tailed, lognormal-like instabilities. By combining truncated, gradually-warmed-up BPTT and MagicNorm, HRM-Text stabilizes the gradient distribution, suppresses extreme events, and maintains useful gradient flow throughout optimization.

Figure 6: Mean absolute gradient magnitudes; full BPTT shows rare large spikes absent in truncated training.

Figure 7: Mechanistic evidence; deeper BPTT increases Jacobian growth and outlier gradient events.

Figure 8: Median absolute gradient magnitude across recurrent runs—HRM provides stable, even training dynamics.

Practical and Theoretical Implications

The empirical evidence demonstrates that compute–performance scaling laws are not fundamental: architectural priors and suitable objectives can yield order-of-magnitude improvements in pretraining efficiency. The conditional instruction–response paradigm, paired with bidirectional prompt attention, further streamlines training in the realistic regime where LLMs are applied to specific prompt–response settings.

Theoretically, HRM-Text opens avenues for decoupling reasoning computation from factual knowledge density. The architecture’s ability to perform deep, multi-step reasoning with modest parameter and data budgets points to future combination with retrieval-augmented or memory-based approaches, where the HRM backbone is dedicated to computation and planning, while factual recall is handled externally.

Additionally, HRM-Text’s recurrent design is naturally compatible with adaptive computation time (ACT) and test-time guidance strategies. Preliminary experiments suggest that interpolation or extrapolation between shallower and deeper recurrent outputs at inference time yields further performance gains, indicating dynamic test-time control as a promising direction.

Future Directions

• Separation of reasoning and factual modules: Leveraging HRM-style models for compute-efficient reasoning, coupled with external knowledge sources, may yield robust small models with strong generalization.
• Adaptive computation schedules: Incorporating ACT could provide competitive performance with further reduction in inference-time FLOPs.
• Scaling and transferability: Application of HRM architecture at larger parameter scales and under broader data regimes, combined with retrieval or structured reasoning modules.

Conclusion

HRM-Text provides a rigorous existence proof that highly efficient LLM pretraining is achievable through judicious architecture–objective co-design. By demonstrating competitive results with two orders-of-magnitude less compute and data, the work reopens foundational LM research to the broader community and challenges the inevitability of brute-force scaling. The approach highlights the importance of structure, conditionality, and stabilized recurrent computation for the next generation of efficient LLMs.

For in-depth experiment details, methodology, and comparative analyses, see "HRM-Text: Efficient Pretraining Beyond Scaling" (2605.20613).