HGRN2: Hierarchically Gated Linear RNN

• HGRN2 is a recurrent sequence architecture that uses outer-product-based state expansion to boost memory capacity while maintaining hierarchical gating.
• It provides a linear-complexity alternative to transformer models, excelling in low-resource language tasks and efficient long-context processing.
• Variants include a multi-head design for reduced computational cost and a distinct 2D hybrid gene regulatory network interpretation emphasizing non-chaotic dynamics.

Searching arXiv for papers on HGRN2 and closely related work. arXiv search: HGRN2 recurrent LLMs, state expansion, BabyHGRN, FoX, scaling laws. HGRN2 is a recurrent sequence architecture introduced as “Hierarchically Gated Linear RNNs with State Expansion”, a successor to HGRN that preserves hierarchical forget-gating while enlarging recurrent memory through an outer-product-based state expansion without introducing additional parameters (Qin et al., 2024). In subsequent work, it is studied as a linear-time or linear-complexity alternative to transformer-based models and other subquadratic sequence models, and it serves as the recurrent backbone of BabyHGRN for low-resource language modeling (Haller et al., 2024, Shen et al., 2024). A distinct and unrelated usage also appears in hybrid dynamical-systems research, where “HGRN2” denotes the 2-dimensional HGRN case (Wurm et al., 2024).

1. Lineage and formal construction

HGRN2 was introduced to address a stated limitation of HGRN: the recurrent state size of HGRN remains relatively small, limiting its expressiveness (Qin et al., 2024). The original HGRN uses a minimal gated linear recurrent layer

$$\begin{aligned} \mathbf g_t & =\sigma\left(\mathbf U \mathbf x_t+ \mathbf b_u\right), \ \mathbf i_t & = \mathbf V\mathbf x_t+\mathbf b_v, \ \mathbf o_t & =\tau\left(\mathbf W \mathbf x_t+\mathbf b_w\right), \ \mathbf h_t & = \mathbf g_t \odot \mathbf h_{t-1}+\left(1-\mathbf g_t\right) \odot \mathbf i_t, \ \mathbf y_t & =\mathbf h_t \odot \mathbf o_t, \end{aligned}$$

with $\sigma$ as sigmoid, $\tau$ as SiLU, and $\odot$ as elementwise product (Qin et al., 2024).

The defining feature of HGRN is a hierarchical lower bound on forget gates. For layer $i$,

$$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$

where $\beta^i$ is a data-independent lower bound, and the lower bounds are made monotone across layers using

$$\beta := \mathrm{cumax}(\Gamma) = \mathrm{cumsum}(\mathrm{softmax}(\Gamma, \mathrm{dim}=0), \mathrm{dim}=0).$$

This biases lower layers toward more forgetting and short-range modeling, and upper layers toward more retention and long-range modeling (Qin et al., 2024).

HGRN2 retains this hierarchical idea but replaces vector-valued recurrence with a matrix-valued recurrent state: $$\begin{aligned} \mathbf h_t &= \mathrm{Diag}\{\mathbf f_t\} \cdot \mathbf h_{t-1} +(1- \mathbf f_t) \otimes \mathbf i_t \in \mathbb R^{d\times d}, \ \mathbf{y_t} &= \mathbf o_t \cdot \mathbf h_t \in \mathbb R^{1 \times d}. \end{aligned}$$ Here $\mathbf h_t$ is a matrix-valued recurrent state, $\sigma$0 denotes matrix multiplication, and $\sigma$1 denotes outer product (Qin et al., 2024). Relative to HGRN’s vector state of size $\sigma$2, HGRN2 uses a $\sigma$3 state, which the paper describes as an effective expansion from $\sigma$4 to $\sigma$5 scalar memory slots (Qin et al., 2024).

2. State expansion, gating, and linear-attention interpretation

The central innovation is the outer-product-based state expansion mechanism. Rather than enlarging projection matrices such as $\sigma$6, $\sigma$7, and $\sigma$8, HGRN2 expands memory by changing the state representation itself, so the additional capacity comes from the recurrent state rather than from widening the learned linear maps (Qin et al., 2024). The update

$\sigma$9

stores pairwise interactions between the input features and the gating dimensions, and the output

$\tau$0

reads from the expanded state through the output gate (Qin et al., 2024).

The paper explicitly interprets HGRN2 as linear-attention-like or fast-weight-like. Its recurrence resembles the accumulation mechanism used in GLA and RWKV-5/6, and the authors give the correspondence: HGRN2 $\tau$1 matches GLA $\tau$2, $\tau$3 matches $\tau$4, $\tau$5 matches $\tau$6, and $\tau$7 matches the forget gate $\tau$8 (Qin et al., 2024). This interpretation matters computationally because it enables chunkwise or scan-based training and reuse of hardware-efficient training algorithms and optimized CUDA kernels from GLA (Qin et al., 2024).

To reduce the cost of a full $\tau$9 state, HGRN2 introduces a multi-head variant with complexity

$\odot$0

instead of $\odot$1, where $\odot$2 is batch size, $\odot$3 is sequence length, $\odot$4 is model dimension, and $\odot$5 is the number of heads (Qin et al., 2024). The paper reports that increasing the expansion ratio improves perplexity, but the gains saturate beyond about 128 head dimension, and the main experiments therefore use head dimension $\odot$6 (Qin et al., 2024).

In the scaling-law study, HGRN2 is the representative linear RNN with data-dependent decay and is implemented with FLA (Shen et al., 2024). The appendix defines per-head lower-bound gating and the token mixer as

$\odot$7

and

$\odot$8

This formulation makes the decay data-dependent rather than fixed (Shen et al., 2024).

3. Empirical profile in the original HGRN2 study

The original evaluation establishes HGRN2 primarily through ablations on state expansion and through comparisons with HGRN and other efficient sequence models (Qin et al., 2024). In the PESE ablation on 10B tokens, the reported losses are: HGRN1 2.343, low-rank with $\odot$9 2.246, HGRN2 $i$0 2.255, HGRN2 $i$1 2.237, and HGRN2 $i$2 2.195 (Qin et al., 2024). The paper’s interpretation is that state expansion generally improves performance and that HGRN2 with outer-product expansion performs best when the expansion ratio is large (Qin et al., 2024).

A closely related comparison concerns complex versus real recurrence. The reported WikiText-103 test perplexities are 24.82 for Complex HGRN1 with state size $i$3, 26.12 for Real HGRN1 with state size $i$4, and 24.55 for Real HGRN1 with state size $i$5 (Qin et al., 2024). The paper uses this to support the “state size matters” hypothesis rather than attributing the gain to complex recurrence itself (Qin et al., 2024).

On WikiText-103 language modeling, HGRN2 improves over HGRN1 from val 24.14 and test 24.82 to val 23.10 and test 23.73 (Qin et al., 2024). On MQAR, the paper states that HGRN2 significantly outperforms HGRN1 across model sizes and sequence lengths (Qin et al., 2024). On Long Range Arena, the reported average rises from 86.91 for HGRN1 to 87.66 for HGRN2, with individual improvements on ListOps, Text, Retrieval, Image, Pathfinder, and Path-X (Qin et al., 2024). On ImageNet-1k classification, HGRN2 improves DeiT-Tiny from 74.40 to 75.39 and DeiT-Small from 80.09 to 80.12 (Qin et al., 2024).

The large-scale Pile experiment compares 1B and 3B models trained for 30B tokens. The paper concludes that at 1B, HGRN2 is slightly below Mamba, while at 3B, HGRN2 outperforms both LLaMA and Mamba (Qin et al., 2024). In downstream commonsense evaluation after 100B tokens of training, the average scores are 42.15 at 0.15B, 46.47 at 0.39B, 50.84 at 1.0B, and 53.24 at 2.9B, consistently exceeding HGRN1 at matched scales (Qin et al., 2024). This supports the paper’s claim that the architectural gain persists as the model is scaled.

4. BabyHGRN and sample-efficient language modeling

In low-resource language modeling, HGRN2 is the recurrent backbone that powers BabyHGRN, a decoder-only LLM for the BabyLM challenge (Haller et al., 2024). The study emphasizes that HGRN2 is not a “vanilla” LSTM-style recurrent model but a more expressive subquadratic architecture designed to preserve linear-time sequence processing while improving long-range modeling (Haller et al., 2024). Its two defining properties are hierarchical gating and state expansion (Haller et al., 2024).

The first BabyHGRN configuration uses 18 layers and hidden size 2048, totaling about 330M parameters, and the final BabyLM submission is scaled to 1.0B parameters (Haller et al., 2024). The base training objective is standard next-token prediction with token-level cross-entropy,

$i$6

The final model is then trained with knowledge distillation from a teacher HGRN2 model using

$i$7

with

$i$8

Here $i$9 and $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$0 are teacher and student logits, and $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$1 denotes softmax (Haller et al., 2024).

The training regimes are the BabyLM “strict-small” 10M-word track and the “strict” 100M-word track. The corpora are built by subsampling from The Pile rather than using the organizers’ default BabyLM mix; inputs are concatenated and chunked into 512-token sequences, tokenized with a 16k BPE vocabulary; training uses Adam, a linear LR schedule, maximum gradient norm 1.0, and a learning-rate sweep over $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$2; and the implementation uses the FLA library for HGRN2 (Haller et al., 2024).

Under matched parameter budgets of roughly 300–360M, trained for 5 epochs on the 10M-word strict-small corpus, HGRN2 is the best performer overall with a macro average of 57.54, ahead of xLSTM at 56.95, Mamba at 56.74, transformer at 55.99, and LSTM at 54.79 (Haller et al., 2024). The strongest individual result in that table is BLIMP at 67.05 for HGRN2, compared with 62.64 for the transformer baseline (Haller et al., 2024). The paper also notes that all architectures prefer the largest learning rate tried, $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$3 (Haller et al., 2024).

The study probes learning dynamics and reports that BLIMP and EWoK peak early, while BLIMP Supplement peaks later, suggesting that HGRN2 quickly captures some linguistic regularities from limited data but that further training yields only modest improvements (Haller et al., 2024). In the data-composition ablation, the 100M-track HGRN2 model trained on the authors’ Pile-derived corpus reaches 72.89 on BLIMP, 57.43 on BLIMP Supplement, 50.61 on EWoK, and 7.38 on BEAR (Haller et al., 2024). For the final distilled model, BabyHGRN obtains a BabyLM macro average of 63.3 on the 10M track with BEAR 7.5, and 64.9 on the 100M track with BEAR 13.6 (Haller et al., 2024). The distillation-free ablation, BabyHGRNce, is consistently worse; in the 10M setting, distillation improves the macro average by about 5.3 points (Haller et al., 2024).

These results are central to one of the current empirical claims about HGRN2: in low-resource settings, a carefully designed RNN can compete with, and in some cases beat, transformer baselines (Haller et al., 2024). A plausible implication is that HGRN2’s advantages are most evident when sample efficiency is the dominant constraint rather than maximal long-context retrieval.

5. Scaling laws and linear-complexity model comparisons

The scaling-law study evaluates HGRN2 alongside TNL, cosFormer2, and LLaMA across six sizes—70M, 160M, 410M, 1B, 3B, and 7B parameters—trained on a 300B-token bilingual corpus with 1,376 intermediate checkpoints (Shen et al., 2024). The training setup uses Metaseq on PyTorch, H100/H800 80G GPUs, Adam, learning rate $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$4, weight decay 0.1, global batch size 4 million tokens, a tiktoken vocabulary of size 100,280, and context length 8192 (Shen et al., 2024).

In the appendix, HGRN2’s token mixer FLOPs are summarized as

$$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$5

and the total training FLOPs are simplified to

$$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$6

The total parameter count is

$$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$7

which becomes

$$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$8

under $$\begin{aligned} \mathbf f_t^{i} &= {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \ \mathbf h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot \mathbf i_t^i, \end{aligned}$$9 (Shen et al., 2024).

The scaling law fit reported for HGRN2 is

$\beta^i$0

with compute-optimal parameter and data scaling

$\beta^i$1

Among the linear models in that study, HGRN2 has the best fitted compute-loss curve (Shen et al., 2024).

The reported validation perplexities improve steadily with scale. On WIKITEXT-2, HGRN2 records 73.0, 43.8, 27.0, 21.0, 15.6, and 13.8 from 70M to 7B; on LAMBADA, it records 270.1, 52.8, 19.3, 10.9, 6.5, and 5.2 (Shen et al., 2024). For commonsense reasoning, the average CSR scores are 40.44, 42.28, 47.28, 50.92, 55.06, and 58.78 across the same scales (Shen et al., 2024). At 70M, HGRN2 is the only model that surpasses LLaMA on CSR, and at 7B all linear models outperform LLaMA, with HGRN2 essentially tied with TNL, slightly below TNL but above cosFormer2 (Shen et al., 2024).

Retrieval remains more difficult. In NIAH easy mode, HGRN2’s weighted average accuracy rises from 0.2 at 70M to 30.8 at 7B, with intermediate values 0.3, 4.8, 10.0, and 17.9 (Shen et al., 2024). On SCROLLS, the average scores are 7.32, 8.29, 10.93, 11.08, 15.43, and 13.46 (Shen et al., 2024). The paper concludes that HGRN2 is comparable to TNL on CSR and validation perplexity and comparable to cosFormer2 on retrieval and generation tasks, making it the “balanced” linear model in that comparison (Shen et al., 2024).

The same study reports two limitations relevant to HGRN2. First, linear models are more sensitive to aspect ratio changes than vanilla transformers (Shen et al., 2024). Second, for HGRN2, CSR and validation PPL are mostly stable across 2K, 4K, and 8K pretraining context lengths, 16K slightly degrades performance, and retrieval tasks do not necessarily improve with longer pretraining context (Shen et al., 2024).

6. Long-context behavior and the comparison with FoX

The Forgetting Transformer study uses HGRN2 as one of its main recurrent sequence model baselines, alongside Mamba-2 and DeltaNet (Lin et al., 3 Mar 2025). In that framing, HGRN2 is a representative recurrent baseline with linear-time recurrence, but it is used to support the central claim that FoX retains the Transformer’s superior long-context capabilities over recurrent sequence models such as HGRN2 (Lin et al., 3 Mar 2025).

The main long-context language-modeling setup uses LongCrawl64 with 760M non-embedding parameters, 48B tokens, training context length 16,384, and validation context length 65,536 (Lin et al., 3 Mar 2025). The paper states that for recurrent models, including HGRN2, the per-token loss curves flatten around 5k tokens and plateau after 10k tokens (Lin et al., 3 Mar 2025). This is interpreted as a failure to exploit the full context. In absolute loss and perplexity, FoX (Pro) clearly outperforms HGRN2, DeltaNet, and Mamba-2 (Lin et al., 3 Mar 2025).

Needle-in-the-haystack evaluation is one of the clearest negative results for HGRN2 in this comparison. The main paper states that HGRN2 performs even worse than Mamba-2, and the appendix includes a dedicated figure, “Needle-in-the-haystack analysis for HGRN2,” in easy mode and standard mode, scored by GPT-4o on a 1–10 scale (Lin et al., 3 Mar 2025). The qualitative takeaway is unambiguous: HGRN2 performs poorly within training length and is part of the class of recurrent models that fail at this retrieval task (Lin et al., 3 Mar 2025).

On short-context downstream tasks in the LM-eval-harness table, HGRN2 records Wiki. ppl 30.57, LAMBADA ppl 20.14, LAMBADA acc 38.60, PIQA 63.49, HellaSwag 34.94, WinoGrande 51.78, ARC-e 50.13, ARC-c 25.51, COPA 66.00, OBQA 30.00, SciQA 75.60, BoolQ 58.41, and Avg 49.45 (Lin et al., 3 Mar 2025). This is competitive but below FoX (Pro) at 50.88 avg, Transformer (Pro) at 50.39 avg, and Mamba-2 at 50.21 avg (Lin et al., 3 Mar 2025). On LongBench, HGRN2 records NarrativeQA 8.78, Qasper 10.94, MFQA 18.66, HotpotQA 7.78, 2WikiMQA 15.29, MuSiQue 4.32, GovReport 6.13, QMSum 12.19, MultiNews 7.83, TREC 16.5, TriviaQA 14.46, SamSum 6.37, LCC 18.17, and RepoBench-P 16.62 (Lin et al., 3 Mar 2025).

The appendix model-comparison table in the FoX study lists HGRN2 with 756M parameters, $\beta^i$2 forward FLOPs/token, throughput 46k tokens/sec, and learning rate $\beta^i$3 (Lin et al., 3 Mar 2025). This is important for interpretation: HGRN2 is computationally cheaper than FoX or Transformer in that setup, but the efficiency does not translate into better long-context retrieval or better long-context loss (Lin et al., 3 Mar 2025).

A frequent misconception is to treat HGRN2’s strong low-resource results as evidence that it universally supersedes attention-based models. The combined evidence does not support that conclusion. In BabyLM-scale, data-constrained language modeling, HGRN2 can beat transformer baselines (Haller et al., 2024). In long-context retrieval and utilization, the FoX analysis places HGRN2 clearly behind both FoX and Transformer (Lin et al., 3 Mar 2025).

7. Distinct usage in hybrid gene regulatory networks

In a separate literature on hybrid dynamical systems, “HGRN2” denotes the 2-dimensional HGRN case rather than the recurrent neural architecture (Wurm et al., 2024). The underlying object there is a hybrid gene regulatory network $\beta^i$4, with discrete states

$\beta^i$5

and celerity function

$\beta^i$6

A hybrid state is $\beta^i$7, where $\beta^i$8 is the fractional part and $\beta^i$9 is the discrete hypercube index (Wurm et al., 2024).

For the 2D case, the paper emphasizes that dimension strongly changes the behavior. In 2D, boundaries are line segments, intersections are isolated points, and bifurcation requires hitting a point where multiple output boundaries coincide (Wurm et al., 2024). The main HGRN2-specific results are that a two-dimensional HGRN never admits proper chaos and that random HGRNs in two dimensions admit chaos only with probability 0 (Wurm et al., 2024). The broader model class is Turing powerful, so the general reachability problem is undecidable, but the 2D case is too geometrically constrained for proper chaos to occur generically (Wurm et al., 2024).

This distinct usage matters terminologically. In contemporary arXiv work, “HGRN2” can refer either to a hierarchically gated linear RNN with state expansion (Qin et al., 2024) or to the 2-dimensional case of hybrid gene regulatory networks (Wurm et al., 2024). The two meanings are unrelated aside from the shared abbreviation.