---
title: 'HGRN2: Hierarchically Gated Linear RNN'
url: https://www.emergentmind.com/topics/hgrn2
type: topic
---

# HGRN2: Hierarchically Gated Linear RNN

Searching arXiv for recent papers on HGRN2 and closely related work.
arXiv search: HGRN2 recurrent language models, state expansion, BabyHGRN, FoX, scaling laws.
HGRN2 is a recurrent sequence architecture introduced as **“Hierarchically Gated Linear RNNs with State Expansion”**, a successor to HGRN that preserves hierarchical forget-gating while enlarging recurrent memory through an outer-product-based state expansion without introducing additional parameters [2404.07904]. In subsequent work, it is studied as a linear-time or linear-complexity alternative to transformer-based models and other subquadratic sequence models, and it serves as the recurrent backbone of BabyHGRN for low-resource language modeling [2412.15978][2406.16690]. A distinct and unrelated usage also appears in hybrid dynamical-systems research, where “HGRN2” denotes the 2-dimensional HGRN case [2404.16197].

## 1. Lineage and formal construction

HGRN2 was introduced to address a stated limitation of HGRN: the recurrent state size of HGRN remains relatively small, limiting its expressiveness [2404.07904]. The original HGRN uses a minimal gated linear recurrent layer
\[
\begin{aligned}
\mathbf g_t & =\sigma\left(\mathbf U \mathbf x_t+ \mathbf b_u\right), \\
\mathbf i_t & = \mathbf V\mathbf x_t+\mathbf b_v, \\
\mathbf o_t & =\tau\left(\mathbf W \mathbf x_t+\mathbf b_w\right), \\
\mathbf h_t & = \mathbf g_t \odot \mathbf h_{t-1}+\left(1-\mathbf g_t\right) \odot \mathbf i_t, \\
\mathbf y_t & =\mathbf h_t \odot \mathbf  o_t,
\end{aligned}
\]
with \(\sigma\) as sigmoid, \(\tau\) as SiLU, and \(\odot\) as elementwise product [2404.07904].

The defining feature of HGRN is a hierarchical lower bound on forget gates. For layer \(i\),
\[
\begin{aligned}
\mathbf f_t^{i} &=  {\beta}^{i} + (1- {\beta}^i) \odot \mathbf g_t^i, \\
\mathbf  h_t^{i} &= \mathbf f_t^i \odot \mathbf h_{t-1}^i + (1 - \mathbf f_t^i) \odot   \mathbf i_t^i,
\end{aligned}
\]
where \(\beta^i\) is a data-independent lower bound, and the lower bounds are made monotone across layers using
\[
\beta := \mathrm{cumax}(\Gamma) = \mathrm{cumsum}(\mathrm{softmax}(\Gamma, \mathrm{dim}=0), \mathrm{dim}=0).
\]
This biases lower layers toward more forgetting and short-range modeling, and upper layers toward more retention and long-range modeling [2404.07904].

HGRN2 retains this hierarchical idea but replaces vector-valued recurrence with a matrix-valued recurrent state:
\[
\begin{aligned}
\mathbf h_t  &= \mathrm{Diag}\{\mathbf f_t\}  \cdot \mathbf h_{t-1} +(1- \mathbf f_t) \otimes  \mathbf  i_t \in \mathbb R^{d\times d}, \\
\mathbf{y_t} &= \mathbf o_t \cdot \mathbf h_t \in \mathbb R^{1 \times d}.
\end{aligned}
\]
Here \(\mathbf h_t\) is a matrix-valued recurrent state, \(\cdot\) denotes matrix multiplication, and \(\otimes\) denotes outer product [2404.07904]. Relative to HGRN’s vector state of size \(d\), HGRN2 uses a \(d \times d\) state, which the paper describes as an effective expansion from \(d\) to \(d^2\) scalar memory slots [2404.07904].

## 2. State expansion, gating, and linear-attention interpretation

The central innovation is the outer-product-based state expansion mechanism. Rather than enlarging projection matrices such as \(\mathbf U\), \(\mathbf V\), and \(\mathbf W\), HGRN2 expands memory by changing the state representation itself, so the additional capacity comes from the recurrent state rather than from widening the learned linear maps [2404.07904]. The update
\[
(1-\mathbf f_t)\otimes \mathbf i_t
\]
stores pairwise interactions between the input features and the gating dimensions, and the output
\[
\mathbf y_t = \mathbf o_t \cdot \mathbf h_t
\]
reads from the expanded state through the output gate [2404.07904].

The paper explicitly interprets HGRN2 as linear-attention-like or fast-weight-like. Its recurrence resembles the accumulation mechanism used in GLA and RWKV-5/6, and the authors give the correspondence: HGRN2 \(\mathbf o\) matches GLA \(\mathbf q\), \(1-\mathbf f\) matches \(\mathbf k\), \(\mathbf i\) matches \(\mathbf v\), and \(\mathbf f\) matches the forget gate \(\boldsymbol{\alpha}\) [2404.07904]. This interpretation matters computationally because it enables chunkwise or scan-based training and reuse of hardware-efficient training algorithms and optimized CUDA kernels from GLA [2404.07904].

To reduce the cost of a full \(d \times d\) state, HGRN2 introduces a multi-head variant with complexity
\[
O(B N d^2 / H),
\]
instead of \(O(B N d^2)\), where \(B\) is batch size, \(N\) is sequence length, \(d\) is model dimension, and \(H\) is the number of heads [2404.07904]. The paper reports that increasing the expansion ratio improves perplexity, but the gains saturate beyond about 128 head dimension, and the main experiments therefore use head dimension \(d_h = 128\) [2404.07904].

In the scaling-law study, HGRN2 is the representative linear RNN with data-dependent decay and is implemented with FLA [2406.16690]. The appendix defines per-head lower-bound gating and the token mixer as
\[
\mathbf{\overline{LR}_i= \mathrm{Softmax}(\mathbf{LR}_i,\mathrm{dim}=0), \ \mathbf{Lr}_i^{(s)} =\mathrm{Cumsum}(\mathbf{\overline{LR}_i,\mathrm{dim}=0)[s], \ \mathbf {LR}_i \in \mathbb R^{L\times d/h},i=1,\ldots, h.
\end{aligned}
\]
and
\[
\begin{aligned}
&\mathbf {\bar X}^{(s)}= \mathrm{Norm}(\mathbf X^{(s)}), \\
&\mathbf {Og}^{(s)}_i,\mathbf {Fg}^{(s)}_i,\mathbf H^{(s)}_i= \mathbf  {\bar X}^{(s)} \mathbf W_{og_i}^{(s)}, \mathbf  {\bar X}^{(s)} \mathbf W_{fg_i}^{(s)}, \mathbf  {\bar X}^{(s)} \mathbf W_{h_i}^{(s)}, \\
&\mathbf {Fg}^{(s)}_i=\mathbf{Lr}_i^{(s)}+(1-\mathbf{Lr}_i^{(s)})(\mathrm{Sigmoid}( \mathbf {Fg}^{(s)}_i)), \\
&\mathbf { O}^{(s)}_i=\mathrm{FLA}\left(\mathbf {Og}^{(s)}_i, {\mathbf {Fg}^{(s)}_i}, \mathbf H^{(s)}_i,1-{\mathbf {Fg}^{(s)}_i}\right), \\
&\mathbf O^{(s)}=\mathrm{Norm}\left( \mathrm{Concat}[\mathbf { O}^{(s)}_1,\ldots, \mathbf { O}^{(s)}_h] \right)  + \mathbf X^{(s)}.
\end{aligned}
\]
This formulation makes the decay data-dependent rather than fixed [2406.16690].

## 3. Empirical profile in the original HGRN2 study

The original evaluation establishes HGRN2 primarily through ablations on state expansion and through comparisons with HGRN and other efficient sequence models [2404.07904]. In the PESE ablation on 10B tokens, the reported losses are: HGRN1 2.343, low-rank with \(n=4\) 2.246, HGRN2 \(n=4\) 2.255, HGRN2 \(n=8\) 2.237, and HGRN2 \(n=128\) 2.195 [2404.07904]. The paper’s interpretation is that state expansion generally improves performance and that HGRN2 with outer-product expansion performs best when the expansion ratio is large [2404.07904].

A closely related comparison concerns complex versus real recurrence. The reported WikiText-103 test perplexities are 24.82 for Complex HGRN1 with state size \(2d\), 26.12 for Real HGRN1 with state size \(d\), and 24.55 for Real HGRN1 with state size \(2d\) [2404.07904]. The paper uses this to support the “state size matters” hypothesis rather than attributing the gain to complex recurrence itself [2404.07904].

On WikiText-103 language modeling, HGRN2 improves over HGRN1 from val 24.14 and test 24.82 to val 23.10 and test 23.73 [2404.07904]. On MQAR, the paper states that HGRN2 significantly outperforms HGRN1 across model sizes and sequence lengths [2404.07904]. On Long Range Arena, the reported average rises from 86.91 for HGRN1 to 87.66 for HGRN2, with individual improvements on ListOps, Text, Retrieval, Image, Pathfinder, and Path-X [2404.07904]. On ImageNet-1k classification, HGRN2 improves DeiT-Tiny from 74.40 to 75.39 and DeiT-Small from 80.09 to 80.12 [2404.07904].

The large-scale Pile experiment compares 1B and 3B models trained for 30B tokens. The paper concludes that at 1B, HGRN2 is slightly below Mamba, while at 3B, HGRN2 outperforms both LLaMA and Mamba [2404.07904]. In downstream commonsense evaluation after 100B tokens of training, the average scores are 42.15 at 0.15B, 46.47 at 0.39B, 50.84 at 1.0B, and 53.24 at 2.9B, consistently exceeding HGRN1 at matched scales [2404.07904]. This supports the paper’s claim that the architectural gain persists as the model is scaled.

## 4. BabyHGRN and sample-efficient language modeling

In low-resource language modeling, HGRN2 is the recurrent backbone that powers BabyHGRN, a decoder-only language model for the BabyLM challenge [2412.15978]. The study emphasizes that HGRN2 is not a “vanilla” LSTM-style recurrent model but a more expressive subquadratic architecture designed to preserve linear-time sequence processing while improving long-range modeling [2412.15978]. Its two defining properties are hierarchical gating and state expansion [2412.15978].

The first BabyHGRN configuration uses 18 layers and hidden size 2048, totaling about 330M parameters, and the final BabyLM submission is scaled to 1.0B parameters [2412.15978]. The base training objective is standard next-token prediction with token-level cross-entropy,
\[
\mathcal{L}_{CE} = -\frac{1}{N}\sum_{i=1}^{N}\log P(x_i \mid x_1,\dots,x_{i-1};\theta).
\]
The final model is then trained with knowledge distillation from a teacher HGRN2 model using
\[
L_{\text{total} = (1-\alpha)L_{CE} + \alpha L_{KD},
\]
with
\[
L_{KD} = KL(\sigma(z_t)\,\|\,\sigma(z_s)).
\]
Here \(z_t\) and \(z_s\) are teacher and student logits, and \(\sigma\) denotes softmax [2412.15978].

The training regimes are the BabyLM “strict-small” 10M-word track and the “strict” 100M-word track. The corpora are built by subsampling from The Pile rather than using the organizers’ default BabyLM mix; inputs are concatenated and chunked into 512-token sequences, tokenized with a 16k BPE vocabulary; training uses Adam, a linear LR schedule, maximum gradient norm 1.0, and a learning-rate sweep over \(\{10^{-3},10^{-4},10^{-5},10^{-6}\}\); and the implementation uses the FLA library for HGRN2 [2412.15978].

Under matched parameter budgets of roughly 300–360M, trained for 5 epochs on the 10M-word strict-small corpus, HGRN2 is the best performer overall with a macro average of 57.54, ahead of xLSTM at 56.95, Mamba at 56.74, transformer at 55.99, and LSTM at 54.79 [2412.15978]. The strongest individual result in that table is BLIMP at 67.05 for HGRN2, compared with 62.64 for the transformer baseline [2412.15978]. The paper also notes that all architectures prefer the largest learning rate tried, \(10^{-3}\) [2412.15978].

The study probes learning dynamics and reports that BLIMP and EWoK peak early, while BLIMP Supplement peaks later, suggesting that HGRN2 quickly captures some linguistic regularities from limited data but that further training yields only modest improvements [2412.15978]. In the data-composition ablation, the 100M-track HGRN2 model trained on the authors’ Pile-derived corpus reaches 72.89 on BLIMP, 57.43 on BLIMP Supplement, 50.61 on EWoK, and 7.38 on BEAR [2412.15978]. For the final distilled model, BabyHGRN obtains a BabyLM macro average of 63.3 on the 10M track with BEAR 7.5, and 64.9 on the 100M track with BEAR 13.6 [2412.15978]. The distillation-free ablation, BabyHGRNce, is consistently worse; in the 10M setting, distillation improves the macro average by about 5.3 points [2412.15978].

These results are central to one of the current empirical claims about HGRN2: in low-resource settings, a carefully designed RNN can compete with, and in some cases beat, transformer baselines [2412.15978]. A plausible implication is that HGRN2’s advantages are most evident when sample efficiency is the dominant constraint rather than maximal long-context retrieval.

## 5. Scaling laws and linear-complexity model comparisons

The scaling-law study evaluates HGRN2 alongside TNL, cosFormer2, and LLaMA across six sizes—70M, 160M, 410M, 1B, 3B, and 7B parameters—trained on a 300B-token bilingual corpus with 1,376 intermediate checkpoints [2406.16690]. The training setup uses Metaseq on PyTorch, H100/H800 80G GPUs, Adam, learning rate \(3\mathrm{e}{-4}\), weight decay 0.1, global batch size 4 million tokens, a tiktoken vocabulary of size 100,280, and context length 8192 [2406.16690].

In the appendix, HGRN2’s token mixer FLOPs are summarized as
\[
C = 8nd^2 + \frac{4nd^2}{h} + \frac{nd^2}{Bh} + 4nBd + nBh+5nd,
\]
and the total training FLOPs are simplified to
\[
\mathrm{FLOPs} = bl\left(72nd^2 + \frac{24nd^2}{h}+29nd \right) +12ndv+12dl.
\]
The total parameter count is
\[
4ld^2+3ldg+dv+ld,
\]
which becomes
\[
12ld^2 + dv + ld
\]
under \(g=8/3d\) [2406.16690].

The scaling law fit reported for HGRN2 is
\[
L(C) = 3.4788C^{-0.0753},
\]
with compute-optimal parameter and data scaling
\[
N_{opt}(C) = (2.66\times 10^8)C^{0.6427},
\qquad
D_{opt}(C) = (4.80\times 10^{10})C^{0.4500}.
\]
Among the linear models in that study, HGRN2 has the best fitted compute-loss curve [2406.16690].

The reported validation perplexities improve steadily with scale. On WIKITEXT-2, HGRN2 records 73.0, 43.8, 27.0, 21.0, 15.6, and 13.8 from 70M to 7B; on LAMBADA, it records 270.1, 52.8, 19.3, 10.9, 6.5, and 5.2 [2406.16690]. For commonsense reasoning, the average CSR scores are 40.44, 42.28, 47.28, 50.92, 55.06, and 58.78 across the same scales [2406.16690]. At 70M, HGRN2 is the only model that surpasses LLaMA on CSR, and at 7B all linear models outperform LLaMA, with HGRN2 essentially tied with TNL, slightly below TNL but above cosFormer2 [2406.16690].

Retrieval remains more difficult. In NIAH easy mode, HGRN2’s weighted average accuracy rises from 0.2 at 70M to 30.8 at 7B, with intermediate values 0.3, 4.8, 10.0, and 17.9 [2406.16690]. On SCROLLS, the average scores are 7.32, 8.29, 10.93, 11.08, 15.43, and 13.46 [2406.16690]. The paper concludes that HGRN2 is comparable to TNL on CSR and validation perplexity and comparable to cosFormer2 on retrieval and generation tasks, making it the “balanced” linear model in that comparison [2406.16690].

The same study reports two limitations relevant to HGRN2. First, linear models are more sensitive to aspect ratio changes than vanilla transformers [2406.16690]. Second, for HGRN2, CSR and validation PPL are mostly stable across 2K, 4K, and 8K pretraining context lengths, 16K slightly degrades performance, and retrieval tasks do not necessarily improve with longer pretraining context [2406.16690].

## 6. Long-context behavior and the comparison with FoX

The Forgetting Transformer study uses HGRN2 as one of its main recurrent sequence model baselines, alongside Mamba-2 and DeltaNet [2503.02130]. In that framing, HGRN2 is a representative recurrent baseline with linear-time recurrence, but it is used to support the central claim that FoX retains the Transformer’s superior long-context capabilities over recurrent sequence models such as HGRN2 [2503.02130].

The main long-context language-modeling setup uses LongCrawl64 with 760M non-embedding parameters, 48B tokens, training context length 16,384, and validation context length 65,536 [2503.02130]. The paper states that for recurrent models, including HGRN2, the per-token loss curves flatten around 5k tokens and plateau after 10k tokens [2503.02130]. This is interpreted as a failure to exploit the full context. In absolute loss and perplexity, FoX (Pro) clearly outperforms HGRN2, DeltaNet, and Mamba-2 [2503.02130].

Needle-in-the-haystack evaluation is one of the clearest negative results for HGRN2 in this comparison. The main paper states that HGRN2 performs even worse than Mamba-2, and the appendix includes a dedicated figure, “Needle-in-the-haystack analysis for HGRN2,” in easy mode and standard mode, scored by GPT-4o on a 1–10 scale [2503.02130]. The qualitative takeaway is unambiguous: HGRN2 performs poorly within training length and is part of the class of recurrent models that fail at this retrieval task [2503.02130].

On short-context downstream tasks in the LM-eval-harness table, HGRN2 records Wiki. ppl 30.57, LAMBADA ppl 20.14, LAMBADA acc 38.60, PIQA 63.49, HellaSwag 34.94, WinoGrande 51.78, ARC-e 50.13, ARC-c 25.51, COPA 66.00, OBQA 30.00, SciQA 75.60, BoolQ 58.41, and Avg 49.45 [2503.02130]. This is competitive but below FoX (Pro) at 50.88 avg, Transformer (Pro) at 50.39 avg, and Mamba-2 at 50.21 avg [2503.02130]. On LongBench, HGRN2 records NarrativeQA 8.78, Qasper 10.94, MFQA 18.66, HotpotQA 7.78, 2WikiMQA 15.29, MuSiQue 4.32, GovReport 6.13, QMSum 12.19, MultiNews 7.83, TREC 16.5, TriviaQA 14.46, SamSum 6.37, LCC 18.17, and RepoBench-P 16.62 [2503.02130].

The appendix model-comparison table in the FoX study lists HGRN2 with 756M parameters, \(1.54 \times 10^9\) forward FLOPs/token, throughput 46k tokens/sec, and learning rate \(2 \times 10^{-3}\) [2503.02130]. This is important for interpretation: HGRN2 is computationally cheaper than FoX or Transformer in that setup, but the efficiency does not translate into better long-context retrieval or better long-context loss [2503.02130].

A frequent misconception is to treat HGRN2’s strong low-resource results as evidence that it universally supersedes attention-based models. The combined evidence does not support that conclusion. In BabyLM-scale, data-constrained language modeling, HGRN2 can beat transformer baselines [2412.15978]. In long-context retrieval and utilization, the FoX analysis places HGRN2 clearly behind both FoX and Transformer [2503.02130].

## 7. Distinct usage in hybrid gene regulatory networks

In a separate literature on hybrid dynamical systems, “HGRN2” denotes the **2-dimensional HGRN** case rather than the recurrent neural architecture [2404.16197]. The underlying object there is a hybrid gene regulatory network \(\mathcal{H}=(E,c)\), with discrete states
\[
E := \left\{ d \in \mathbb{N}^N \mid \forall i\in\{1,\dots,N\}: 0\le d_i\le n_i\right\},
\]
and celerity function
\[
c:E\to \mathbb{R}^N.
\]
A hybrid state is \(h=(\pi,d)\), where \(\pi \in [0,1]^N\) is the fractional part and \(d\in E\) is the discrete hypercube index [2404.16197].

For the 2D case, the paper emphasizes that dimension strongly changes the behavior. In 2D, boundaries are line segments, intersections are isolated points, and bifurcation requires hitting a point where multiple output boundaries coincide [2404.16197]. The main HGRN2-specific results are that a two-dimensional HGRN never admits proper chaos and that random HGRNs in two dimensions admit chaos only with probability 0 [2404.16197]. The broader model class is Turing powerful, so the general reachability problem is undecidable, but the 2D case is too geometrically constrained for proper chaos to occur generically [2404.16197].

This distinct usage matters terminologically. In contemporary arXiv work, “HGRN2” can refer either to a hierarchically gated linear RNN with state expansion [2404.07904] or to the 2-dimensional case of hybrid gene regulatory networks [2404.16197]. The two meanings are unrelated aside from the shared abbreviation.

Source: https://www.emergentmind.com/topics/hgrn2