---
title: 'Gated Linear RNNs: Efficient Sequence Models'
url: https://www.emergentmind.com/topics/gated-linear-rnns
type: topic
---

# Gated Linear RNNs: Efficient Sequence Models

Gated Linear Recurrent Neural Networks (Gated Linear RNNs, GLRNNs) are a class of sequence models that combine linear recurrences with data-dependent gating, yielding architectures with highly efficient training and inference, robust extrapolation, and a strong connection to various forms of (linear) self-attention. Recent advances have unified a spectrum of models under the GLRNN regime, including HGRN, HGRN2, RG-LRU, Griffin, Gated Slot Attention, and others. These architectures have demonstrated competitive or superior performance to Transformers on multiple sequence modeling tasks, while offering hardware efficiency, explainability, and theoretical tractability.

## 1. Core Principles and Model Formulations

The central design of Gated Linear RNNs comprises two components: a (linear or affine) diagonal recurrence and data-dependent element-wise gates. A canonical formulation is:
\[
h_t = f_t \odot h_{t-1} + i_t \odot x_t,
\]
where $f_t$ and $i_t$ are "forget" and "input" gates (typically with $f_t, i_t \in (0,1)^d$ parameterized via sigmoid nonlinearities), and $x_t$ is the input at time $t$. Some variants exploit pure addition with ReLU gating instead of standard sigmoids and multiplications [2308.05629].

The gates control the flow of information and dynamic memory retention. Notably, the states can often be shown to be convex, data-dependent, per-component mixtures of past inputs, giving the recurrences equivalence to a form of implicit attention [1705.07393, 2309.01775, 2405.16504].

Modern GLRNNs generalize this basic structure by:

- Including complex-valued recurrences and fixed relative-position rotations (e.g., HGRN) [2311.04823],
- Employing bilinear gates and output projections to realize exact linear self-attention layers [2309.01775],
- Expanding hidden state via structured or outer-product growth for enhanced expressivity (e.g., HGRN2) [2404.07904],
- Integrating gating into linear attention (GLA) or slot-based architectures to improve recall and memory efficiency [2409.07146].

In all cases, the architectures are designed for $O(Td)$ inference and training per layer per sequence of length $T$ with hidden size $d$, and support efficient parallelization.

## 2. Hierarchically Gated RNNs and State Expansion

The Hierarchically Gated Recurrent Neural Network (HGRN) introduced a layered gating hierarchy, wherein each layer $k$ enforces a monotonic, learnable lower bound $\gamma^k$ on the data-dependent forget gate:
\[
\lambda_t = \gamma^k + (1 - \gamma^k)\odot \tilde{\mu}_t, \quad \tilde{\mu}_t = \sigma(x_t W_{\mu} + b_{\mu})
\]
The lower bound $\gamma^k$ is computed as a cumulative sum over a softmax across layers, enforcing that lower layers emphasize local/short-term dependencies, while higher layers capture longer-term context [2311.04823].

HGRN2 extends this by state expansion using an outer product mechanism:
\[
H_t = \mathrm{Diag}(f_t) H_{t-1} + (1-f_t) \otimes i_t, \quad y_t = o_t^\top H_t
\]
The hidden state becomes matrix-valued (up to $d^2$), drastically increasing capacity with negligible additional parameters. HGRN2 thereby matches the expressiveness of linear attention mechanisms, while supporting hardware-efficient training using blockwise matrix operations [2404.07904].

## 3. Connection to Attention and Unified Implicit Causal Self-Attention

GLRNNs are mathematically equivalent to implicit, data-dependent causal self-attention layers, as shown by the mapping:
\[
y = W_g(x)\;\Alpha(x)\;Z(x)\;M\,x
\]
where $W_g(x)$ is a diagonal gating matrix, $\Alpha(x)$ is a lower-triangular, data-dependent attention matrix parameterized via the recurrence, $Z(x)$ is an activation branch, and $M$ is a causal filter (e.g., Conv1D) [2405.16504].

This framework encompasses Mamba, RWKV, Griffin (RG-LRU), GateLoop, HGRN2, and related models, unifying them as sub-quadratic, attention-like operators. It has led to explainability methods for GLRNNs analogous to Transformer-based attention visualization, attribution, and rollout procedures, with empirical evidence that these provide faithful and sharp relevance maps in both vision and language contexts [2405.16504].

Gated Linear RNNs can exactly implement (linear) self-attention by orchestrating their gating, state, and output projections to replicate the "key–value–query" structure of attention. Gradient descent converges on this solution in practice, as observed empirically [2309.01775].

## 4. Architectural Spectrum and Variants

The GLRNN design space encompasses:

- **Purely additive networks** (RANs): $h_t = f_t \odot h_{t-1} + i_t \odot x_t$ without nonlinear recurrences; performance on par with LSTM for language modeling, at lower parameter and compute cost [1705.07393].
- **Bilinear and multiplicative gates:** Used in attention-equivalent GLRNNs to manufacture exact key, value, and query representations [2309.01775].
- **Addition-based Gated RNNs:** Replace sigmoid and multiplication with ReLU and addition, providing substantial computational savings in plaintext and homomorphic encryption settings, while preserving long-term memory and sequence learning power [2308.05629].
- **RG-LRU (Griffin):** Employs a recurrence gate and input gate; integrates into hybrid architectures with local attention, achieving hardware-efficient training, excellent extrapolation, and high accuracy at scale [2402.19427].
- **Slot attention and bounded-memory extensions:** Gated Slot Attention (GSA) manages a matrix-valued slot state and realizes a two-pass Gated Linear Attention, linking via softmax to stabilize gradients and enhance recall [2409.07146].

## 5. Empirical Performance and Scaling Properties

GLRNNs achieve highly competitive results on a wide range of benchmarks, often equaling or outperforming efficient Transformer derivatives and prior RNNs:

| Model         | Language Modeling (WT-103, PPL) | Long-Range Arena (Avg. Acc) | ImageNet-1k (Top-1%) |
|---------------|---------------------------------|-----------------------------|----------------------|
| Transformer   | 24.40 (val) / 24.78 (test)      | –                           | 72.20 (DeiT-Tiny)    |
| HGRN          | 24.14 / 24.82                   | 86.91                       | 74.40 (Tiny)         |
| HGRN2         | 23.10 / 23.73                   | 87.66                       | 75.39 (Tiny)         |
| Griffin-14B   | –                               | –                           | –                    |

In addition to maintaining state-of-the-art accuracy, GLRNNs exhibit:

- Linear time and memory scaling ($O(Td)$) in sequence length, compared to quadratic scaling of full self-attention,
- Superior extrapolation: stable performance at much longer sequence lengths than observed during training,
- Hardware efficiency: high throughput and low latency at inference due to fixed-state size and diagonal (or block-diagonal) recurrences,
- Minimal to no regularization requirements; training with Adam or AdamW, warmup schedules, and weight decay is typically sufficient [2311.04823, 2404.07904, 2402.19427].

## 6. Explainability and Theoretical Interpretations

GLRNNs allow for direct extraction of implicit attention matrices from the structure of their recurrences. The unified theory provides:

- Explicit attention-weight visualization for attributions,
- Attention rollout and propagation methods,
- Analysability of long-range dependencies via the gate-product unrolling [2405.16504],
- Weighted-sum interpretation of state as data-dependent mixing of past inputs, supporting interpretability [1705.07393].

The implicit self-attention structure bridges architectural and theoretical gaps between RNNs and Transformers, facilitating method transfer and automated analysis tools.

## 7. Extensions, Limitations, and Future Directions

Contemporary models extend GLRNNs with:

- Expanded state/memory via outer products, multi-head splitting, or slot-based storage [2404.07904, 2409.07146],
- Hybridization with local/global attentional blocks (e.g., Griffin),
- Efficient quantization/hardware specialization, especially in addition-based variants.

Challenges remain in further improving recall-heavy task performance, dynamic capacity allocation, integration of adaptive local attention, and architecture-agnostic explainability.

Ongoing empirical work, particularly on Gated Slot Attention and large-scale hybrids, continues to push the performance, scaling, and interpretability boundaries of GLRNNs across modality and benchmark suites [2409.07146, 2402.19427]. The consensus is that Gated Linear RNNs offer a highly efficient, theoretically principled, and practically effective alternative to both classical RNNs and the Transformer family.

Source: https://www.emergentmind.com/topics/gated-linear-rnns