Gated Recurrent Linearization

Updated 23 March 2026

Gated Recurrent Linearization is a sequence modeling paradigm that uses data-dependent gating in linear recurrent architectures to dynamically control long-range dependencies.
It integrates classical RNN properties with linear attention and Transformer-style encoders by employing per-step gates for selective retention and efficient inference.
GRL variants like GateLoop and HGRN2 achieve state-of-the-art performance with linear-time, constant-space complexity, making them ideal for large-scale, context-rich applications.

Gated Recurrent Linearization (GRL) is a foundational paradigm in modern sequence modeling, defined by inserting data-dependent gating functions into linear recurrent architectures to control the accumulation, weighting, and transformation of long-range context. This approach yields highly expressive and efficient models that bridge classical RNNs, linear attention, and Transformer-style sequence encoders. GRL forms the mathematical and architectural backbone of numerous state-of-the-art models enabling linear-time, constant-space inference while directly generalizing or subsuming structures such as S4, S5, LRU, RetNet, and Gated Linear Attention (GLA). The core principle is to introduce per-step gates that perform content-selective retention or forgetting by modulating recurrent state evolution, often interpreted as data-controlled products in the recurrence, and to provide explicit control over relative position encodings, attention weighting, and hierarchical memory scales (Katsch, 2023, Li et al., 6 Apr 2025, Zucchet et al., 2023, Qin et al., 2023, Qin et al., 2024).

1. Formal Model Definitions and Recurrence Structures

At the heart of Gated Recurrent Linearization is the fully data-controlled recurrence, exemplified by GateLoop and related frameworks. The prototypical recurrence is given by: $h_n = a_n \odot h_{n-1} + k_n^\top v_n,\quad y_n = q_n h_n$ where:

$a_n\in\mathbb{C}^{d_h}$ is the data-dependent gate, often parametrized as $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ with $f$ and $g$ controlling magnitude and phase.
$k_n, v_n, q_n$ are linearly projected input–key–value–output gates.

Unrolling yields: $y_n = q_n \sum_{m=1}^n \left(k_m^\top v_m\right) \prod_{j=m+1}^n a_j$ Each past timestep contributes by an input-content term ( $k_m^\top v_m$ ), scaled by the (potentially complex-valued) product of intervening gates—thereby granting dynamic, data-driven control to every state transition.

This generalizes a wide class of gated RNNs:

Recurrent Additive Networks (RANs): Purely additive, with gating entered solely via elementwise sigmoids, yielding a closed-form weighted sum of past inputs (Lee et al., 2017).
HGRN/HGRN2: Explicit forget gates with hierarchical, learnable lower bounds, expanding states via outer products for greater expressivity and linear attention equivalence (Qin et al., 2023, Qin et al., 2024).
Liger, GLA, and GateLoop: Incorporate gating directly within the key-projection or update mechanisms to effect content-based retention, hybridized with attention-like or sliding-window softmax for increased modeling power (Katsch, 2023, Lan et al., 3 Mar 2025, Li et al., 6 Apr 2025).

2. Algorithmic Implementations and Computational Complexity

GRL models unify both recurrent and parallel computation modes:

O( $\ell$ ) recurrent step: Implements the full recurrence in a standard forward scan, using O(1) auxiliary memory per step. Example pseudocode: $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ 4
O( $\ell \log \ell$ ) parallel mode: Uses associative-scan primitives with the binary operator

$a_n\in\mathbb{C}^{d_h}$ 0

operating on $a_n\in\mathbb{C}^{d_h}$ 1 tuples; enables fast throughput on GPUs via frameworks such as JAX or Triton (Katsch, 2023, Qin et al., 2024).

Surrogate attention O( $a_n\in\mathbb{C}^{d_h}$ 2): Offers a direct matrix-view formulation matching masked attention structures for explicit relative position encoding.

This structure admits efficient inference over extremely long contexts, avoids quadratic space scaling, and, in models like HGRN2, further leverages matrix- or head-wise state expansion to grow expressive capacity without parameter inflation (Qin et al., 2024, Katsch, 2023).

Mode	Time Complexity	Space Complexity	Typical Implementation
Recurrent	O( $a_n\in\mathbb{C}^{d_h}$ 3)	O(1)	For-loop scan
Parallel	O( $a_n\in\mathbb{C}^{d_h}$ 4)	O( $a_n\in\mathbb{C}^{d_h}$ 5)	Associative scan (e.g., `lax.associative_scan`)
Surrogate Attention	O( $a_n\in\mathbb{C}^{d_h}$ 6)	O( $a_n\in\mathbb{C}^{d_h}$ 7)	Batched matmul with mask

3. Theoretical Insights: Gating as Weighting and Relative Positionality

Fundamentally, GRL transforms the state update from a fixed linear dynamic to a flexible, content-adaptive one. Theoretical results establish:

Context-aware weighting: Gating induces data-dependent weights over the history, precisely controlling each step's contribution to the output. In Gated Linear Attention, this is directly linked to the implementation of Weighted Preconditioned Gradient Descent (WPGD), where gates $a_n\in\mathbb{C}^{d_h}$ 8 generate the weights $a_n\in\mathbb{C}^{d_h}$ 9 for context mixing (Li et al., 6 Apr 2025).
Data-controlled relative position encoding: In GateLoop and HGRN2, the product of gates over time explicitly encodes a learned, content-driven relative positional bias, surpassing fixed or hand-engineered schemes (Katsch, 2023, Qin et al., 2024).
Linear attention equivalence and universality: Modern GRL architectures can exactly reproduce (causally-masked) linear self-attention by choosing the gating and feedforward weights to construct the appropriate key–value accumulators and output mappings. Gradient descent is observed to reliably “discover” these parameterizations in practice (Zucchet et al., 2023).

When vanilla linear attention (i.e., no gating) is compared to GRL, the latter strictly improves modeling power whenever contextual weighting (e.g., for multitask or in-context learning) deviates from uniformity (Li et al., 6 Apr 2025).

4. Architectural Variants and Extensions

The GRL framework supports a spectrum of architectures:

GateLoop: Uses fully data-controlled, complex-valued gating for both magnitude and phase, with output and input gating, and supports both recurrent and scan-based parallel operation. The surrogate attention interpretation connects it to attention mechanisms with explicit learned relative-position kernels (Katsch, 2023).
HGRN and HGRN2: Impose monotonic lower bounds on forget gates across layers to achieve hierarchically multi-timescale memory. Outer product–based state expansion (HGRN2) aligns the recurrence with the mechanics of linear attention while keeping parameter count and compute efficient (Qin et al., 2023, Qin et al., 2024).
Liger: Converts transformer LLMs to gated-linear RNNs by recycling key-projection weights for gating, applies pooling+sigid to form update gates, supplements with hybrid attention (GRM + sliding window softmax), and employs LoRA for fast adaptation (Lan et al., 3 Mar 2025).
Addition/ReLU Gated RNNs: Replace multiplicative and sigmoid gating with addition and ReLU, preserving linearized memory dynamics while reducing hardware and cryptographic inference cost. This variant is especially effective for quantized or encrypted settings (Brännvall et al., 2023).
Multiplicative Gates and Outer-Product Memory: The use of multiplicative gating (as in $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ 0) enables construction of quadratic features and outer-product memories required for attention-like behavior in RNNs (Zucchet et al., 2023).

5. Empirical Performance and Applications

Extensive empirical results show that models based on GRL achieve or exceed the performance of contemporary transformers and large RNNs:

GateLoop: On WikiText-103, achieves a test perplexity 13.4 with 125M parameters, outperforming softmax Transformer's 18.6. In synthetic memory-span benchmarks, data-controlled products enable longer memory retention (Katsch, 2023).
HGRN2: Delivers PPL $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ 1 23.73 (44M params) on WikiText-103, outperforming baseline HGRN1 and matching or exceeding LLaMa2 and Mamba at large scale (Qin et al., 2024).
Liger: Recovers 93–98% of original Transformer LLM performance with only 0.02B LoRA fine-tune tokens, and provides O(T) inference time with constant memory—suitable for deployment at tens of thousands token context (Lan et al., 3 Mar 2025).
Addition-based gates: Achieve comparable (<0.3pp difference) accuracy to standard GRUs and LSTMs on MNIST sequence tasks, with substantial (2–3x) speed-ups in CPU and encrypted environments (Brännvall et al., 2023).

GRL architectures are deployed in language modeling, in-context learning, vision, and privacy-preserving contexts, serving as drop-in replacements for memory-intensive Transformer layers, and facilitating efficient on-device or large-context inference.

6. Connections, Theoretical Guarantees, and Future Directions

GRL reveals a deep mathematical connection between RNN-type gating and attention-style mechanisms:

Equivalence to Linear Attention: Gated recurrent networks with properly configured gates and updates can represent the full linear self-attention mechanism, with gradient descent experimentally shown to learn such equivalences (Zucchet et al., 2023).
Optimization-theoretic guarantees: In the multitask prompt setting, multilayer GLA models provably converge to a unique optimal Weighted Preconditioned GD solution, with explicit characterizations of when scalar versus vector gating is sufficient for optimality (Li et al., 6 Apr 2025).
Interpretability: Explicit formulas for memory, attention weights, and recurrence unrolling yield interpretable, decomposable attributions for model predictions.

Open research directions include investigation of non-diagonal or block-diagonal gating, richer phase/magnitude parameterizations, integration of non-linear attention atop GRL kernels, and the analysis of emergent patterns in learned gating sequences for interpretability and linguistic insight (Katsch, 2023, Qin et al., 2024).

7. Limitations and Application-Specific Trade-offs

Not all gating mechanisms preserve full model capacity. Simpler, addition-only gates offer efficiency but may incur a slight performance penalty. Multiplicative gating is essential for capturing higher-order dependencies and implementing attention-equivalent operations; attention mechanisms without gating cannot recover the context-dependent weighting vital for multitask or in-context learning scenarios (Zucchet et al., 2023, Li et al., 6 Apr 2025).

Applying GRL to encrypted or quantized inference confers substantial fiscal or privacy-preserving advantages by reducing non-linear and multiplicative operation counts, though current benchmarks primarily concern CPU and cryptographic settings. Scaling state expansion (as in HGRN2) trades off model capacity and memory footprint, but O( $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ 2) per-token costs can remain practical for moderate $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ 3 and head count (Qin et al., 2024, Brännvall et al., 2023).