---
title: Gated Recurrent Linearization
url: https://www.emergentmind.com/topics/gated-recurrent-linearization
type: topic
---

# Gated Recurrent Linearization

Gated Recurrent Linearization (GRL) is a foundational paradigm in modern sequence modeling, defined by inserting data-dependent gating functions into linear recurrent architectures to control the accumulation, weighting, and transformation of long-range context. This approach yields highly expressive and efficient models that bridge classical RNNs, linear attention, and Transformer-style sequence encoders. GRL forms the mathematical and architectural backbone of numerous state-of-the-art models enabling linear-time, constant-space inference while directly generalizing or subsuming structures such as S4, S5, LRU, RetNet, and Gated Linear Attention (GLA). The core principle is to introduce per-step gates that perform content-selective retention or forgetting by modulating recurrent state evolution, often interpreted as data-controlled products in the recurrence, and to provide explicit control over relative position encodings, attention weighting, and hierarchical memory scales [2311.01927][2504.04308][2309.01775][2311.04823][2404.07904].

## 1. Formal Model Definitions and Recurrence Structures

At the heart of Gated Recurrent Linearization is the fully data-controlled recurrence, exemplified by GateLoop and related frameworks. The prototypical recurrence is given by:
\[
h_n = a_n \odot h_{n-1} + k_n^\top v_n,\quad y_n = q_n h_n
\]
where:
- $a_n\in\mathbb{C}^{d_h}$ is the data-dependent gate, often parametrized as $\alpha(x_n)=f(\mathrm{Linear}_\gamma(x_n)) \exp(i\,g(\mathrm{Linear}_\theta(x_n)))$ with $f$ and $g$ controlling magnitude and phase.
- $k_n, v_n, q_n$ are linearly projected input–key–value–output gates.

Unrolling yields:
\[
y_n = q_n \sum_{m=1}^n \left(k_m^\top v_m\right) \prod_{j=m+1}^n a_j
\]
Each past timestep contributes by an input-content term ($k_m^\top v_m$), scaled by the (potentially complex-valued) product of intervening gates—thereby granting dynamic, data-driven control to every state transition.

This generalizes a wide class of gated RNNs:
- Recurrent Additive Networks (RANs): Purely additive, with gating entered solely via elementwise sigmoids, yielding a closed-form weighted sum of past inputs [1705.07393].
- HGRN/HGRN2: Explicit forget gates with hierarchical, learnable lower bounds, expanding states via outer products for greater expressivity and linear attention equivalence [2311.04823][2404.07904].
- Liger, GLA, and GateLoop: Incorporate gating directly within the key-projection or update mechanisms to effect content-based retention, hybridized with attention-like or sliding-window softmax for increased modeling power [2311.01927][2503.01496][2504.04308].

## 2. Algorithmic Implementations and Computational Complexity

GRL models unify both recurrent and parallel computation modes:
- **O($\ell$) recurrent step**: Implements the full recurrence in a standard forward scan, using O(1) auxiliary memory per step. Example pseudocode:
  ```python
  h = 0
  for n in 1..l:
      a_n, kv_n = compute_gates(x_n)
      h = h * a_n + kv_n
      y_n = q_n @ h
  ```
- **O($\ell \log \ell$) parallel mode**: Uses associative-scan primitives with the binary operator
  \[
  (p_1, p_2) \bullet (q_1, q_2) = (p_1q_1, q_1p_2 + q_2)
  \]
  operating on $(a_n, k_n^\top v_n)$ tuples; enables fast throughput on GPUs via frameworks such as JAX or Triton [2311.01927][2404.07904].
- **Surrogate attention O($\ell^2$)**: Offers a direct matrix-view formulation matching masked attention structures for explicit relative position encoding.

This structure admits efficient inference over extremely long contexts, avoids quadratic space scaling, and, in models like HGRN2, further leverages matrix- or head-wise state expansion to grow expressive capacity without parameter inflation [2404.07904][2311.01927].

| Mode         | Time Complexity | Space Complexity | Typical Implementation                        |
|--------------|----------------|------------------|-----------------------------------------------|
| Recurrent    | O($\ell$)      | O(1)             | For-loop scan                                |
| Parallel     | O($\ell\log\ell$)| O($\ell$)      | Associative scan (e.g., `lax.associative_scan`)|
| Surrogate Attention | O($\ell^2$)| O($\ell^2$)    | Batched matmul with mask                      |

## 3. Theoretical Insights: Gating as Weighting and Relative Positionality

Fundamentally, GRL transforms the state update from a fixed linear dynamic to a flexible, content-adaptive one. Theoretical results establish:
- **Context-aware weighting**: Gating induces data-dependent weights over the history, precisely controlling each step's contribution to the output. In Gated Linear Attention, this is directly linked to the implementation of Weighted Preconditioned Gradient Descent (WPGD), where gates $\{G_t\}$ generate the weights $\omega_i = \prod_{t=i+1}^{n+1}G_t$ for context mixing [2504.04308].
- **Data-controlled relative position encoding**: In GateLoop and HGRN2, the product of gates over time explicitly encodes a learned, content-driven relative positional bias, surpassing fixed or hand-engineered schemes [2311.01927][2404.07904].
- **Linear attention equivalence and universality**: Modern GRL architectures can exactly reproduce (causally-masked) linear self-attention by choosing the gating and feedforward weights to construct the appropriate key–value accumulators and output mappings. Gradient descent is observed to reliably “discover” these parameterizations in practice [2309.01775].

When vanilla linear attention (i.e., no gating) is compared to GRL, the latter strictly improves modeling power whenever contextual weighting (e.g., for multitask or in-context learning) deviates from uniformity [2504.04308].

## 4. Architectural Variants and Extensions

The GRL framework supports a spectrum of architectures:
- **GateLoop**: Uses fully data-controlled, complex-valued gating for both magnitude and phase, with output and input gating, and supports both recurrent and scan-based parallel operation. The surrogate attention interpretation connects it to attention mechanisms with explicit learned relative-position kernels [2311.01927].
- **HGRN and HGRN2**: Impose monotonic lower bounds on forget gates across layers to achieve hierarchically multi-timescale memory. Outer product–based state expansion (HGRN2) aligns the recurrence with the mechanics of linear attention while keeping parameter count and compute efficient [2311.04823][2404.07904].
- **Liger**: Converts transformer LLMs to gated-linear RNNs by recycling key-projection weights for gating, applies pooling+sigid to form update gates, supplements with hybrid attention (GRM + sliding window softmax), and employs LoRA for fast adaptation [2503.01496].
- **Addition/ReLU Gated RNNs**: Replace multiplicative and sigmoid gating with addition and ReLU, preserving linearized memory dynamics while reducing hardware and cryptographic inference cost. This variant is especially effective for quantized or encrypted settings [2308.05629].
- **Multiplicative Gates and Outer-Product Memory**: The use of multiplicative gating (as in $g(x)=(W_mx)\odot(W_xx)$) enables construction of quadratic features and outer-product memories required for attention-like behavior in RNNs [2309.01775].

## 5. Empirical Performance and Applications

Extensive empirical results show that models based on GRL achieve or exceed the performance of contemporary transformers and large RNNs:
- **GateLoop**: On WikiText-103, achieves a test perplexity 13.4 with 125M parameters, outperforming softmax Transformer's 18.6. In synthetic memory-span benchmarks, data-controlled products enable longer memory retention [2311.01927].
- **HGRN2**: Delivers PPL$_\text{test}$ 23.73 (44M params) on WikiText-103, outperforming baseline HGRN1 and matching or exceeding LLaMa2 and Mamba at large scale [2404.07904].
- **Liger**: Recovers 93–98% of original Transformer LLM performance with only 0.02B LoRA fine-tune tokens, and provides O(T) inference time with constant memory—suitable for deployment at tens of thousands token context [2503.01496].
- **Addition-based gates**: Achieve comparable (<0.3pp difference) accuracy to standard GRUs and LSTMs on MNIST sequence tasks, with substantial (2–3x) speed-ups in CPU and encrypted environments [2308.05629].

GRL architectures are deployed in language modeling, in-context learning, vision, and privacy-preserving contexts, serving as drop-in replacements for memory-intensive Transformer layers, and facilitating efficient on-device or large-context inference.

## 6. Connections, Theoretical Guarantees, and Future Directions

GRL reveals a deep mathematical connection between RNN-type gating and attention-style mechanisms:
- **Equivalence to Linear Attention**: Gated recurrent networks with properly configured gates and updates can represent the full linear self-attention mechanism, with gradient descent experimentally shown to learn such equivalences [2309.01775].
- **Optimization-theoretic guarantees**: In the multitask prompt setting, multilayer GLA models provably converge to a unique optimal Weighted Preconditioned GD solution, with explicit characterizations of when scalar versus vector gating is sufficient for optimality [2504.04308].
- **Interpretability**: Explicit formulas for memory, attention weights, and recurrence unrolling yield interpretable, decomposable attributions for model predictions.

Open research directions include investigation of non-diagonal or block-diagonal gating, richer phase/magnitude parameterizations, integration of non-linear attention atop GRL kernels, and the analysis of emergent patterns in learned gating sequences for interpretability and linguistic insight [2311.01927][2404.07904].

## 7. Limitations and Application-Specific Trade-offs

Not all gating mechanisms preserve full model capacity. Simpler, addition-only gates offer efficiency but may incur a slight performance penalty. Multiplicative gating is essential for capturing higher-order dependencies and implementing attention-equivalent operations; attention mechanisms without gating cannot recover the context-dependent weighting vital for multitask or in-context learning scenarios [2309.01775][2504.04308].

Applying GRL to encrypted or quantized inference confers substantial fiscal or privacy-preserving advantages by reducing non-linear and multiplicative operation counts, though current benchmarks primarily concern CPU and cryptographic settings. Scaling state expansion (as in HGRN2) trades off model capacity and memory footprint, but O($d^2/H$) per-token costs can remain practical for moderate $d$ and head count [2404.07904][2308.05629].

## References
- GateLoop: [2311.01927]
- Liger: [2503.01496]
- HGRN/HGRN2: [2311.04823], [2404.07904]
- ReLU/Add gates: [2308.05629]
- Gated Linear Attention and WPGD: [2504.04308]
- Recurrent Additive Networks: [1705.07393]
- Gated RNN attention equivalence: [2309.01775]

Source: https://www.emergentmind.com/topics/gated-recurrent-linearization