Gated Linear Attention (GLA)

Updated 30 June 2025

Gated Linear Attention (GLA) is a neural attention mechanism that adds learnable, data-dependent gating to linear attention for improved scalability and retention of expressivity.
It achieves constant O(k²) computational complexity and reduced memory usage, making it ideal for long-context, large-scale, and real-time applications.
Empirical studies demonstrate that GLA narrows the accuracy gap with softmax attention while powering advanced systems in NLP, vision, diffusion models, and more.

Gated Linear Attention (GLA) refers to a family of neural attention mechanisms that introduce learnable, data-dependent gating into the framework of linear (non-softmax) attention. GLA is designed to address the computational and storage limitations of traditional softmax-based attention—especially in large-scale, long-context, or real-time systems—while recovering much of the lost expressivity through adaptive gating. These mechanisms are now foundational in a wide spectrum of efficient sequence and multimodal architectures.

1. Foundations and Mathematical Formulation

The core of GLA is built upon linear attention, which refactors softmax attention by removing the nonlinear normalization, resulting in operations with linear computational complexity in sequence length. For a sequence with hidden states $H \in \mathbb{R}^{n \times k}$ and a query $q \in \mathbb{R}^{k}$ , standard softmax attention computes: $R(D, Q) = H^T \softmax(H q)$ with $O(nk^2)$ computation per query and $O(nk)$ memory, as all hidden states must be stored.

Linear attention replaces the softmax with a direct inner product, summarizing the context via a fixed-size matrix: $R(D, Q) = H^T H q = C q$ where $C = H^T H$ (a $k \times k$ covariance matrix) is precomputed per document and updated iteratively as: $C_{t+1} = C_t + h_{t+1} h_{t+1}^T$

GLA extends this further by adding learnable, nonlinear gates. The general GLA update is: $C_{t+1} = C_t + [\sigma(W h_{t+1} + b) \odot h_{t+1}] [\sigma(W h_{t+1} + b) \odot h_{t+1}]^T$ where $\sigma$ is a nonlinearity (e.g., sigmoid) and $\odot$ is elementwise multiplication. This adaptive gate modulates the contribution of each new state to the accumulated context, mimicking the information retention mechanisms found in GRUs and LSTMs.

A broader GLA recurrence, prevalent in recent work, is: $S_t = G_t \odot S_{t-1} + k_t v_t^T$ where $G_t$ is a data-dependent gate determining how much prior state is retained.

2. Computational Efficiency and Hardware Considerations

GLA mechanisms offer substantial advantages over softmax and quadratic attention in terms of scalability:

Query time and memory for GLA is $O(k^2)$ , regardless of sequence length $n$ , since $C$ (or its recurrent analogs) is fixed-size.
Storage requirements are dramatically reduced; there is no need to keep all prior activations.
Training and inference benefit from RNN-like recurrence structures, but GLA can also be implemented in parallel using techniques such as chunked and tiled computation (e.g., FlashLinearAttention), maximizing utilization of modern GPU hardware.

A notable implementation, FlashLinearAttention (2312.06635), employs chunkwise processing and optimized memory movement, outperforming even the fastest softmax-based kernels (FlashAttention-2) on both short and long sequences while retaining linear-time complexity.

3. Expressivity, Gating, and Accuracy-Efficiency Tradeoffs

While classic linear attention sacrifices the nonlinearity and normalization of softmax mechanisms, leading to lower baseline accuracy, GLA introduces gating to recover expressivity. Empirical results demonstrate that:

GLA greatly narrows the gap in accuracy with softmax attention.
Gated mechanisms adaptively learn to amplify or suppress contributions from different parts of the sequence, enabling selective retention, dynamic forgetting, and improved focus.
In ablation studies (1609.05866), adding GLA boosts performance far above purely linear attention and approaches softmax-attention accuracy, while retaining constant or linear time/memory scaling.

In practical large-scale deployments (e.g., search, retrieval, multimodal fusion, foundation models), GLA models can handle millions of queries per second and very large input sizes without prohibitive cost.

4. Relationships to and Integration with Recurrent and State-Space Models

GLA is closely related to a broader class of gated linear RNNs and state-space model (SSM) approaches (e.g., Mamba, RWKV, GateLoop, Griffin (2402.19427), ReGLA (2502.01578), Gated Slot Attention (2409.07146)), all of which blend recurrent and attention concepts:

These architectures use multiplicative or bilinear gating to modulate the flow of information. Recent theoretical analysis (2309.01775) shows that GLA and related RNNs can exactly implement linear attention.
A unified implicit attention framework (2405.16504) formulates GLA as a data-controlled, lower-triangular mixing matrix, where diagonal gates provide selective, per-token or per-dimension scaling analogous to attention weightings.

Such equivalence blurs the conceptual boundary between classical attention and modern RNNs, and enables interpretability tools (e.g., attention matrices, attribution rollouts) for GLA models.

5. Gated Linear Attention in Real-World Applications

GLA is now a foundational component in multiple practical domains:

Natural Language Processing and LLMs: GLA is the basis for efficient Transformers and RNN-hybrids in LLMing, machine translation, and retrieval-based tasks, allowing for extreme context lengths and low-latency inference (2312.06635, 2402.19427, 2504.04308).
Vision: In ViG (2405.18425), GLA with hardware-friendly bidirectional and locality-aware gating powers linear-complexity visual backbones, achieving state-of-the-art tradeoffs on ImageNet and COCO, and enabling high-resolution (4096p) vision with 90% less memory compared to ViTs.
Diffusion Models: DiG (2405.18428) applies GLA to 2D diffusion transformers, reducing training cost and memory at high resolution, while matching or exceeding the quality of DiT and Mamba-based models.
Speech and Audio: Lina-Speech (2410.23320) uses GLA blocks in TTS with initial-state tuning for rapid, parameter-efficient voice cloning, allowing robust adaptation even with brief samples.
Sequential Recommendation: RecGRELA (2506.13315) introduces rotary-enhanced GLA with a local gating mechanism, achieving state-of-the-art accuracy in sequential recommendation with lower computation.
Video Object Segmentation: LiVOS (2411.02818) utilizes gated linear matching to replace quadratic space-time memory attention, achieving competitive segmentation at 4096p resolution within standard memory limits.

GLA’s gating principle has also been extended to multimodal fusion (e.g., sensor fusion in adverse-weather object detection (2204.10803)), temporal reasoning (Griffin, (2402.19427)), and finetuning transfer (GSA, (2409.07146)).

6. Recent Advances and Theoretical Insights

Recent work has addressed GLA’s feature map design, normalization, and gate trainability:

Feature Maps: ReGLA (2502.01578) formalizes a normalized exponential feature map, ensuring bounded, non-negative activations and introducing variance reduction scaling for stability.
Normalization: External normalization layers (e.g., LayerNorm, TransNormer) remain essential to decouple variance from sequence length and maintain gradient flow.
Gate Saturation: ReGLA refines sigmoid-based gates with an additional "refining gate", improving gradient propagation near saturation and learning dynamics.

Theoretical studies (2504.04308) demonstrate that GLA can implement weighted preconditioned gradient descent (WPGD) learning rules via gating. In multitask or heterogeneous in-context learning, GLA is provably more expressive than vanilla linear attention, optimally weighting context examples through its gating mechanism.

7. Limitations, Tradeoffs, and Ongoing Directions

GLA mechanisms, while highly efficient, generally underperform softmax attention in unconstrained expressivity, especially for complex, non-monotonic tasks. Hybrid architectures (e.g., combining GLA with local/global softmax attention or SSMs) are a promising direction and have reached or surpassed softmax performance in some configurations. The field is actively investigating:

Hierarchical/log-linear GLA variants (2506.04761) that use logarithmically growing hidden state sets (Fenwick tree-style) for multi-scale memory and improved recall.
Further refinements to feature mappings, gating architectures (e.g., direction-wise and locality-aware gating), and explainability extensions.
Post-linearization or transformer-to-RNN finetuning (2409.07146) by leveraging GLA-compatible slots/gates, often with softmax readouts for better transfer.

Mechanism	Complexity (lookup)	Memory	Expressivity	Hardware Fit	Key Limitation
Softmax Attention	$O(nk^2)$	$O(nk)$	Highest	Less (quadratic)	Scalability
Linear Attention	$O(k^2)$	$O(k^2)$	Lower (no data-dependent)	Excellent	Expressivity
Gated Linear Attention	$O(k^2)$	$O(k^2)$	High (adaptive gates)	Excellent	Slightly less than softmax

GLA mechanisms provide constant-time, fixed-size, and memory-efficient attention with learned selectivity, bridging the gap between scalability and expressiveness for modern sequence and multimodal architectures. They have become a central tool in high-throughput, long-context, and hardware-constrained neural models, and ongoing research continues to refine their feature, gating, and normalization designs for robust deployment in increasingly demanding applications.