Linear Attention Models Overview

• Linear attention models are linear-complexity architectures that replace quadratic softmax attention with kernel factorizations and recurrent state updates.
• They compress past context into fixed-size or bounded recurrent memory, enhancing training speed and decoding efficiency.
• Hybrid and hierarchical extensions mitigate expressivity limits by blending full and linear attention to improve long-range recall.

Linear attention models are a family of linear-complexity sequence architectures that replace the quadratic token-token interaction of causal softmax attention with kernel factorizations, recurrent state updates, or closely related fixed-state sequence-mixing mechanisms. Across recent work, this family is often written either in associative form, $\phi(Q)\phi(K)^\top V=\phi(Q)(\phi(K)^\top V)$, or in recurrent form, with a compact state such as $S_t=S_{t-1}+k_t v_t^\top$ and output $o_t=q_t S_t$; several papers further unify linear Transformers, state space models, and linear RNNs under this broader linear-attention view (You et al., 2024, Chou et al., 2024, Guo et al., 5 Jun 2025).

1. Core formulation and computational rationale

Standard causal softmax attention is repeatedly presented as the baseline from which linear attention departs. In the notation used across the literature, it takes forms such as

$\operatorname{softmax}(QK^\top+M)V$

or

$$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$

with quadratic dependence on sequence length due to explicit or implicit construction of all query-key interactions (Guo et al., 5 Jun 2025, Liu et al., 29 Mar 2026). This quadratic compute is paired, in several discussions, with KV-cache growth during autoregressive decoding and substantial runtime contribution from attention at long sequence lengths (You et al., 2024, Arora et al., 2024).

Linear attention replaces the softmax interaction by a linearly separable kernel or equivalent structured recurrence. Representative formulations include

$O^\phi=\phi(Q)\phi(K)^T V,$

$$\mathrm{Attn}(Q,K,V)\approx \phi(Q)\big(\phi(K)^\top V\big),$$

and the normalized causal recurrence

$$S_t=\sum_{j=1}^t k'_j v_j^\top,\qquad z_t=\sum_{j=1}^t k'_j,\qquad o_t=\frac{q'_t S_t}{q'_t\cdot z_t+\varepsilon},$$

which avoid constructing an $N\times N$ attention matrix and thereby reduce sequence-length scaling from quadratic to linear or effectively linear in common regimes (You et al., 2024, Goel et al., 19 Feb 2026, Arora et al., 2024). In several papers this associative reordering is also expressed through a compact matrix state, for example

$$\mathbf{kv}_s=\lambda \mathbf{kv}_{s-1}+\mathbf{k}_s\mathbf{v}_s^\top,\qquad \mathbf{o}_s^\top=\mathbf{q}_s^\top \mathbf{kv}_s,$$

or, without decay, $S_t=S_{t-1}+k_t v_t^\top$0, $S_t=S_{t-1}+k_t v_t^\top$1 (Sun et al., 2024, Wang et al., 9 Jun 2026).

The primary computational motivation is therefore consistent: linear attention seeks $S_t=S_{t-1}+k_t v_t^\top$2-style sequence scaling, fixed-size or bounded recurrent memory, and hardware-favorable matrix multiplication patterns. Several works stress that this matters not only for training, but also for decoding throughput, KV-cache pressure, and long-context deployment (You et al., 2024, Goldstein et al., 5 May 2025, Team et al., 30 Oct 2025).

2. Recurrent state, memory compression, and theoretical interpretations

A central property of linear attention models is that they compress the past into a hidden state. Recent surveys and model analyses explicitly identify this as both the source of their efficiency and the source of their limitations: past key-value information is merged into a fixed-size recurrent state, making decoding constant-memory or bounded-memory, but constraining recall and long-range retrieval (Guo et al., 5 Jun 2025, Deng et al., 3 Feb 2026, Wang et al., 8 Jul 2025).

This recurrent perspective has enabled several theoretical reinterpretations. In time-series forecasting, a single linear attention layer is shown to admit a dynamic VAR form,

$S_t=S_{t-1}+k_t v_t^\top$3

so that the keys act as lagged observations and the rank-1 coefficient matrices act as time-varying autoregressive weights (Lu et al., 11 Feb 2025). The same work argues that ordinary multi-layer Transformer plumbing is structurally misaligned with autoregressive forecasting objectives, and proposes SAMoVAR to rearrange MLP, attention, and input-output flow so that stacked linear attention remains interpretable as a mixture of dynamic VAR processes (Lu et al., 11 Feb 2025).

In in-context learning, linear attention has been studied as an algorithmic learner rather than only an efficiency device. For multi-head linear self-attention trained on in-context linear regression, the training dynamics depend sharply on parametrization: with merged key-query weights, gradient flow has two fixed-point manifolds and a single abrupt loss drop; with separate key and query matrices, the dynamics exhibit exponentially many fixed points and saddle-to-saddle transitions corresponding to progressively learned principal components (Zhang et al., 27 Jan 2025). A complementary empirical study on the canonical linear-regression benchmark reports that both quadratic and linear attention can learn in-context regression, that single-layer models are weak, and that at 6 layers linear attention with $S_t=S_{t-1}+k_t v_t^\top$4 reaches test loss $S_t=S_{t-1}+k_t v_t^\top$5 versus $S_t=S_{t-1}+k_t v_t^\top$6 for quadratic attention, while converging faster in samples seen (Goel et al., 19 Feb 2026).

These results suggest that linear attention should not be understood solely as a crude softmax approximation. In some domains it functions as a recurrent estimator with explicit algorithmic structure, dynamic lag weighting, or progressively acquired regression behavior (Lu et al., 11 Feb 2025, Zhang et al., 27 Jan 2025).

3. Expressivity limits and softmax-approximation strategies

The dominant critique of linear attention is that the fixed-state bottleneck constrains expressivity. Log-Linear Attention states this explicitly: linear attention and state-space models are still RNNs at their core, so their use of a fixed-size hidden state is a fundamental limitation, especially for associative recall and other long-range memory tasks (Guo et al., 5 Jun 2025). The systematic hybrid study reaches a related conclusion at scale: purely linear models typically show weaker recall than full attention, even when language modeling remains relatively stable (Wang et al., 8 Jul 2025).

Several papers therefore analyze the gap between linear and softmax attention at the level of attention geometry or concentration. FLatten Transformer attributes performance degradation in existing linear attention to weak focus ability and limited feature diversity, noting the low-rank bottleneck

$S_t=S_{t-1}+k_t v_t^\top$7

and proposes a focused mapping

$S_t=S_{t-1}+k_t v_t^\top$8

together with a depthwise convolution residual

$S_t=S_{t-1}+k_t v_t^\top$9

to sharpen attention and restore feature diversity (Han et al., 2023). NaLaFormer similarly argues that standard linear kernels neglect query norms and suppress negative inner-product structure; it introduces norm-direction decomposition, a query-norm-aware power

$o_t=q_t S_t$0

and a cosine-inhibit positive mapping to restore norm-guided spikiness and norm consistency (Meng et al., 26 Jun 2025).

A more explicitly statistical route is taken by Linear Log-Normal Attention, which analyzes softmax attention as approximately log-normal with concentration controlled by temperature, then constructs exponential feature maps

$o_t=q_t S_t$1

to match softmax-like distributional and spectral-gap behavior while retaining linear complexity (Nahshan et al., 2023). MetaLA instead asks for the optimal linear approximation to the softmax attention map under three conditions—dynamic memory ability, static approximation ability, and least parameter approximation—and concludes that the minimal theoretically preferred design is based on query plus dynamic decay, not on a separate key matrix (Chou et al., 2024).

Recent work also explores expressivity by going beyond first-order statistics. Higher-order Linear Attention introduces second- and third-order causal streaming operators with compact prefix sufficient statistics such as

$o_t=q_t S_t$2

and second-order output

$o_t=q_t S_t$3

while preserving constant-size state per head and linear-time sequence processing (Zhang et al., 31 Oct 2025). This indicates one route by which linear attention can become more expressive without reverting to quadratic token-token maps.

4. Hierarchical, adaptive, and hybrid memory constructions

A major trend is the move from a single recurrent state to multiple states, hierarchical states, or selective combinations of linear and full attention. Log-Linear Attention is the clearest hierarchical construction. It replaces the single fixed hidden state with a logarithmically growing set of states, using a Fenwick tree partition of the prefix into up to

$o_t=q_t S_t$4

buckets, and computes

$o_t=q_t S_t$5

With a HODLR-like hierarchical mask, it attains $o_t=q_t S_t$6 training complexity, $o_t=q_t S_t$7 training memory, and $o_t=q_t S_t$8 time and space at decoding (Guo et al., 5 Jun 2025).

Dynamic Linear Attention pushes the multi-state idea further by replacing fixed bucket schedules with information-aware state formation. It defines per-token state information

$o_t=q_t S_t$9

uses a thresholded boundary indicator $\operatorname{softmax}(QK^\top+M)V$0, and maintains a capacity-bounded cache

$\operatorname{softmax}(QK^\top+M)V$1

merging adjacent low-information states when capacity is full (Wang et al., 9 Jun 2026). This is explicitly motivated by a deviation bound showing that summarization error increases with within-block heterogeneity, so adaptive boundaries can outperform any fixed blocking policy on non-stationary sequences (Wang et al., 9 Jun 2026).

Hybridization is the other major strategy. BASED combines global Taylor-style linear attention with exact local sliding-window softmax attention, using the feature map

$\operatorname{softmax}(QK^\top+M)V$2

and a sliding window of width $\operatorname{softmax}(QK^\top+M)V$3 to traverse the recall-memory frontier (Arora et al., 2024). NAtS-L performs token-level hybridization within the same layer: chunk scores

$\operatorname{softmax}(QK^\top+M)V$4

assign some chunks to Gated DeltaNet-style linear attention and others to softmax attention, so only selected chunks pay the full-attention cost (Deng et al., 3 Feb 2026). Kimi Linear adopts a layerwise hybrid at larger scale, interleaving three KDA layers with one MLA layer, and reports that this 3:1 ratio gave the best quality-throughput trade-off in its study (Team et al., 30 Oct 2025).

The systematic hybrid study provides a broader empirical frame for these design choices. It evaluates six linear attention variants across five hybridization ratios and finds that standalone language-modeling strength does not predict hybrid success; recall improves sharply as more full-attention layers are added, especially when the linear-to-full ratio moves toward $\operatorname{softmax}(QK^\top+M)V$5, and the authors recommend HGRN-2 or GatedDeltaNet with a ratio between $\operatorname{softmax}(QK^\top+M)V$6 and $\operatorname{softmax}(QK^\top+M)V$7 for efficient Transformer-level recall (Wang et al., 8 Jul 2025).

5. Autoregressive decoding, parallel training, and systems support

Although linear attention is often motivated by asymptotic complexity, several papers emphasize that practical utility depends on systems design. In autoregressive language modeling, a comprehensive evaluation of seven linear attention methods—FLASH-Local, FLASH-Global, Linformer, Performer, TransNormer, YOSO, and ReLU-based linear attention—shows that methods successful in encoder settings often degrade in decoder-based GPT-2 and T5 (You et al., 2024). The same paper identifies two concrete issues: future-token leakage from naive depthwise convolution augmentation, and incompatibility between ordinary causal convolution and speculative decoding’s tree-structured token dependencies. It addresses these with masked DWConv, grouped linear attention, and an unfolded DWConv implementation compatible with Medusa-style speculative decoding, reporting up to a $\operatorname{softmax}(QK^\top+M)V$8 perplexity reduction on LLaMA-2-7B and roughly $\operatorname{softmax}(QK^\top+M)V$9 speculative-decoding throughput (You et al., 2024).

Sequence-parallel training has likewise been specialized for linear attention. LASP exploits the right-product-first structure

$$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$0

and the compact inter-chunk state

$$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$1

to replace sequence-length-dependent collective communication by ring-style point-to-point passing of $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$2 states (Sun et al., 2024). On linear-attention models including TransNormerLLM, LASP scales sequence length to $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$3 on 128 GPUs, reported as $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$4 longer than existing sequence-parallel methods, while preserving convergence (Sun et al., 2024).

At deployment time, teacher-student conversion has emerged as another systems pathway. RADLADS presents a three-step protocol—attention weight transfer, attention hidden-state alignment, and knowledge distillation—to convert softmax-attention transformers into purely recurrent linear-attention-style decoders based on new RWKV variants. The conversion requires only $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$5–$$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$6 million tokens, described as less than $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$7 of the teacher’s token count, and produces 7B, 32B, and 72B recurrent models with constant-memory inference and quality close to the original transformer (Goldstein et al., 5 May 2025).

These systems results indicate that linear attention is no longer only an abstract complexity class. It is also a software-and-kernel problem involving chunkwise scans, P2P state passing, speculative-decoding compatibility, and distillation pipelines (You et al., 2024, Sun et al., 2024, Goldstein et al., 5 May 2025).

6. Model families and domain-specific instantiations

The recent literature spans several distinct but overlapping linear-attention families. Kernelized Transformer variants remain prominent in vision and controlled comparisons. Focused Linear Attention in FLatten Transformer improves ImageNet-1K classification when substituted into DeiT, PVT, PVTv2, Swin, and CSwin; representative gains include DeiT-T from $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$8 to $$\text{Attention}(Q,K,V)=\text{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,$$9 and PVT-T from $O^\phi=\phi(Q)\phi(K)^T V,$0 to $O^\phi=\phi(Q)\phi(K)^T V,$1 (Han et al., 2023). Agent Attention introduces a quadruple $O^\phi=\phi(Q)\phi(K)^T V,$2 and the two-stage factorization

$O^\phi=\phi(Q)\phi(K)^T V,$3

which the paper identifies as a generalized form of linear attention; on Stable Diffusion it is reported as $O^\phi=\phi(Q)\phi(K)^T V,$4 faster than Stable Diffusion and $O^\phi=\phi(Q)\phi(K)^T V,$5 lower FID than ToMeSD at equivalent speed (Han et al., 2023). NaLaFormer extends this line by adding norm-aware spikiness and cosine-inhibit mappings, reporting ImageNet-1K accuracies up to $O^\phi=\phi(Q)\phi(K)^T V,$6 and gains on COCO, ADE20K, and language modeling (Meng et al., 26 Jun 2025).

A second family comprises gated recurrent or delta-rule models. Gated DeltaNet, Mamba-2-derived constructions, HGRN, HGRN-2, RetNet, GLA, and RWKV-style models are repeatedly analyzed as linear attention in recurrent form (Guo et al., 5 Jun 2025, Wang et al., 8 Jul 2025, Goldstein et al., 5 May 2025). Kimi Delta Attention exemplifies the delta-rule lineage: $O^\phi=\phi(Q)\phi(K)^T V,$7 with channel-wise forget gates, and is used as the dominant token-mixing block in Kimi Linear (Team et al., 30 Oct 2025). In forecasting, SAMoVAR treats linear attention as a dynamic VAR-like predictor and reports best or near-best performance on 12 real multivariate TSF datasets (Lu et al., 11 Feb 2025).

A third family consists of hybrids and locality-aware interpolations. BASED mixes linear attention with sliding-window softmax to balance recall and throughput (Arora et al., 2024). Local Linear Attention takes a nonparametric-regression perspective, viewing ordinary softmax attention as Nadaraya–Watson local constant regression and linear attention as a global linear model, then introducing query-local linear correction with weights

$O^\phi=\phi(Q)\phi(K)^T V,$8

to reduce boundary bias (Zuo et al., 1 Oct 2025). This line is more computationally demanding, but it explicitly targets the interpolation between linear and softmax attention rather than only subquadratic speed (Zuo et al., 1 Oct 2025).

Linear attention also appears in diffusion and controllable generation. GateControl is built on the linear-attention diffusion backbone SANA and inserts a post-attention token-wise gating module,

$O^\phi=\phi(Q)\phi(K)^T V,$9

to integrate aligned and unaligned conditions without multimodal attention. The gated control module adds only $$\mathrm{Attn}(Q,K,V)\approx \phi(Q)\big(\phi(K)^\top V\big),$$0M parameters, and one training example reports strong canny alignment in about $$\mathrm{Attn}(Q,K,V)\approx \phi(Q)\big(\phi(K)^\top V\big),$$1 steps versus over $$\mathrm{Attn}(Q,K,V)\approx \phi(Q)\big(\phi(K)^\top V\big),$$2 for the ungated baseline (Liu et al., 29 Mar 2026).

Taken together, these papers show that “linear attention model” now refers not to a single architecture, but to a design space spanning kernelized approximations, gated recurrent state updates, hierarchical memory systems, local-regression variants, and hybrid stacks.

7. Limitations, controversies, and open directions

The most persistent limitation is recall. Multiple papers state that fixed-size recurrent compression degrades long-range retrieval, exact copying, or associative memory relative to full attention (Guo et al., 5 Jun 2025, Arora et al., 2024, Team et al., 30 Oct 2025). Even when language-modeling perplexity is competitive, recall-specific benchmarks often remain the decisive differentiator in favor of hybrid or multi-state designs (Wang et al., 8 Jul 2025). This suggests that linear attention’s main unresolved question is not merely perplexity, but which memories should be preserved, how long they should persist, and whether bounded state can emulate a KV cache closely enough for exact retrieval.

A second open issue is architecture sensitivity. In autoregressive LLMs, several existing linear attention methods deteriorate when տեղափոխed from encoder-style to decoder-style settings, and naive locality augmentations can produce future-token leakage (You et al., 2024). In in-context learning, performance depends strongly on feature map choice: the identity map performs very poorly, while squared ReLU is substantially better (Goel et al., 19 Feb 2026). This indicates that “linear attention” is not a uniform operating point; implementation details, parametrization, and normalization schemes materially change both optimization and capability.

A third issue is that added expressivity often erodes the original simplicity of linear attention. Log-linear, higher-order, local-linear, and dynamic multi-state methods improve memory or approximation quality, but they introduce hierarchical masks, extra summaries, matrix inversions, scan operators, gating modules, or specialized kernels (Guo et al., 5 Jun 2025, Zhang et al., 31 Oct 2025, Zuo et al., 1 Oct 2025, Wang et al., 9 Jun 2026). This suggests a broad trade-off: improving beyond the single-state recurrence tends to require either more states, richer sufficient statistics, or periodic fallback to full attention.

Finally, several papers explicitly reject the idea of a universal replacement. Log-Linear Attention notes that some tasks do not improve, that Fenwick-tree partitioning may not be optimal everywhere, and that implementation is more complex than standard linear attention (Guo et al., 5 Jun 2025). Local Linear Attention reports stronger adaptation to non-stationarity and test-time regression, but also emphasizes higher compute cost than softmax attention and leaves LLM-scale training as an open engineering problem (Zuo et al., 1 Oct 2025). The current literature therefore supports a more specific conclusion: linear attention models are best viewed as a family of efficiency-expressivity trade-offs, increasingly augmented by hierarchy, gating, and hybridization, rather than as a single settled alternative to softmax attention.