Linear Transformer Architectures

• Linear transformers are architectures that compute self-attention in O(N) time, enabling efficient processing of long or high-resolution data.
• They utilize techniques such as kernel feature mappings, cross-normalization, and low-rank projections to approximate traditional softmax attention.
• Empirical studies in vision, NLP, and scientific computing demonstrate competitive accuracy with dramatically reduced computational costs.

A linear transformer is a family of Transformer architectures in which the self-attention mechanism is structured so that its time and space complexity is $O(N)$ in the number of input tokens $N$, in contrast to the $O(N^2)$ scaling of conventional (softmax) self-attention. This radical shift in scaling is achieved through algebraic factorization, kernelization, low-rank projection, or architectural re-design, enabling transformers to process much longer sequences or higher-resolution visual data within fixed compute and memory budgets. Linear transformers have emerged as a vital paradigm in language modeling, vision, scientific computing, and beyond, with multiple variants and theoretical frameworks unifying this class.

1. Mathematical Foundations of Linear Self-Attention

Linear transformers fundamentally alter the calculation of the attention matrix. In standard Transformers, self-attention is computed as

$$A(x) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d}}\right) V$$

where $Q$, $K$, and $V$ are learned projections of the input $x \in \mathbb{R}^{N\times D}$. The bottleneck is forming the $N \times N$ score matrix $QK^T$ and normalizing by softmax, both of which entail quadratic cost in $N$.

Linear transformers avoid this by (i) replacing softmax normalization with either algebraically linear normalizations or kernel feature maps, (ii) reordering matrix multiplications to exploit associativity, or (iii) compressing the key/value sequences via low-rank projections. Examples include:

• Kernel Methods: The inner product kernel $K(Q_i, K_j) = e^{Q_i^T K_j}$ is replaced with a feature map $\phi$ so that $K(Q_i, K_j) \approx \phi(Q_i)^T \phi(K_j)$; enabling $A(Q, K, V) = \phi(Q) \left( \phi(K)^T V \right)$ and thus $O(N)$ cost (Mercat, 2020).
• Cross-Normalization: UFO-ViT replaces softmax with $L_2$-based normalizations along both channel and spatial axes (cross-normalization, XNorm), reorders as $Q(K^T V)$ with normalization applied to both $Q$ and $K^T V$, thus again $O(N)$ (Song, 2021).
• Low-Rank Projection: Linformer and its variants project $K$ and $V$ down to dimension $k\ll N$, compute attention in this low-dimensional space, yielding $O(Nk)$ cost (Hernandez et al., 23 Jan 2025, Wang et al., 24 Oct 2025).
• Blockwise or Vector Quantization: Transformer-VQ bins keys via learnable VQ, such that attention is aggregated over codewords rather than full tokens; enabling exact softmax attention in $O(N)$ (Lingle, 2023).

These approaches are not mutually exclusive; hybrid mechanisms and further kernelizations exist.

2. Principal Architectures and Variants

Linear transformers extend across NLP, vision, signal processing, and scientific computing. Key architectural classes include:

• Kernel-Based Linear Transformers: As introduced by Katharopoulos et al., softmax is replaced with kernel expansions, e.g., $\mathrm{elu}(x)+1$. Second-order (Higher-Order) forms capture quadratic terms for better softmax approximation while increasing runtime from $O(nrd_v)$ to $O(nd_k^2 d_v)$ (Mercat, 2020).
• Cross-Normalization-Based Linear Transformers: UFO-ViT eliminates softmax via dual-axis normalization, with minimal changes to standard attention code and retains competitive empirical accuracy (Song, 2021).
• Low-Rank and Partitioned Attention: Linformer-style projections are employed, as in 5G LDPC Linear Transformer (for channel decoding) (Hernandez et al., 23 Jan 2025) and SAL-T, which uses physically-informed spatial partitioning for particle physics (Wang et al., 24 Oct 2025).
• Vector Quantization and Caching: Transformer-VQ achieves linear-time softmax attention over long sequences by VQ-ing keys into codebook codewords, organizing the value cache for incremental updates (Lingle, 2023).
• Hybrid and Task-Specific Variants: FLASH combines a Gated Attention Unit (GAU) that absorbs expressivity into gating and applies mixed chunk attention for linear cost (Hua et al., 2022). SOFT applies a pure Gaussian RBF kernel with Nyström-based low-rank approximation and Newton–Raphson pseudoinversion, yielding robust linear attention in vision (Lu et al., 2021).
• Decoupled/Asymmetric Models: CARE Transformer uses channel-splitting, applying linear attention to a compact global branch and local convolutions to larger spatial branches, fusing via a dual-interaction module to improve mobile deployment (Zhou et al., 2024).

3. Empirical Performance and Practical Deployments

Linear transformers enable practical operation at high sequence lengths ($N \gg 1$k) and/or large token counts:

• Vision: UFO-ViT outperforms most prior vision transformers and matches state-of-the-art on ImageNet-1k (e.g., 83.3% top-1 accuracy for UFO-ViT-B with 64M params, matching or beating DeiT, XCiT, Swin) and offers batch size and throughput scaling advantages (up to 3× throughput on high-res images) (Song, 2021). SOFT improves over Linformer, Performer, and Nyströmformer for linear attention in pyramidal visual transformers (Lu et al., 2021). CARE achieves 78.4–82.1% top-1 accuracy on ImageNet-1k at 0.7–1.9 GMACs, state-of-the-art in the mobile regime (Zhou et al., 2024).
• Language Modeling: FLASH and Transformer-VQ match or slightly underperform full transformers at long context lengths (e.g., PG-19/Enwik8), but provide $4.9\times$–$12.1\times$ training/inference speedups (Hua et al., 2022, Lingle, 2023). TransNormer achieves competitive or superior perplexity and accuracy relative to vanilla transformers across multiple NLP tasks and the Long-Range Arena benchmark, with linear time/space (Qin et al., 2022).
• Scientific Computing: Transolver (with Physics-Attention) is re-cast as a special case of linear attention, and LinearNO shows that simple, canonical linear attention matches or surpasses Transolver/Physics-Attention on a wide range of PDE benchmarks, reducing parameters and compute by 30–40% (Hu et al., 9 Nov 2025).
• Signal Processing: 5G LDPC Linear Transformer demonstrates linear transformers can match full-transformer performance in neural LDPC code decoders, offering 2× latency improvement and more training updates within an equivalent wall-clock budget (Hernandez et al., 23 Jan 2025).

4. Implementation Strategies, Complexity, and Code Examples

The practical transition from quadratic to linear attention universally exploits algebraic rearrangement, feature-kernel expansions, and/or projection:

• Reordering and Factoring: Compute $K^T V$ first (cost $O(Nhd)$), then left-multiply by $Q$, insert norm as needed. UFO-ViT's modification to baseline Transformer code involves 5–6 changed lines: removes $QK^T$, drops softmax, applies two XNorm calls, swaps order (Song, 2021).
• Kernel Expansion: Map $Q,K$ with $\phi(\cdot)$ and carry out $\phi(Q)(\phi(K)^T V)$, normalizing per row/column per kernel (Mercat, 2020, Lu et al., 2021).
• Low-Rank/Partitioning: Project $K,V$ via learned $P_K,P_V$, compute attention in the $k \ll N$ low-rank space, e.g., in Linformer, SAL-T, 5G LDPC, leading to efficient $O(Nk)$ routines (Hernandez et al., 23 Jan 2025, Wang et al., 24 Oct 2025).
• Vector Quantization/Cache: Hash keys with VQ, aggregate values and codes, then perform softmax-based attention over the much smaller codebook or block-wise tokens, with incremental updates for each token step (e.g., Transformer-VQ) (Lingle, 2023).
• Post-Normalization: To resolve instability, as identified in TransNormer, normalization is shifted after the attention computation (e.g., RMSNorm after $Q(K^T V)$), which bounds gradients and avoids attention dilution, restoring reliable convergence (Qin et al., 2022).

5. Theoretical and Empirical Limitations

Although linear transformers offer compelling efficiency gains, inherent or empirically-observed trade-offs persist:

• Probabilistic Semantics: Replacing softmax with norm or kernel destroys the simplex constraint—attention weights are not interpretable as probabilities. UFO-ViT and CARE explicitly note this (Song, 2021, Zhou et al., 2024).
• Expressivity Loss: Removing softmax nonlinearity weakens the model's ability to simulate sparse or extremely peaked attention, limiting effectiveness in tasks that depend on focused context (e.g., long-context NLP); empirical ablation in UFO-ViT, TransNormer, and Higher-Order Linear Transformer domains supports this (Song, 2021, Mercat, 2020, Qin et al., 2022).
• Dimensionality Coupling: Many linear mechanisms scale linearly only if the per-token embedding dimension $h, d$ or projection size $k$ is small and independent of $N$; for large $d$ the cost reduction relative to quadratic is diminished (Song, 2021).
• Gradient Instability and Attention Dilution: Kernel-based linear attention can result in unbounded gradients, harming convergence, and attention dilution over long sequences (scores are too diffuse). TransNormer resolves this via post-attention normalization and block-wise attention in early layers to reintroduce locality (Qin et al., 2022).
• Approximation Quality: For low-rank or blockwise methods (Linformer, Transformer-VQ), approximation quality drops if the codebook or projection rank is too small, degrading fine-grained attention; parameter tuning is essential (Hernandez et al., 23 Jan 2025, Lingle, 2023).

6. Extensions, Task Domains, and Future Directions

Linear transformers have demonstrated impact across diverse modalities and tasks:

• Vision: Linear attention is mainstream in ViTs, scene understanding, dense predictions, and multi-resolution architectures; active research targets optimal kernel/function design and hybrid models combining linear and local softmax attention (Lu et al., 2021, Zhou et al., 2024, Wang et al., 22 Jan 2025).
• Scientific/Engineering Computing: Advances in data-driven PDE solvers and neural operators, where large unstructured grids or spatial points are common, directly exploit linear-attention frameworks (Hu et al., 9 Nov 2025).
• Biomedical Signal Processing: Dynamic Linear Transformers handle arbitrarily sized or 3D volumes using ROI-based dynamic token reduction, yielding significant compute savings in medical segmentation (Zhang et al., 2022).
• Sparse and Mobile Transformer Deployment: CARE Transformer and LiT optimize linear attention for edge devices and fast generative image modeling, respectively, leveraging channel-wise partitioning, low-head-count attention, transfer learning, and knowledge distillation (Zhou et al., 2024, Wang et al., 22 Jan 2025).
• Open Research: Outstanding questions include optimal kernel feature maps (e.g., ReLU, GELU, randomized), mixed linear/nonlinear attention for mid-range contexts, advances in knowledge distillation, and scaling to massive multimodal or ultra-high-resolution models (Wang et al., 22 Jan 2025).

7. Comparative Empirical Performance

A survey of recent models demonstrates that linear transformers match or improve on both compute and accuracy metrics compared to quadratic transformers, with results collated below (all cited from the original works):

Model	Task	Main Results
UFO-ViT	ImageNet-1k	83.3% top-1 (B, 64M params), throughput 2–3× vanilla ViT
SOFT	ImageNet-1k	79.3% (Tiny, 1.9G FLOPs); bested Linformer/Performer/Nyström
CARE	ImageNet-1k	78.4–82.1% top-1, 0.7–1.9 GMACs
Transformer-VQ	Enwik8, PG-19, ImageNet64	0.99 bpb (Enwik8), 26.6 ppl (PG-19), >12× faster at 32k sequence
FLASH	C4, Wiki40B, PG-19	Parity with Transformer++; up to 12.1× speedup at long context
5G LDPC Linear Transformer	Channel decoding	BER within 0.1 dB of full Transformer; 2× decoding speed
TransNormer	GLUE, LRA, Wikitext-103	Matches/bests vanilla Transformer and Performer at 1.5–3× speed
LinearNO	PDE neural operators	30–40% fewer params/flops than Transolver, lower $L_2$ error

All values appear verbatim from the referenced papers (Song, 2021, Lu et al., 2021, Zhou et al., 2024, Wang et al., 22 Jan 2025, Lingle, 2023, Hua et al., 2022, Hernandez et al., 23 Jan 2025, Qin et al., 2022, Hu et al., 9 Nov 2025).

---

Linear transformers represent a theoretically mature and practically impactful class of architectures, unifying diverse strategies (kernelization, factorization, low-rank, blockwise, quantization) to break the $O(N^2)$ bottleneck of self-attention. Current evidence indicates linear attention can achieve state-of-the-art results in multiple modalities provided the architectural and normalization choices mitigate potential weaknesses (expressivity, instability, dilution). Active lines of research include stability-enhancing normalizations, locality augmentation, hybrid noise-variance distillation, and efficient mobile deployment.