---
title: Linear Self-Attention (LSA) Models
url: https://www.emergentmind.com/topics/linear-self-attention-lsa-models
type: topic
---

# Linear Self-Attention (LSA) Models

Linear Self-Attention (LSA) models encompass a spectrum of mechanisms designed to approximate or reformulate the traditional self-attention operation from Transformers, achieving linear time and space complexity with respect to sequence length. These methods address the quadratic bottleneck of standard attention by leveraging structural assumptions (e.g., low-rank, kernel factorization, randomized mappings, histogram aggregation, or external memory) and, in some cases, kernelized or graph signal processing perspectives. LSA models have been instantiated as both primitives in NLP and vision backbones, as well as domain-specialized modules for recommendations, multimodal fusion, and supporting efficient in-context learning.

## 1. Foundational Principles and Motivations

Standard self-attention as in Transformers computes an $n\times n$ matrix for a sequence of length $n$, requiring $\mathcal{O}(n^2)$ computation and memory. LSA models circumvent this via assumptions such as:

- **Low-rank approximation**: The attention matrix is assumed or empirically found to be low-rank in many practical settings [2006.04768]. This allows projection into a lower-dimensional space.
- **Kernel factorization**: The exponential similarity kernel (e.g., $\exp(q^\top k)$) is factorized as a product of feature mappings and approximated using random features or trainable kernels [2211.04076, 2204.04667].
- **Sparsity and structured aggregation**: Structural bias, such as codebook quantization or local attention, enables compressive representations and efficient aggregation [2105.14068, 2210.17115].
- **Randomized/sampling-based approximation**: The attention weights are sampled or estimated using Locality Sensitive Hashing or randomized hash collision mechanisms [2111.09714].
- **External attention and memory**: Lightweight external memory matrices allow for attentive computation that avoids the full quadratic affinity matrix [2105.02358].

The central motivation across LSA variants is to maintain the representation power of dense attention while achieving substantial efficiency gains, making attention practical for very long sequences or for fast inference in real-time and resource-limited scenarios.

## 2. Linearization Methodologies in LSA Mechanisms

Several core methodologies underpin LSA models:

| Approach              | Computational Principle                  | Key Examples                     |
|-----------------------|------------------------------------------|----------------------------------|
| Low-rank projections  | Keys/values projected via $\mathbf{E}, \mathbf{F}$ ($n\times n$ $\rightarrow$ $n\times k$), $k \ll n$ | Linformer [2006.04768], MLSA4Rec [2407.13135] |
| Random feature kernel | Exponential kernels factorized as products of random or trainable mappings $\phi(q)$, $\phi(k)$ | Performer, RFA [2204.04667], LARA [2204.04667], [2211.04076] |
| Histogram aggregation | Codeword histogram replaces full interactions; prefix sums/histograms over quantized codebooks | LISA [2105.14068]              |
| Sampling-based        | Bernoulli/Lokality Sensitive Hashing samples attention contributions | YOSO [2111.09714]              |
| External/global memory| Attention computed via two learnable memories, affording $O(n)$ complexity | External Attention [2105.02358]|
| Trainable feedforward | Self-attention kernel approximated by shallow or GLU-based FFNs ensuring positivity | [2211.04076]                   |
| Singular value domain | Attentive graph filter with polynomial filtering in learned singular value domain | AGF [2505.08516]               |
| Element-wise expansion| Exponential per-channel Taylor expansion replaces dense dot-products | Element-wise Attention [2501.05730] |

For instance, the Linformer approximates the attention matrix as:
$$
\mathrm{head}_i = \operatorname{softmax}\left(Q W^Q_i (E_i K W^K_i)^\top / \sqrt{d}\right)\cdot (F_i V W^V_i)
$$
where projection matrices $E_i, F_i$ are $n\times k$. This reduces cost to $O(nk)$.

In histogram-based approaches such as LISA, the input is quantized into a fixed number of codewords across multiple codebooks. Each token's representation is encoded as a histogram over these codewords, and attention is aggregated by computing context-dependent weighted averages over codewords, yielding fixed attention complexity that is independent of sequence length.

## 3. Theoretical Properties and Trade-offs

### Low-Rank and Approximation Guarantees

- For methods based on low-rank projections (Linformer, Revisiting Linformer [2101.10277]), theoretical results guarantee that with $k = O(\log n)$ or $k = O(d/\varepsilon^2)$, the low-rank projection retains most of the self-attention effect with bounded error, by a Johnson–Lindenstrauss-type argument [2006.04768].
- Bias-variance trade-offs arise in random feature and sampling-based approaches [2204.04667, 2111.09714]. LARA [2204.04667] reduces bias relative to random feature attention by employing multiple importance sampling and self-normalization, balancing the expressiveness of softmax attention with linear resource requirements.

### Flexibility Versus Approximation Quality

- Trainable kernels [2211.04076] and element-wise Taylor expansions [2501.05730] increase the representational flexibility but introduce extra parameterization and tuning requirements. The order of the polynomial in element-wise attention controls the "spikiness" and expressiveness, trading off efficiency against fidelity to the original exponential attention distribution.

### Limiting Cases and Efficiency

- Recent analyses show that for some variants, e.g., with codebook histogram-based LISA [2105.14068] and external attention [2105.02358], the limiting computational complexity is $\mathcal{O}(n)$ (linear in sequence length), provided codebook/memory size is held fixed.
- Methods that remove explicit dependencies on hyperparameters or matrix rank (via rearrangement of computations) offer further simplification and robustness across tasks [2101.10277].

## 4. Practical Applications and Empirical Results

### NLP and Language Modeling

- LSA models such as Linformer and LLN Attention [2311.13541] deliver comparable performance to standard softmax attention on masked language modeling, GLUE, and long-sequence reading comprehension, at a fraction of time/memory consumption.
- Sampling-based and histogram-based approaches enable long document encoding, sequence tagging, and high-throughput generation [2105.14068, 2111.09714].

### Computer Vision

- Plug-and-play LSA modules have been successfully integrated into vision transformers, object detection, and image segmentation backbones [2104.02301, 2402.17507, 2210.17115]. The ELSA module [2112.12786] demonstrates improved local feature extraction via Hadamard attention and ghost head augmentation.
- AGF [2505.08516] mitigates over-smoothing in deep vision transformers, yielding state-of-the-art results on Long Range Arena and UEA time series benchmarks.

### Multimodal and Recommendation Systems

- LSA architectures have been adapted for multimodal fusion of hyperspectral and LiDAR inputs [2104.02301], multimodal plug-in modules (channel and spatial attention), and codebook histogram-based sequence representations in recommendation [2105.14068, 2407.13135].
- MLSA4Rec [2407.13135] illustrates linear-complexity hybrid recommender models by combining a low-rank LSA module (latent interest aggregation) with a selective state-space model (Mamba), outperforming both isolated LSA- and SSM-based recommenders.

## 5. Model Properties, Extensions, and Theoretical Insights

### Decomposition and Parametrization

- The in-context learning ability of linear attention models is highly sensitive to parametrization. Models with merged (shared) key/query matrices exhibit abrupt acquisition of ICL through a single large jump in loss when escaping an unstable fixed point, implementing least-squares regression over cubic features [2501.16265].
- In contrast, models with separate parametrization for keys and queries exhibit saddle-to-saddle training dynamics, corresponding to progressive learning of principal components of the input covariance—a mechanism interpretable as in-context principal component regression.

### Algorithmic Expressivity

- The introduction of bias terms in extended linear self-attention (ELSA) modules enables arbitrary constant outputs, skip connections, and general matrix multiplications [2503.23814]. This supports heuristic implementations of computational algorithms—such as unrolled gradient descent for ridge regression—by directly manipulating input and context in a sequence of ELSA blocks.

### Graph Signal Processing and Spectral Properties

- AGF [2505.08516] provides a principled connection between self-attention and graph signal processing, revealing that standard attention acts as a low-pass graph filter. Learning advanced graph filters in the singular value domain allows LSA models to adaptively capture high-frequency information, thereby increasing expressive power while retaining $O(nd^2)$ complexity.

## 6. Limitations, Challenges, and Future Directions

- The assumption of attention matrix low-rankness may fail on certain classes of tasks or highly entropic/disordered inputs, warranting further empirical and theoretical study [2101.10277].
- Bias and approximation errors introduced by histogram quantization, random feature mapping, or sampling may degrade performance in regimes requiring fine-grained or rare-event modeling [2105.14068, 2204.04667].
- Architectural and training choices—including the rank of projections, size of codebooks, choice of kernel basis (e.g., Jacobi for stability), and the interplay of local/global attention—remain active research areas with significant impact on scalability and generalization.
- The integration of LSA mechanisms with other efficient architectures—such as SSMs (Mamba), deep recurrent mechanisms, or attention sharing across layers (LiSA)—offers new opportunities for both empirical gains and deeper theoretical understanding of what efficient, scalable attention should entail in modern large-scale models [2408.01890, 2407.13135].
- Further progress is anticipated in hybrid models that dynamically blend local and global LSA modules, data-adaptive graph filters, and task-aware parametrization for robust, high-throughput training and inference across modalities.

## 7. Summary Table of Representative LSA Models

| Name/Ref.                        | Core Linearization Method                | Domain(s)         | Notable Properties                  |
|-----------------------------------|------------------------------------------|-------------------|-------------------------------------|
| Linformer [2006.04768]            | Low-rank projection via learned matrices | NLP, vision       | $O(nk)$, theoretical guarantees     |
| Revisiting Linformer [2101.10277] | Rearranged low-rank, k-independent      | NLP, vision, audio| Eliminates projection hyperparam    |
| LISA [2105.14068]                 | Codebook histogram aggregation           | Recommendation    | $O(n)$, supports causal/noncausal   |
| External Attention [2105.02358]   | Shared external memories, linear layers  | Vision, MLP       | Dataset-level priors, $O(n)$        |
| LARA [2204.04667]                 | Multiple randomized proposals, kernel    | CV, NLP, video    | Reduces RFA bias, linear complexity |
| ViT-LSLA [2210.17115]             | Q projection only (replace K, V with input) | Vision         | Reduces parameters, adds position   |
| AGF [2505.08516]                  | Polynomial graph filter in singular value| LRA, CV, time series | Graph spectrum, mitigates smoothing |
| Element-wise Attention [2501.05730]| Per-channel Taylor expansion, RNN reform| Time series, generic| $O(tLD)$ train, $O(tD)$ inference   |
| ELSA [2112.12786]                 | Hadamard attention, ghost heads          | Vision, ViT       | Drop-in, high-order local mapping   |
| Interactive MHSA [2402.17507]     | Landmarking, cross-head layer           | Vision            | Decomposed, interactive, $O(N)$     |
| MLSA4Rec [2407.13135]             | Low-rank interest, hybrid with Mamba     | Recommendation    | Local-global, linear, SSM hybrid    |
| LiSA [2408.01890]                 | Cross-layer sharing, low-rank diff.      | LLMs              | 6× Q/K compression, per-layer fusion|

The continuous development and refinement of LSA models reflect a fundamental trend in deep learning: the pursuit of expressive global context modeling with favorable resource scaling, facilitated by principled architecture and algorithm design grounded in approximation theory, optimization, and domain structure.

Source: https://www.emergentmind.com/topics/linear-self-attention-lsa-models