---
title: Sub-quadratic Self-Attention
url: https://www.emergentmind.com/topics/sub-quadratic-self-attention
type: topic
---

# Sub-quadratic Self-Attention

Sub-quadratic self-attention refers to algorithmic strategies and architectural mechanisms that reduce the time and space complexity of the Transformer’s self-attention operation from the classical O(n²) (where n is the sequence length) to o(n²)—typically O(n k), O(n log n), or even O(n)—while retaining as much of the full dot-product attention’s expressiveness as possible. The field, originating from the computational bottlenecks of dense attention in natural language and vision tasks, has developed a spectrum of approaches ranging from low-rank kernel factorizations to adaptive sparsification, randomized projection, and data-driven block-wise evaluation. The goal is to enable the training and deployment of Transformer models on long contexts, high-resolution grids, or resource-constrained hardware, without catastrophic loss in accuracy or model capacity.

## 1. Low-rank and Covariance-based Approximations

A significant portion of sub-quadratic self-attention methods exploits the empirical observation that attention score matrices in practice, especially over natural language and pre-training corpora, are intrinsically low-rank. The work “Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation” performs a principal component analysis of vectorized attention matrices A (post-softmax) over large samples from BERT-base and finds that the top k≈200 eigenvalues capture over 90% of the total variance even for n=128 (n²=16,384). This low effective rank is even more pronounced in later transformer layers. This eigenspectrum concentration motivates a workflow where, instead of computing all n² attention scores per head:

- A subset P of k≪n² token pairs is selected for exact computation using a greedy, covariance-informed algorithm that aims to minimize the posterior unexplained variance via the Schur complement of Σ_a, the attention score covariance.
- All other (uncomputed) entries are then reconstructed by a Bayes-optimal (Posterior Mean) linear estimator under a Gaussian assumption, yielding for the missing indices
  $$
  \hat{a}_{\bar{P}} = \Sigma_{\bar{P},P}\Sigma_{P,P}^{-1} a_P,
  $$
  where Σ is the empirical covariance.

The total time scales as O(k d + n² k) (for batch reconstruction), but in practical per-row reconstructions, the cost becomes O(n k′ d) for k′≪n, yielding a truly sub-quadratic cost profile. Empirical results show that using only 24 or 32 entries per row reduces FLOPs by 44–25% with minimal drop (<2%) in downstream accuracy, and yields 1.8–2.7× speedups in runtime [2106.08823].

## 2. Kernelization, Projection, and Feature Map Approaches

Many sub-quadratic algorithms recast the exponential dot-product attention kernel as a finite, sometimes learned, inner product in an auxiliary feature space. This kernelization allows a key computational rearrangement:

$$
\exp(q_i^\top k_j) \approx \phi(q_i)^\top \phi(k_j)
$$

Enabling the classic attention contraction to:

$$
\mathrm{Att}(q_i; K, V) \approx \frac{\phi(q_i)^\top \sum_j \phi(k_j)v_j^\top}{\phi(q_i)^\top \sum_j \phi(k_j)}
$$

with all “per-key” terms in the denominator and numerator pre-aggregated, resulting in O(nCd) total time (for n tokens, d dimensions, C feature map dimensionality).

Several instantiations exist:

- **Trainable kernelization**: A small feedforward network, such as a Softplus-activated or Gated Linear Unit mapping, parameterizes the feature map φ, with all parameters learned end-to-end. This enables the kernel to adapt to the data distribution and task [2211.04076].
- **Log-normal feature maps**: Derivation of feature map φ via moment-matching to the log-normal statistics of actual self-attention and deterministic exponential parameterization with scaling. This provides unbiased estimators and preserves the empirical “concentration” phenomenon [2311.13541].
- **Polynomial (Taylor) attention**: Truncated Taylor expansions of the exponential, as in TaylorShift, allow the full attention to be written as a sum of Hadamard and tensor-power products, with efficient contraction and normalization. For k=2 (quadratic order), the cost is O(nd³), advantageous for long n once n≫d² [2403.02920].

Such methods universally achieve linear or near-linear cost in n, at the expense of an inner feature dimension that must scale with accuracy and model width.

## 3. Block-wise, Structured, and Hybrid Compression

Recent work exploits the block-structured sparsity of real attention matrices and applies this to both causal-masked and symmetric attention. In “Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity,” it is observed that, for entity tracking over long temporal or structured contexts, most attention mass lies in block-diagonal neighborhoods with a weak, compressible cross-block residue.

- The proposed method partitions the sequence into k=n/m blocks (size m), keeps within-block attention exact using a resolvent-style operator, and collapses cross-block dependencies to a k×k reduced system via fixed down-/up-samplers (e.g., strided pick and nearest-neighbor expand).
- All blocks are then solved with (blockwise) forward substitution. By balancing block size, overall complexity is optimized at m∼n^{2/3}, achieving a total cost of $\widetilde{O}(n^{4/3}d)$ ($n^{7/3}$ for d∼n) [2605.22476].

A similar philosophy is embodied by hybrid compression architectures such as SCOUT, which, after a local mixer phase (Mamba or sliding-window), compresses the sequence into O(n/s) checkpoints and restricts sparse attention to self plus these compressed tokens, yielding O(n²/s) attention cost [2509.00935].

## 4. Algorithmic Sparsity, Sampling, and Hashing

Sub-quadratic self-attention can also be realized via algorithmic sparsity, either with learned adaptive graphs or via randomization:

- **SAC (Sparse Adaptive Connection)** constructs learned sparse graphs by sequentially predicting a set of αN edges per layer via an LSTM policy. Attention is only computed over these O(Nk) edges, which can be tuned to trade expressivity for cost [2003.09833].
- **LSH/Bernoulli sampling**: YOSO-Attention replaces the quadratic softmax kernel by a randomized Bernoulli sampling scheme, with success probability set via Locality Sensitive Hashing (LSH) collision probabilities. This allows all n² pairwise interactions to be “sampled at once” in O(n) via a single pass over bucketed hash tables [2111.09714].
- **Approximate Nearest Neighbor Attention (ANNA)**: A locality-sensitive hash-based engine identifies for each query a sublinear set of keys to attend, leveraging multi-table bucketing and distributed SUM/COUNT aggregation in O(N^{1+3ρ}) time for LSH quality ρ<1/3 [2509.09001].
- **$k$NN-Softmax with lazy Gumbel sampling**: Self-attention is reformulated as a softmax expectation, which is then approximated by Gumbel–Max sampling over a union of top-k MIPS candidates and a few randomly sampled outsiders, resulting in Ō(n^{3/2}) total time for k=√n [2411.04013].

These approaches are often further combined with stochastic ablations and blockwise or sliding-window masks to preserve coverage and minimize approximation error.

## 5. Algorithmic and Complexity Lower Bounds

Fundamental limitations on sub-quadratic self-attention have been established under the Strong Exponential Time Hypothesis (SETH):

- **Quadratic lower bound**: Unless SETH fails, no truly sub-quadratic (O(n^{2−ε})) algorithm can compute exact or constant-factor approximate dot-product softmax attention for general inputs, with or without sparsity/temperature control [2209.04881].
- **Polynomial attention barrier**: Subquadratic exact attention is possible if and only if the head-dimension (or rank) is constant or polylogarithmic in n; otherwise, for d=2^{Θ(log* n)} the cost remains n^{2−o(1)} [2505.14840].
- **Truncated Taylor approximation** provides a partial escape: for bounded inner product (∥QK∥≤B), a degree-r Taylor kernel achieves additive error ε in time O(n d^{r}), but this is exponential in r, i.e., superpolynomial for high-precision/large B [2209.04881]. Thus, all efficient, deterministic sub-quadratic algorithms must randomize, restrict the input domain, or tolerate exponential accuracy scaling.

## 6. Comparative Performance and Empirical Observations

Empirical studies of these diverse approaches show:

- Low-rank and kernelized schemes retain >90% of full attention energy with ∼10–25× speedup for modest k, and negligible (<2%) test accuracy loss on language and vision tasks of moderate context.
- Learned structured/graph sparsity approaches achieve per-layer speedups of 2–10×, often matching or exceeding the performance of fixed-pattern sparse (e.g., block or strided) attention of equal edge budget.
- Blockwise and checkpoint/hybrid approaches enable efficient scaling to very long contexts (e.g., 100K+ tokens), with ablations showing critical dependencies on block size, head–property ratio, and segment length.
- Linearized or polynomial/taylorized schemes yield FLOP and memory crossover points for sequence lengths n≫d², making them preferable for high-resolution vision, long document processing, or genomic modeling [2403.02920].
- Methods that exploit input-driven distributions (eigen/covariance selection or adaptive sparsity) achieve better accuracy–efficiency trade-offs than uniform or random sampling.

## 7. Open Challenges and Future Directions

Several open questions and limitations persist:

- Systematic trade-offs between reconstruction error, empirical eigenspectrum decay, and true downstream task accuracy lack closed-form generalization guarantees.
- The dependence of overall error on feature map dimension C, block size, or kernel truncation order can be exponential or polynomial, demanding careful architecture–hyperparameter tuning for each workload.
- Multi-property (multi-head/entity) limitations: blockwise and hybrid methods can exhibit sharp performance collapse when the number of independent properties/tracks exceeds the head count, suggesting a lower bound on required model width.
- Adaptive, learned hybridizations—such as blockwise, graph, and kernelized mixtures—are the subject of ongoing research, as are theoretical lower bounds for randomized, approximate, or task-regularized settings.

The study and deployment of sub-quadratic self-attention remain central for next-generation large-scale and real-time Transformer models, with choice of method tuned to application, sequence length, and hardware characteristics [2106.08823, 2504.06704, 2505.14840, 2605.22476].

Source: https://www.emergentmind.com/topics/sub-quadratic-self-attention