Sub-quadratic Self-Attention

• Sub-quadratic self-attention is a framework that reduces the O(n²) complexity of Transformers using low-rank approximations, kernel methods, and sparsity techniques.
• It employs approaches like covariance-based estimations, randomized projection, and structured block-wise compression to achieve significant speedups and lower computational costs.
• Empirical results show that leveraging these methods can yield 2–25× runtime improvements while retaining over 90% of attention energy and minimal accuracy drop.

Sub-quadratic self-attention refers to algorithmic strategies and architectural mechanisms that reduce the time and space complexity of the Transformer’s self-attention operation from the classical O(n²) (where n is the sequence length) to o(n²)—typically O(n k), O(n log n), or even O(n)—while retaining as much of the full dot-product attention’s expressiveness as possible. The field, originating from the computational bottlenecks of dense attention in natural language and vision tasks, has developed a spectrum of approaches ranging from low-rank kernel factorizations to adaptive sparsification, randomized projection, and data-driven block-wise evaluation. The goal is to enable the training and deployment of Transformer models on long contexts, high-resolution grids, or resource-constrained hardware, without catastrophic loss in accuracy or model capacity.

1. Low-rank and Covariance-based Approximations

A significant portion of sub-quadratic self-attention methods exploits the empirical observation that attention score matrices in practice, especially over natural language and pre-training corpora, are intrinsically low-rank. The work “Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation” performs a principal component analysis of vectorized attention matrices A (post-softmax) over large samples from BERT-base and finds that the top k≈200 eigenvalues capture over 90% of the total variance even for n=128 (n²=16,384). This low effective rank is even more pronounced in later transformer layers. This eigenspectrum concentration motivates a workflow where, instead of computing all n² attention scores per head:

• A subset P of k≪n² token pairs is selected for exact computation using a greedy, covariance-informed algorithm that aims to minimize the posterior unexplained variance via the Schur complement of Σ_a, the attention score covariance.
• All other (uncomputed) entries are then reconstructed by a Bayes-optimal (Posterior Mean) linear estimator under a Gaussian assumption, yielding for the missing indices

$$\hat{a}_{\bar{P}} = \Sigma_{\bar{P},P}\Sigma_{P,P}^{-1} a_P,$$

where Σ is the empirical covariance.

The total time scales as O(k d + n² k) (for batch reconstruction), but in practical per-row reconstructions, the cost becomes O(n k′ d) for k′≪n, yielding a truly sub-quadratic cost profile. Empirical results show that using only 24 or 32 entries per row reduces FLOPs by 44–25% with minimal drop (<2%) in downstream accuracy, and yields 1.8–2.7× speedups in runtime (Bhojanapalli et al., 2021).

2. Kernelization, Projection, and Feature Map Approaches

Many sub-quadratic algorithms recast the exponential dot-product attention kernel as a finite, sometimes learned, inner product in an auxiliary feature space. This kernelization allows a key computational rearrangement:

$$\exp(q_i^\top k_j) \approx \phi(q_i)^\top \phi(k_j)$$

Enabling the classic attention contraction to:

$$\mathrm{Att}(q_i; K, V) \approx \frac{\phi(q_i)^\top \sum_j \phi(k_j)v_j^\top}{\phi(q_i)^\top \sum_j \phi(k_j)}$$

with all “per-key” terms in the denominator and numerator pre-aggregated, resulting in O(nCd) total time (for n tokens, d dimensions, C feature map dimensionality).

Several instantiations exist:

• Trainable kernelization: A small feedforward network, such as a Softplus-activated or Gated Linear Unit mapping, parameterizes the feature map φ, with all parameters learned end-to-end. This enables the kernel to adapt to the data distribution and task (Yorsh et al., 2022).
• Log-normal feature maps: Derivation of feature map φ via moment-matching to the log-normal statistics of actual self-attention and deterministic exponential parameterization with scaling. This provides unbiased estimators and preserves the empirical “concentration” phenomenon (Nahshan et al., 2023).
• Polynomial (Taylor) attention: Truncated Taylor expansions of the exponential, as in TaylorShift, allow the full attention to be written as a sum of Hadamard and tensor-power products, with efficient contraction and normalization. For k=2 (quadratic order), the cost is O(nd³), advantageous for long n once n≫d² (Nauen et al., 2024).

Such methods universally achieve linear or near-linear cost in n, at the expense of an inner feature dimension that must scale with accuracy and model width.

3. Block-wise, Structured, and Hybrid Compression

Recent work exploits the block-structured sparsity of real attention matrices and applies this to both causal-masked and symmetric attention. In “Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity,” it is observed that, for entity tracking over long temporal or structured contexts, most attention mass lies in block-diagonal neighborhoods with a weak, compressible cross-block residue.

• The proposed method partitions the sequence into k=n/m blocks (size m), keeps within-block attention exact using a resolvent-style operator, and collapses cross-block dependencies to a k×k reduced system via fixed down-/up-samplers (e.g., strided pick and nearest-neighbor expand).
• All blocks are then solved with (blockwise) forward substitution. By balancing block size, overall complexity is optimized at m∼n^{2/3}, achieving a total cost of $\widetilde{O}(n^{4/3}d)$ ($n^{7/3}$ for d∼n) (Zhao et al., 21 May 2026).

A similar philosophy is embodied by hybrid compression architectures such as SCOUT, which, after a local mixer phase (Mamba or sliding-window), compresses the sequence into O(n/s) checkpoints and restricts sparse attention to self plus these compressed tokens, yielding O(n²/s) attention cost (Jafari et al., 31 Aug 2025).

4. Algorithmic Sparsity, Sampling, and Hashing

Sub-quadratic self-attention can also be realized via algorithmic sparsity, either with learned adaptive graphs or via randomization:

• SAC (Sparse Adaptive Connection) constructs learned sparse graphs by sequentially predicting a set of αN edges per layer via an LSTM policy. Attention is only computed over these O(Nk) edges, which can be tuned to trade expressivity for cost (Li et al., 2020).
• LSH/Bernoulli sampling: YOSO-Attention replaces the quadratic softmax kernel by a randomized Bernoulli sampling scheme, with success probability set via Locality Sensitive Hashing (LSH) collision probabilities. This allows all n² pairwise interactions to be “sampled at once” in O(n) via a single pass over bucketed hash tables (Zeng et al., 2021).
• Approximate Nearest Neighbor Attention (ANNA): A locality-sensitive hash-based engine identifies for each query a sublinear set of keys to attend, leveraging multi-table bucketing and distributed SUM/COUNT aggregation in O(N^{1+3ρ}) time for LSH quality ρ<1/3 (Liu et al., 10 Sep 2025).
• $k$NN-Softmax with lazy Gumbel sampling: Self-attention is reformulated as a softmax expectation, which is then approximated by Gumbel–Max sampling over a union of top-k MIPS candidates and a few randomly sampled outsiders, resulting in Ō(n^{3/2}) total time for k=√n (Haris, 2024).

These approaches are often further combined with stochastic ablations and blockwise or sliding-window masks to preserve coverage and minimize approximation error.

5. Algorithmic and Complexity Lower Bounds

Fundamental limitations on sub-quadratic self-attention have been established under the Strong Exponential Time Hypothesis (SETH):

• Quadratic lower bound: Unless SETH fails, no truly sub-quadratic (O(n^{2−ε})) algorithm can compute exact or constant-factor approximate dot-product softmax attention for general inputs, with or without sparsity/temperature control (Keles et al., 2022).
• Polynomial attention barrier: Subquadratic exact attention is possible if and only if the head-dimension (or rank) is constant or polylogarithmic in n; otherwise, for d=2^{Θ(log* n)} the cost remains n^{2−o(1)} (Gupta et al., 20 May 2025).
• Truncated Taylor approximation provides a partial escape: for bounded inner product (∥QK∥≤B), a degree-r Taylor kernel achieves additive error ε in time O(n d^{r}), but this is exponential in r, i.e., superpolynomial for high-precision/large B (Keles et al., 2022). Thus, all efficient, deterministic sub-quadratic algorithms must randomize, restrict the input domain, or tolerate exponential accuracy scaling.

6. Comparative Performance and Empirical Observations

Empirical studies of these diverse approaches show:

• Low-rank and kernelized schemes retain >90% of full attention energy with ∼10–25× speedup for modest k, and negligible (<2%) test accuracy loss on language and vision tasks of moderate context.
• Learned structured/graph sparsity approaches achieve per-layer speedups of 2–10×, often matching or exceeding the performance of fixed-pattern sparse (e.g., block or strided) attention of equal edge budget.
• Blockwise and checkpoint/hybrid approaches enable efficient scaling to very long contexts (e.g., 100K+ tokens), with ablations showing critical dependencies on block size, head–property ratio, and segment length.
• Linearized or polynomial/taylorized schemes yield FLOP and memory crossover points for sequence lengths n≫d², making them preferable for high-resolution vision, long document processing, or genomic modeling (Nauen et al., 2024).
• Methods that exploit input-driven distributions (eigen/covariance selection or adaptive sparsity) achieve better accuracy–efficiency trade-offs than uniform or random sampling.

7. Open Challenges and Future Directions

Several open questions and limitations persist:

• Systematic trade-offs between reconstruction error, empirical eigenspectrum decay, and true downstream task accuracy lack closed-form generalization guarantees.
• The dependence of overall error on feature map dimension C, block size, or kernel truncation order can be exponential or polynomial, demanding careful architecture–hyperparameter tuning for each workload.
• Multi-property (multi-head/entity) limitations: blockwise and hybrid methods can exhibit sharp performance collapse when the number of independent properties/tracks exceeds the head count, suggesting a lower bound on required model width.
• Adaptive, learned hybridizations—such as blockwise, graph, and kernelized mixtures—are the subject of ongoing research, as are theoretical lower bounds for randomized, approximate, or task-regularized settings.

The study and deployment of sub-quadratic self-attention remain central for next-generation large-scale and real-time Transformer models, with choice of method tuned to application, sequence length, and hardware characteristics (Bhojanapalli et al., 2021, Yamada, 9 Apr 2025, Gupta et al., 20 May 2025, Zhao et al., 21 May 2026).