Quadratic Attention in Transformers

• Quadratic attention is a self-attention mechanism that computes all pairwise token comparisons, resulting in O(N²) complexity and significant memory demands.
• It forms the backbone of standard Transformer models, driving dense global interactions in tasks like high-resolution vision and long-context language modeling.
• Researchers are exploring efficient approximations and higher-order extensions, such as sparse and polynomial attention, to balance performance with computational cost.

Searching arXiv for recent and foundational papers on quadratic attention and related efficient alternatives. Searching specifically for the provided Castling-ViT paper and related efficient/self-attention work. Quadratic attention most commonly denotes the standard self-attention regime in which each token compares with every other token, producing an $N\times N$ similarity matrix and therefore $O(N^2)$ time and memory scaling with sequence length. In transformers this is the mechanism $A=\mathrm{softmax}(QK^\top/\sqrt{d})$ followed by $Y=AV$, and it is the central computational bottleneck in high-resolution vision and long-context language modeling. At the same time, a distinct line of work uses related language such as “quadratic attention,” “Qttention,” or polynomial attention for mechanisms derived from quadratic neurons or higher-order interactions. The term therefore names both a computational regime—dense all-to-all pairwise attention—and a broader family of second-order or polynomial attention constructions (You et al., 2022, Keles et al., 2022, Liao et al., 2022).

1. Standard formulation and the meaning of “quadratic”

In a standard Transformer layer, with input $X\in\mathbb{R}^{N\times d}$, queries, keys, and values are formed as

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$

and vanilla self-attention computes

$$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$

The bottleneck is forming the full similarity matrix $QK^\top$, which costs $O(N^2 d)$ operations and $O(N^2)$ memory for the attention matrix; this is what is commonly called quadratic attention (You et al., 2022).

The same scaling appears in broader complexity analyses of self-attention. A straightforward implementation computes $O(N^2)$0, applies row-wise normalization, and multiplies by $O(N^2)$1, so both time and memory scale quadratically with sequence length. In the language of complexity theory, the basic object that makes attention quadratic is the dense all-pairs interaction pattern over sequence positions (Keles et al., 2022).

In Vision Transformers, a $O(N^2)$2-D image of spatial size $O(N^2)$3 is divided into patches, giving $O(N^2)$4 tokens for patch size $O(N^2)$5. At higher resolutions, $O(N^2)$6 can easily be several thousand, so the $O(N^2)$7 behavior of the attention map dominates the cost and memory footprint. This is particularly problematic for detection, segmentation, feature matching, stereo, and other dense prediction tasks (You et al., 2022, Tang et al., 2022).

2. Why quadratic attention becomes the dominant bottleneck

Quadratic attention scales poorly with input size. In ViTs, doubling the spatial resolution multiplies the number of tokens by $O(N^2)$8, but the attention cost by approximately $O(N^2)$9. The result is poor scalability to high resolution, significant memory pressure from storing an $A=\mathrm{softmax}(QK^\top/\sqrt{d})$0 attention matrix, and latency constraints that become prohibitive in real-time or edge settings (You et al., 2022).

The same issue appears in long-context language modeling. Standard self-attention scales quadratically with sequence length in computation, and training additionally incurs an $A=\mathrm{softmax}(QK^\top/\sqrt{d})$1 activation-memory component from the attention matrix. This creates a fundamental bottleneck in long-context scenarios and motivates sub-quadratic attention, recurrent alternatives, and hybrid architectures (Jafari et al., 31 Aug 2025, Fichtl et al., 6 Oct 2025).

A common misconception is that optimized kernels eliminate this bottleneck. Flash Attention reduces memory and IO by processing attention block-wise and avoiding materialization of the full $A=\mathrm{softmax}(QK^\top/\sqrt{d})$2 matrix in HBM, but its time complexity remains $A=\mathrm{softmax}(QK^\top/\sqrt{d})$3. Likewise, exact sparse attention obtained through entmax still requires quadratic computation to determine the sparsity pattern unless that pattern is predicted or imposed beforehand (Sharma et al., 2024, Treviso et al., 2021).

3. Approximation, compression, and hierarchical replacements

One major response to quadratic attention is to replace token-to-token interactions with compressed summaries or restricted regions. Agglomerative Attention does not construct a pairwise affinity matrix; instead, it assigns tokens to $A=\mathrm{softmax}(QK^\top/\sqrt{d})$4 soft classes, summarizes reference tokens within each class, and lets each query consume a mixture of class summaries. This removes explicit $A=\mathrm{softmax}(QK^\top/\sqrt{d})$5 interactions and yields linear requirements in memory and computation time, while remaining competitive with full attention on language modeling tasks (Spellings, 2019).

A second strategy is coarse-to-fine hierarchical restriction. QuadTree Attention builds token pyramids and computes attention in a coarse-to-fine manner. At each level, the top $A=\mathrm{softmax}(QK^\top/\sqrt{d})$6 patches with the highest attention scores are selected, such that at the next level, attention is evaluated only within the relevant regions corresponding to these top $A=\mathrm{softmax}(QK^\top/\sqrt{d})$7 patches. The resulting complexity is linear rather than quadratic in the number of image tokens, and the reported gains include $A=\mathrm{softmax}(QK^\top/\sqrt{d})$8–$A=\mathrm{softmax}(QK^\top/\sqrt{d})$9 improvement in top-1 accuracy on ImageNet classification, $Y=AV$0–$Y=AV$1 improvement on COCO object detection, $Y=AV$2–$Y=AV$3 improvement on semantic segmentation, and about $Y=AV$4 flops reduction in stereo matching (Tang et al., 2022).

Castling-ViT adopts a different compromise: it trains with both linear-angular attention and masked softmax-based quadratic attention, then switches to only linear-angular attention at inference. Its linear-angular branch is derived from an angular kernel decomposed into linear terms and high-order residuals, while the residual structure is absorbed during training by a depthwise convolution and an auxiliary masked softmax attention whose masks are regularized to gradually become zeros. The result is linear-time attention at inference while preserving quadratic attention’s global and local supervision during training; reported gains include up to a $Y=AV$5 higher accuracy or $Y=AV$6 MACs reduction on ImageNet classification and $Y=AV$7 higher mAP on COCO detection under comparable FLOPs (You et al., 2022).

SCOUT pushes the same idea into long-context language modeling through segment compression. Each token is first enriched by a linear local mixer, Mamba or sliding-window attention, and then attends only to compressed checkpoint tokens that summarize the history. The expensive term becomes $Y=AV$8 rather than $Y=AV$9, where $X\in\mathbb{R}^{N\times d}$0 is the checkpoint interval. At $X\in\mathbb{R}^{N\times d}$1M and $X\in\mathbb{R}^{N\times d}$2B scales, SCOUT with both Mamba and SWA mixers matches full-attention Transformers on language modeling and common-sense reasoning tasks, while achieving higher end-to-end throughput than strong long-sequence baselines (Jafari et al., 31 Aug 2025).

4. Sparse, mask-aware, and distillation-based retention of quadratic behavior

Another research line keeps full softmax attention as the reference behavior but tries to compute only the important part of it. Sparsefinder begins from entmax attention, which yields exact sparse attention weights but still requires quadratic computation to determine the sparsity pattern. It then predicts the sparse attention graph before computing attention, using distance-based, quantization-based, or clustering-based estimators. With balanced buckets, the quantization and clustering variants yield a theoretical $X\in\mathbb{R}^{N\times d}$3 regime, reframing efficient attention as a graph-prediction problem rather than only a kernelization problem (Treviso et al., 2021).

Binary Block Masking makes Flash Attention mask-aware. Instead of processing sparse or partially filled masks as though they were dense, it encodes which blocks of the attention matrix can contain any non-masked entries and dispatches work only on those blocks. Two additional optimizations target contiguous non-zero patterns and extremely sparse masks, and the reported improvement reaches up to $X\in\mathbb{R}^{N\times d}$4 runtime speedup on real masks while preserving exact masked attention semantics (Sharma et al., 2024).

SEA combines linear estimation with sparse softmax execution. It first estimates an attention matrix with linear complexity via kernel-based linear attention, then creates a sparse attention matrix with top-$X\in\mathbb{R}^{N\times d}$5 selection and performs sparse attention under that mask. Unlike many earlier sparse or linear methods, SEA is designed to distill directly from a teacher’s attention matrix and to retain an interpretable attention representation. On Wikitext2, it achieves better perplexity than the quadratic OPT-1.3B baseline while using roughly half the memory of OPT-1.3B (Lee et al., 2023).

Delta Attention focuses on sparse inference rather than pretraining or replacement. It starts from the observation that sparse prefills induce a distributional shift between sparse attention outputs and quadratic attention outputs, causing decoding-time queries to misalign with prefill keys. It then corrects this shift by computing a sparse estimate of the missing dense contribution for a small subset of queries and reusing that delta across nearby queries. When applied on top of sliding-window attention with sink tokens, it recovers $X\in\mathbb{R}^{N\times d}$6 of quadratic attention accuracy on the $X\in\mathbb{R}^{N\times d}$7K RULER benchmark, maintains approximately $X\in\mathbb{R}^{N\times d}$8 sparsity over full quadratic attention, and makes the model $X\in\mathbb{R}^{N\times d}$9 times faster than Flash Attention 2 when processing $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$0M token prefills (Willette et al., 16 May 2025).

5. Quadratic attention as second-order feature interaction

Outside the standard Transformer-complexity sense, “quadratic attention” often refers to attention derived from quadratic neurons. In “Qttention,” the starting point is a quadratic neuron of the conceptual form

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$1

whose factorization yields an attention-like structure. The paper argues that the attention mechanism can be independently derived from a quadratic neuron by factorizing the learned quadratic function in analogue to attention, making the model with quadratic neurons inherently interpretable for bearing fault diagnosis (Liao et al., 2022).

Q-GAT uses the same second-order idea on graphs. Its quadratic neuron can be rewritten through a feature-wise attention vector

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$2

after which standard GAT-style neighbor attention is computed on quadratic features. The resulting model has a dual attention mechanism: feature-level attention through quadratic neurons and node-level attention through graph attention. Experiments on $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$3 and $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$4 suggest that Q-GAT outperforms state-of-the-art baselines in robustness under adversarial perturbations, and the paper attributes that robustness to nonlinear aggregation of quadratic neurons and to the broader feature usage enabled by dual attention (Zhang et al., 2023).

A related but distinct direction replaces softmax with a polynomial function. The polynomial-attention analysis defines

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$5

and studies when a single-layer polynomial attention network can distinguish between two datasets $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$6 and $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$7, one of which contains a significantly larger feature. The paper shows that sufficiently high degree $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$8 can separate the datasets, whereas low degree $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$9 cannot, underscoring the greater effectiveness of high-degree polynomials in amplifying large values and distinguishing between datasets (Song et al., 2023).

These constructions fit a more general taxonomy developed in “The Quarks of Attention.” There, additive activation attention, multiplicative output attention, and multiplicative synaptic attention are identified as the most important mechanisms. The gating mechanisms correspond to multiplicative extensions of the standard model and leverage the power of quadratic activations without incurring their full cost, because they create sparse quadratic terms such as $$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$0 rather than the full combinatorial family of quadratic interactions (Baldi et al., 2022).

6. Complexity theory, higher-order generalizations, and the broader landscape

The strongest negative results say that quadratic attention is not merely an implementation artifact. “On The Computational Complexity of Self-Attention” proves, under the Strong Exponential Time Hypothesis, that the time complexity of self-attention is necessarily quadratic in the input length unless SETH is false. The argument applies to standard softmax attention and several related mechanisms, even when the computation is only approximate. The same paper also shows that dot-product self-attention can be approximated using finite Taylor series in linear-time, at the cost of an exponential dependence on the polynomial order (Keles et al., 2022).

Poly-attention extends this theoretical picture to higher-order mechanisms. It defines a general class of self-attention generalizations that can incorporate arbitrary higher-order tensor computations and arbitrary relationship structures between tokens, with self-attention, tensor attention, and Strassen attention all appearing as special cases. Its central constructive result is a new tree-attention mechanism that can be computed exactly in quadratic time and can perform function composition for any fixed number of functions; prior mechanisms, even for composing two functions, required superquadratic time (Chakrabarti et al., 2 Feb 2026).

Recent architectural work suggests two complementary outlooks. First, self-attention’s pairwise all-to-all structure remains attractive enough that frontier LLMs are still mostly pure Transformer or Transformer++ architectures with quadratic attention, especially at the largest scales (Fichtl et al., 6 Oct 2025). Second, several recent proposals seek to preserve some of the sharpness or “spikiness” of quadratic attention while discarding its sequence-length cost. One example is element-wise attention, which replaces the dot product with the element-wise squared Euclidean distance, approximates the term $$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$1 with a Taylor polynomial, and attains $$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$2 training complexity and $$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$3 inference complexity while achieving performance comparable to self-attention in both causal and non-causal forms (Feng, 10 Jan 2025).

Taken together, the literature presents quadratic attention as both a capability and a constraint. It is powerful because it enables all-to-all interactions, dense retrieval, and in many settings the strongest empirical performance. It is costly because those same all-to-all interactions enforce $$A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)\in\mathbb{R}^{N\times N},\qquad Y=AV.$$4 scaling or, in higher-order variants, even worse asymptotics. Current research therefore oscillates between three objectives: compressing quadratic attention into linear or sub-quadratic surrogates, exploiting sparsity or hierarchy to retain exact behavior where possible, and generalizing pairwise attention into higher-order mechanisms without abandoning the computational tractability that made standard quadratic self-attention usable in the first place (You et al., 2022, Jafari et al., 31 Aug 2025, Chakrabarti et al., 2 Feb 2026).