Attention Scaling: Methods and Trends

• Attention scaling is a family of techniques that reweight or structurally modify attention mechanisms to better allocate computation and capacity as models and contexts grow.
• It includes methods such as head/channel rescaling, inverse-temperature and entropy control, structured sparsity, and disaggregated KV serving to address various scaling challenges.
• By dynamically adjusting attention distributions and system architectures, these approaches enhance long-context retrieval, efficiency, and overall model performance.

Searching arXiv for papers on “attention scaling” and closely related formulations. Attention scaling denotes a family of techniques and scaling laws that modify how attention allocates computation, capacity, or probability mass as model size, sequence length, task structure, or serving demand grows. In contemporary literature, the term does not refer to a single intervention. It includes learned rescaling of heads and channels for long-context retrieval (Lee et al., 25 Jan 2025), temperature and entropy control of attention distributions (Hayase et al., 12 May 2026), entropy-regularized stabilization of large transformer critics in reinforcement learning (Dong et al., 1 Feb 2026), structural redesigns that scale attention along sequence, feature, or state axes (Cai et al., 2023, Xie et al., 17 Mar 2026, Gelada et al., 6 Jul 2025, Wen et al., 1 Feb 2026, Dentamaro, 11 Jul 2025), and system-level disaggregation of core-attention serving from weight computation (Xiang et al., 28 Jun 2026). The resulting field is best understood as a collection of related but non-equivalent answers to one question: how should attention change when context, capacity, and compute scale?

1. Conceptual scope

The phrase “attention scaling” is used across several technical axes rather than one canonical axis. Some papers scale the contribution of existing attention components without changing the forward operator; others scale the sharpness of attention distributions, the dimensionality of latent state, the sparsity pattern over tokens or features, or the serving substrate that stores and computes over KV state.

Axis	Representative mechanism	Representative papers
Internal reweighting	Head/channel multipliers; gradient-component scaling	(Lee et al., 25 Jan 2025, Kim et al., 15 Dec 2025)
Distribution control	Inverse-temperature schedules; entropy regularization	(Hayase et al., 12 May 2026, Dong et al., 1 Feb 2026)
Structural/operator scaling	Structured, sparse, feature-sparse, wavelet, or fast-weight attention	(Cai et al., 2023, Xie et al., 17 Mar 2026, Wen et al., 1 Feb 2026, Gelada et al., 6 Jul 2025, Dentamaro, 11 Jul 2025, Zhang et al., 30 Dec 2025)
Capacity growth inside Q/K/V	Nonlinear expandable Q/K/V projections	(Zhao et al., 21 Apr 2026)
Serving elasticity	Disaggregated attention nodes for dynamic KV demand	(Xiang et al., 28 Jun 2026)
Broader empirical scaling laws	Model-size effects on attention patterns; collective-attention dispersion	(Gao et al., 2023, Okamura, 18 Jan 2026)

This diversity matters because results are not interchangeable. A method that scales attention by rescaling head outputs answers a different problem from a method that chooses a critical inverse-temperature law or from a serving system that elastically scales KV capacity. A plausible implication is that “attention scaling” is best treated as an umbrella term for capacity allocation in and around attention, rather than as a single algorithmic primitive.

2. Reweighting existing attention mechanisms

A direct form of attention scaling leaves the standard multi-head attention architecture intact and changes only the relative contribution of its components. “SEAL: Scaling to Emphasize Attention for Long-context retrieval” identifies specific heads and channels whose ablation is positively or negatively correlated with long-context retrieval quality, then learns multiplicative scales on those outputs (Lee et al., 25 Jan 2025). For head-wise scaling,

$\tilde{H}^{(\ell,h)} = s_{\ell,h} H^{(\ell,h)},$

and for channel-wise scaling,

$$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$

The scaling is applied post-softmax to head outputs, not to logits or positional encodings, and can be folded into adjacent projection weights offline so inference cost is unchanged (Lee et al., 25 Jan 2025).

SEAL is motivated by a concrete long-context failure mode: quality degradation within the sequence limit. On LongChat-7B-32K line retrieval, baseline accuracy falls from 0.98 at 9K tokens to 0.32 at 31K tokens, and head-wise pruning reveals roughly ±20 percentage-point head effects (Lee et al., 25 Jan 2025). With only 1.0K tunable parameters in SEAL-H or 131.1K in SEAL-C for 7B models, SEAL raises LongChat-7B-v1.5-32K line-retrieval accuracy at 31K from 0.32 to 0.80 and 0.88 respectively, while MMLU remains essentially unchanged at 42.53 for baseline, 42.34 for SEAL-H, and 42.17 for SEAL-C (Lee et al., 25 Jan 2025). The same paper shows that attention scaling is orthogonal to training-free context extension: on Llama-2-7B-Chat, NTK scaling alone yields 0.00 accuracy at 16K on LongEval, whereas NTK + SEAL-C reaches 0.88 (Lee et al., 25 Jan 2025).

A distinct backward-pass version appears in “Scaling Bidirectional Spans and Span Violations in Attention Mechanism” (Kim et al., 15 Dec 2025). That work keeps the canonical forward attention unchanged but decomposes gradients into 0th-order bidirectional parallel spans and higher-order span violations, then scales the components as

$$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$

On WikiText-2, the pure 0th-order setting [1000] yields a 0.56% reduction in validation loss, supporting the paper’s claim that the standard attention gradient is suboptimal (Kim et al., 15 Dec 2025).

A more structural variant replaces linear Q/K/V maps themselves. Nexusformer introduces a Nexus-Rank layer,

$D \to M \to A \to D,\qquad D < M < A,$

with dual GeLU activations in progressively higher-dimensional spaces, and expands it through zero-initialized blocks so pretrained behavior is preserved exactly at the growth step (Zhao et al., 21 Apr 2026). In progressive scaling from 240M to 440M, Nexusformer matches Tokenformer’s perplexity using up to 41.5% less training compute, and its growth dynamics admit an empirical geometric scaling law relating perplexity to a distributional alignment metric $R$ (Zhao et al., 21 Apr 2026).

3. Temperature, entropy, and critical concentration

Another major meaning of attention scaling concerns the sharpness or entropy of attention distributions. “A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention” studies a single attention row with logits $z_{n,1},\dots,z_{n,n}$ and inverse temperature $\beta$,

$$p_{n,j}(\beta)=\frac{e^{\beta z_{n,j}}}{\sum_{\ell=1}^n e^{\beta z_{n,\ell}}}.$$

The paper defines a gap-counting function

$N_n(t)=\#\{j: z_n^*-z_{n,j}\le t\},$

and the upper-tail accumulation scale

$\Lambda_n=\sup_{t>0}\frac{\log N_n(t)}{t}.$

Its main theorem states that $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$0 is the critical top-two–entropy scale: if $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$1, top-two collapse occurs; if $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$2, attention entropy collapses (Hayase et al., 12 May 2026). This unifies earlier, apparently conflicting laws such as $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$3, $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$4, and $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$5 as arising from different competitor-growth structures $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$6 rather than from a universal exponent (Hayase et al., 12 May 2026). In practical diagnostics, Qwen-7B-Chat above its training window yields an empirically optimal $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$7, while GPT-124M with learned per-position $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$8 gives $$\tilde{H}^{(\ell,h)} = H^{(\ell,h)} \odot \boldsymbol{s}_{\ell,h}.$$9, closely matching the estimated $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$0 in each case (Hayase et al., 12 May 2026).

A related but task-specific use of entropy control appears in reinforcement learning. “TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse” identifies a failure mode in large transformer critics: as parameter count grows from roughly 0.4M to 26M, average attention entropy decreases monotonically, performance on a cube-double task set falls from about 46% to about 6%, and Q-value landscapes become highly non-smooth (Dong et al., 1 Feb 2026). TQL introduces per-layer temperatures $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$1 and optimizes

$$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$2

together with the critic loss (Dong et al., 1 Feb 2026). The method yields up to a 43% performance improvement when scaling from the smallest to the largest network size, whereas baseline critics degrade by about 10.6% on average (Dong et al., 1 Feb 2026). In this usage, attention scaling is not primarily about context length; it is about maintaining a non-collapsed entropy regime as capacity increases.

4. Structural scaling of the attention operator

A third family changes the operator itself so that attention scales more favorably with sequence length, feature dimension, or fast-weight capacity. In in-context learning, SAICL replaces full encoder self-attention with a structured pattern in which each demonstration attends only to itself and the test segment, while the test attends globally (Cai et al., 2023). This removes direct demo–demo dependencies, makes the architecture permutation-invariant over demonstrations, and changes encoder complexity from $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$3 to $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$4 in the number of demonstrations $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$5. SAICL achieves comparable or better performance than full attention, up to 3.4x inference speed-up, and scales to hundreds of demonstrations with continued gains (Cai et al., 2023).

Sequence-axis sparsity also appears in LongCat ZigZag Attention. LoZA converts about 50% of MLA layers to Streaming Sparse Attention with block size $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$6, $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$7 sink block, and $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$8 local blocks, so each token attends to at most $$\frac{\partial L}{\partial Q}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial Q}_{i\text{th}},\qquad \frac{\partial L}{\partial K}^{\text{scaled}}=\sum_{i=0}^3 \alpha_i \frac{\partial L}{\partial K}_{i\text{th}}.$$9 keys (Zhang et al., 30 Dec 2025). The result is more than 50% prefill speed-up and more than 30% decode cost savings at 256k tokens, while enabling LongCat-Flash-Exp to process up to 1 million tokens (Zhang et al., 30 Dec 2025). The paper’s central systems insight is that calibrated layer-level sparsity is far more effective than hand-crafted interleaving, which sharply degrades LongEval without calibration (Zhang et al., 30 Dec 2025).

Feature-axis sparsity is explored in Sparse Feature Attention. SFA keeps the full $D \to M \to A \to D,\qquad D < M < A,$0 token interaction graph but applies row-wise Top-$D \to M \to A \to D,\qquad D < M < A,$1 to queries and keys,

$D \to M \to A \to D,\qquad D < M < A,$2

so attention scores are computed only on overlapping active coordinates (Xie et al., 17 Mar 2026). Under balanced support assumptions, the dominant complexity falls from $D \to M \to A \to D,\qquad D < M < A,$3 to

$D \to M \to A \to D,\qquad D < M < A,$4

and FlashSFA implements this without materializing dense score matrices (Xie et al., 17 Mar 2026). Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to 2.5x and reducing FLOPs and KV-cache by nearly 50% (Xie et al., 17 Mar 2026).

MiTA Attention formulates efficient attention as fast-weight scaling. It interprets single-head attention as an $D \to M \to A \to D,\qquad D < M < A,$5-width fast-weight MLP, then compresses it to a shared expert using $D \to M \to A \to D,\qquad D < M < A,$6 landmark queries and augments it with deformable experts formed by top-$D \to M \to A \to D,\qquad D < M < A,$7 activated key–value pairs for each landmark (Wen et al., 1 Feb 2026). This produces a compress-and-route cost of roughly $D \to M \to A \to D,\qquad D < M < A,$8 rather than $D \to M \to A \to D,\qquad D < M < A,$9 for fixed $R$0, and the paper positions MiTA as a unifying synthesis of compression-based and routing-based efficient attention (Wen et al., 1 Feb 2026).

Power attention changes the state-size scaling of linear attention itself. It replaces the exponential kernel with a polynomial kernel $R$1, yielding a linear-time mechanism whose state size can be adjusted independently of parameters through symmetric or tiled tensor-power feature maps (Gelada et al., 6 Jul 2025). The central claim is not merely that quadratic softmax attention is too expensive, but that many existing sub-quadratic methods are “too inexpensive” in state FLOPs relative to weight FLOPs; power attention is proposed to rebalance this ratio for long-context in-context learning (Gelada et al., 6 Jul 2025).

WERSA combines Haar wavelets, content-adaptive scale filtering, and random spectral features to obtain linear $R$2 attention (Dentamaro, 11 Jul 2025). On ArXiv classification it improves accuracy over vanilla attention from 85.0% to 86.2% while reducing training time from 1554 s to 296 s and FLOPs from 98.4G to 26.2G; on ArXiv-128k it remains viable where vanilla attention and FlashAttention-2 run out of memory, reaching 79.1% accuracy and 0.979 AUC (Dentamaro, 11 Jul 2025).

Outside transformers, FastBoost’s DSPA uses dynamic scaling of channel and spatial attention pathways together with training-phase residual scaling,

$R$3

with $R$4 rising from about 0.5 to about 0.72 during training (Yuan, 2 Nov 2025). On CIFAR-10, the dynamic mechanism raises a 3-layer MBConv baseline from 89.14% to 91.24%, versus 90.19% for static dual attention (Yuan, 2 Nov 2025). This usage broadens attention scaling beyond transformers to training-phase modulation of lightweight attention modules.

5. Systems, deployment, and physical limits

At serving time, attention scaling can mean scaling KV capacity and attention bandwidth rather than changing the attention algorithm. KernelFlume disaggregates decode into stable weight nodes, which execute projection and FFN kernels, and weightless attention nodes, which store token-range KV partitions and execute core-attention over those shards (Xiang et al., 28 Jun 2026). A routing table maps token ranges to attention-node endpoints and is updated only at token boundaries; host-visible graph signals drive UCX communication outside the captured CUDA Graph (Xiang et al., 28 Jun 2026). On real GPU testbeds serving Llama-3.1-8B, KernelFlume sustains flat p99 TPOTs of about 74 ms on A6000 and about 34 ms on H100 while reducing cost per million output tokens by up to 32% and 61%, respectively, relative to full-instance elastic scaling with ServerlessLLM (Xiang et al., 28 Jun 2026). In simulation at larger model scale, the projected reduction is 56–66%, widening to 80–85% with heterogeneous attention-node hardware and persisting into the million-token context regime (Xiang et al., 28 Jun 2026).

A complementary systems-scale example is MiniMax-M1, which combines a hybrid MoE with lightning attention and softmax attention blocks so that the model natively supports a 1 million-token context and 40K/80K thinking budgets (MiniMax et al., 16 Jun 2025). The base architecture contains 456 billion parameters with 45.9 billion activated per token, and the combination of hybrid-attention and CISPO enables full RL training on 512 H800 GPUs in only three weeks at a rental cost of $534,700 (MiniMax et al., 16 Jun 2025). This is not a new attention law, but it is a concrete demonstration that hybrid attention can make large-scale test-time compute economically tractable.

At the opposite end of abstraction, “Circuit Complexity From Physical Constraints: Scaling Limitations of Attention” argues that dense attention faces a principled physical scaling limit (Prada et al., 23 Sep 2025). In its $R$5 framework, physically admissible circuit growth requires $R$6, corresponding to an ideal $R$7 regime, whereas softmax and average-hard attention transformers lie in $R$8 and are not in $R$9 (Prada et al., 23 Sep 2025). The paper’s broader conclusion is that attention mechanisms with $z_{n,1},\dots,z_{n,n}$0 runtime cannot scale to accommodate the entropy of increasingly complex datasets (Prada et al., 23 Sep 2025).

6. Broader empirical meanings and outlook

The term also appears in empirical studies of how attention changes as models scale. In “Roles of Scaling and Instruction Tuning in Language Perception,” increasing model size from GPT-2 Large to LLaMA-65B raises normalized human-attention resemblance from 34.99% to 64.05% for native speakers and from 40.03% to 70.07% for non-native speakers, while scaling reduces trivial pattern reliance and instruction tuning does not (Gao et al., 2023). Here, attention scaling refers neither to context length nor to efficient kernels, but to changes in the qualitative structure of learned attention as parameter count grows.

Beyond machine learning, “Logarithmic scaling and stochastic criticality in collective attention” studies Wikipedia page views and finds

$z_{n,1},\dots,z_{n,n}$1

with the logarithmic regime occurring at the critical boundary $z_{n,1},\dots,z_{n,n}$2 between power-law growth and saturation (Okamura, 18 Jan 2026). This usage concerns collective human attention rather than neural attention, but it reinforces a recurrent theme: scaling laws for attention often depend on balancing growth and concentration.

A common misconception is that attention scaling is synonymous with long-context extension. The literature instead shows several non-equivalent interventions. Some methods reweight heads or channels; some control entropy or inverse temperature; some sparsify token interactions, feature dimensions, or KV service topology; some enlarge the nonlinear manifold that produces Q/K/V; and some study how attention patterns or collective-attention variance evolve under scaling. Taken together, these works suggest that future progress will depend less on a universal “scaled attention” recipe than on matching the scaling axis to the bottleneck: retrieval reliability inside a fixed window (Lee et al., 25 Jan 2025), entropy collapse in oversized critics (Dong et al., 1 Feb 2026), fast-weight width at extreme sequence lengths (Wen et al., 1 Feb 2026), state–weight imbalance in long-context learning (Gelada et al., 6 Jul 2025), or KV elasticity in agentic serving (Xiang et al., 28 Jun 2026).