Attention-Space Extrapolation

• Attention-space extrapolation is a method that modifies the attention operator’s scores, biases, and sparsity patterns to extend model capacity beyond training sequence lengths.
• It employs techniques such as ALiBi rescaling, entropy alignment, and operator sketching to preserve the geometry of the attention distribution when processing longer contexts.
• Applications in NLP, video diffusion, and code completion show improved retrieval accuracy and perplexity without requiring full model retraining.

Searching arXiv for papers on attention-space extrapolation and closely related long-context attention methods. arxiv.search query: "attention-space extrapolation long-context attention positional interpolation ALiBi streaming attention" arxiv.search query: "\"attention-space extrapolation\" Transformer length extrapolation attention alignment sparse attention" Attention-space extrapolation denotes a family of methods that extend a model beyond the sequence lengths, resolutions, or operating regimes seen during training by modifying the attention mechanism itself—its scores, biases, kernels, sparsity pattern, computational space, or streaming approximation—rather than relying only on retraining or naive context-window enlargement. In the cited literature, the term covers inference-time rescaling of ALiBi biases, temperature-based attention alignment, entropy-preserving score scaling, probabilistic positional priors, one-pass sublinear-space attention approximation, and analogous constructions in video diffusion, diffusion transformers, code completion, and Krylov-sequence forecasting (Al-Khateeb et al., 2023, Chi et al., 2023, Li et al., 15 Jan 2025, Bianchessi et al., 28 May 2025, Addanki et al., 2023, Zhao et al., 25 Nov 2025, Rajabi et al., 21 May 2026, Ghosh et al., 25 Feb 2026, Qi et al., 12 Jan 2026).

1. Definition and intellectual background

The underlying problem is extrapolation rather than interpolation. Earlier work in NLP distinguished interpolation from extrapolation and argued that extrapolation outside the training space is easier for models that capture global structures rather than merely maximizing local fit (Mitchell et al., 2018). In sequence modeling, a concrete form of extrapolation is evaluation on inputs of length $n_* > n_{\mathcal D}$, where $n_{\mathcal D}$ is the longest training sequence; this formulation motivated explicit location-based attention mechanisms designed to remain stable when sequence length grows beyond the training regime (Dubois et al., 2019).

Within modern Transformer literature, attention-space extrapolation usually refers to preserving meaningful attention behavior when the model is asked to process longer contexts than those seen during training, or to operate over larger spatiotemporal grids than those used during optimization. Some papers formulate the problem as attention flattening or dispersed attention on long inputs; others treat it as positional out-of-distribution behavior, attention score dilution, or failure of the raw query–key interaction to remain expressive outside the training range (Chi et al., 2023, Li et al., 15 Jan 2025, Li et al., 4 Feb 2025, Zheng et al., 2024).

A recurring theme is that extrapolation is not reducible to enlarging the input tensor. The cited work repeatedly treats the attention distribution itself as the object that must be stabilized, sharpened, biased, factorized, sparsified, or approximated. This suggests that “attention-space” is best understood as the collection of representations and operators that intervene between raw query–key similarity and the final weighted aggregation.

2. What is being extrapolated in attention space

The literature modifies several distinct objects inside attention. Some methods reinterpret the pre-softmax score matrix; some preserve distributional properties such as entropy or maximum probability; some replace dense attention by sketches or sparse patterns; others move computation into alternative spaces such as wavelet coefficients or attention-score feature maps (Zheng et al., 2024, Li et al., 15 Jan 2025, Addanki et al., 2023, Zhuang et al., 2022).

Attention-space object	Representative formulation	Papers
Positional prior or bias	$$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$; $b(n)$	(Bianchessi et al., 28 May 2025, Vetcha, 3 Jan 2026)
Score distribution shape	entropy alignment, $P_{\max}$-alignment, InfoScale, CosScale	(Chi et al., 2023, Li et al., 15 Jan 2025)
Score tensor processing	attention logits as $[B,H,T,T]$ feature maps with $1\times k$ convolution	(Zheng et al., 2024)
Operator approximation	$A\approx U_1U_2^\top$ plus sketches $\Phi D^{-1}U_1$, $\Psi U_2$, $n_{\mathcal D}$0	(Addanki et al., 2023)
Sparse connectivity pattern	Local, Sparse, Global attention; dual full/sparse alignment	(Condevaux et al., 2022, Shen et al., 25 Nov 2025)
Alternative computation space	wavelet coefficient space; Krylov-sequence attention history	(Zhuang et al., 2022, Qi et al., 12 Jan 2026)

Two formal reframings are especially influential. In the Bayesian Attention Mechanism, attention weights are treated as a joint distribution over content similarity and positional preference, with positional encoding interpreted as a prior; NoPE becomes a uniform prior over valid past positions, while ALiBi corresponds to a Uniform $n_{\mathcal D}$1 Laplace prior (Bianchessi et al., 28 May 2025). In the unified infinite-length framework, positional methods are written directly in attention-score space as

$n_{\mathcal D}$2

which subsumes RoPE and ALiBi as special cases and motivates Adaptive Positional Encoding with adaptive multiplicative and additive terms (Vetcha, 3 Jan 2026).

A separate line of work treats the attention distribution itself as a quantity with invariants. T5-based alignment methods match either the average maximum attention probability or the entropy between training-length and extrapolated inputs, while InfoScale derives a length-dependent temperature by requiring approximate information-entropy invariance (Chi et al., 2023, Li et al., 15 Jan 2025). This suggests a shift from “extend positions” to “preserve the geometry of the attention distribution.”

3. Long-context LLMs: principal method families

A large class of methods performs inference-time extrapolation by rescaling positional structure while leaving pretrained weights unchanged. For ALiBi, linear position interpolation replaces each head slope by

$n_{\mathcal D}$3

when $n_{\mathcal D}$4, so that longer test-time ranges are mapped back into the training regime. On BTLM-3B-8K and MPT-7B-8K, baseline ALiBi extrapolates only to about 9K–10K tokens, whereas position interpolation keeps perplexity low up to at least 16K; on BTLM-3B-8K summarization at 16K, QMSum improves from ROUGE-1/2/L $n_{\mathcal D}$5 to $n_{\mathcal D}$6, and GovReports from $n_{\mathcal D}$7 to $n_{\mathcal D}$8 (Al-Khateeb et al., 2023).

Another line combines flexible positional embeddings with explicit attention sharpening. T5 relative positional bias is described as unusually flexible, but the model still exhibits dispersed attention as sequence length increases. The proposed remedy is temperature scaling chosen by either maximum-probability alignment or entropy alignment, improving language modeling, retrieval, multi-document QA, and code completion without fine-tuning (Chi et al., 2023). A related but more explicitly theoretical approach derives InfoScale and analyzes CosScale under information entropy invariance; on GAU-$n_{\mathcal D}$9, the combination achieves state-of-the-art behavior up to $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$0 the training length and outperforms seven existing methods, while the paper identifies attention score dilution as a key bottleneck (Li et al., 15 Jan 2025).

Probabilistic and unified-score formulations generalize these ideas. BAM models positional encoding as a learnable prior and introduces a Generalized Gaussian positional prior,

$$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$1

arguing that heavier-tailed or far-focused priors improve long-range retrieval. In the reported experiments, BAM enables retrieval at $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$2 the training context length while maintaining comparable perplexity and introducing only 384 additional parameters in a $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$3M model (Bianchessi et al., 28 May 2025). The unified APE framework instead combines adaptive frequency modulation, a distance-dependent multiplicative damping, and a composite decay bias with linear, logarithmic, and square-root terms; it is analyzed through convergent normalization, entropy boundedness, Long-Distance Correlation Preservation, and Gradient Positional Sensitivity, and is reported to remain stable up to 16,384 tokens on TinyStories and on LongTinyStories with stories up to 32,000 words (Vetcha, 3 Jan 2026).

Training-free RoPE-specific interval manipulation forms another family. GALI greedily reuses pretrained positional intervals chunk by chunk and interpolates attention logits rather than directly relying on interpolated RoPE embeddings, with the stated goal of eliminating logit outliers while preserving local positional information (Li et al., 4 Feb 2025). Mesa-Extrapolation rewires relative positions through weave PE, chunk-based triangular attention, and Stair PE on the final chunk; its reported memory complexity is $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$4, and it is described as the fastest inference method among the compared plug-ins, with reported passkey extrapolation to 60k for LLaMA-3B and extension of Phi-3-mini-128k-instruct to at least 192k under available hardware (Ma et al., 2024).

4. Streaming, sparse, and retrofitted attention operators

Some work treats attention-space extrapolation as an operator-approximation problem rather than a positional-bias problem. For single-layer self-attention with

$$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$5

the one-pass streaming algorithm begins from a polynomial low-rank approximation $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$6 and avoids materializing the $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$7 approximation by storing only three sketches:

$$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$8

It processes the data in one pass, uses sublinear space, yields an $$p_{ij}=p(\mathrm{cont}_{ij}\mid \mathrm{pos}_{ij})\cdot p(\mathrm{pos}_{ij})$$9-sparse output columnwise, and provides an error guarantee

$b(n)$0

with success probability at least $b(n)$1 (Addanki et al., 2023). The same source explicitly contrasts this with FlashAttention, which improves exact attention IO efficiency but still uses the full quadratic computation pattern.

Other approaches replace full attention by structured sparsity that can retrofit pretrained models. LSG Attention substitutes full self-attention with a Local, Sparse, Global pattern, duplicates positional embeddings to the target length, and can adapt checkpoints such as RoBERTa, BART, and LEGAL-BERT with no additional training in the extrapolation setting. In masked language modeling at 4,096 tokens, RoBERTa full attention degrades to $b(n)$2 BPC and $b(n)$3 accuracy, whereas LSG-Norm $b(n)$4 reaches $b(n)$5 BPC and $b(n)$6 accuracy (Condevaux et al., 2022).

SSA addresses a different sparse-attention problem: native sparse methods can exhibit lower attention sparsity than full-attention models because excluded key–value pairs receive neither forward contribution nor backward gradients. SSA therefore alternates full-attention and sparse-attention streams during training and aligns their outputs bidirectionally in feature space. In the reported 1B setting, AttnSparsity sparse/full is $b(n)$7 for SSA, compared with $b(n)$8 for MoBA and $b(n)$9 for FullAttn; models are pretrained on 8K contexts and evaluated up to 32K, with SSA retaining $P_{\max}$0 Needle-in-a-Haystack accuracy at 16K and $P_{\max}$1 at 32K under full-attention inference (Shen et al., 25 Nov 2025).

These results motivate an important distinction: efficient attention, sparse attention, and extrapolatable attention are overlapping but non-identical categories. Some methods primarily reduce complexity; others aim to preserve retrieval geometry, suppress sink behavior, or approximate the dense operator itself.

5. Alternative attention spaces and transformed representations

Several papers argue that extrapolation improves when attention is computed in a different representational space. WavSpA moves attention from token space into wavelet coefficient space by applying a forward wavelet transform, performing attention on the coefficients, and reconstructing with a backward transform. Because wavelets retain both localization in position/time and multiscale frequency structure, the method is reported to outperform Fourier-space attention in 21 out of 25 architecture/task combinations on Long Range Arena; the best adaptive variant, Transformer-AdaWavSpA, reaches average accuracy $P_{\max}$2 versus $P_{\max}$3 for the vanilla Transformer, and the method improves reasoning extrapolation on the LEGO chain-of-reasoning task (Zhuang et al., 2022).

DAPE V2 makes an even more direct claim about score-space processing. It interprets the pre-softmax attention tensor as a feature map of shape $P_{\max}$4 and applies $P_{\max}$5 convolution across neighboring attention scores and across heads before softmax. In this formulation, additive relative positional methods appear as

$P_{\max}$6

and DAPE becomes a learned transformation of the score tensor rather than merely a new positional encoding. On Arxiv with training length 128 and evaluation length 8192, Kerple has perplexity $P_{\max}$7, DAPE-Kerple $P_{\max}$8 improves to $P_{\max}$9, and DAPE$[B,H,T,T]$0-Kerple to $[B,H,T,T]$1; on Books3, the corresponding numbers are $[B,H,T,T]$2, $[B,H,T,T]$3, and $[B,H,T,T]$4 (Zheng et al., 2024).

Attention-based extrapolation also appears outside standard token attention. In Krylov-space forecasting, a decoder-only transformer autoregressively predicts future Lanczos-coefficient increments $[B,H,T,T]$5 from short prefixes and reconstructs physical observables from the predicted sequence. The model is reported to reduce RMSE by about an order of magnitude relative to asymptotic fitting in the quantum test case, improve long-time reconstruction of autocorrelation and Krylov complexity by orders of magnitude, and transfer from system size $[B,H,T,T]$6 to $[B,H,T,T]$7 without retraining (Qi et al., 12 Jan 2026).

Historically, an earlier precursor appears in location attention for recurrent seq2seq models. There, the attention distribution is built from a Gaussian over relative positions $[B,H,T,T]$8, with a mix attender combining location and content attention. The reported results on Lookup Table variants show markedly better extrapolation to longer sequences than standard content-based mechanisms, although the paper also identifies an unresolved <eos> failure mode (Dubois et al., 2019).

6. Cross-domain generalizations, evaluation practice, and open issues

Attention-space extrapolation is not confined to text. In video diffusion transformers, UltraViCo identifies two long-video failure modes—periodic content repetition and universal quality degradation—and attributes both to attention dispersion, meaning that tokens beyond the training window dilute learned attention patterns. The proposed training-free fix suppresses attention for out-of-window tokens with a constant decay factor and, for repetition-prone models, further downscales risky harmonic positions. The reported result is an extension of practical extrapolation from about $[B,H,T,T]$9 to $1\times k$0, with Dynamic Degree improving by $1\times k$1 and Imaging Quality by $1\times k$2 over the previous best method at $1\times k$3 extrapolation on Hunyuan Video (Zhao et al., 25 Nov 2025).

In diffusion transformers for image synthesis, SEGA treats resolution extrapolation as a frequency-aware attention problem. Rather than uniformly scaling all RoPE components, it computes per-dimension scaling from the current latent’s spectral-energy structure at each denoising step. On Flux and Qwen, the method is evaluated at $1\times k$4, $1\times k$5, $1\times k$6, $1\times k$7, and more extreme settings such as $1\times k$8 and $1\times k$9; the paper reports consistent improvements in semantic alignment, image quality, structural coherence, and fine-detail fidelity over state-of-the-art training-free baselines (Rajabi et al., 21 May 2026).

A related but distinct application is field-of-view extrapolation. FoV-Net uses attention-based feature aggregation and gated self-attention to fuse geometrically propagated evidence from narrow-FoV video into a wider-FoV canvas, while also predicting a pixelwise uncertainty map. The reported evidence on KITTI and Cityscapes indicates improved LPIPS, FID, and FVD relative to the compared baselines, with attention and uncertainty used jointly to decide what evidence to trust and where hallucination is necessary (Ma et al., 2022).

Evaluation results in long code reinforce a common caution: efficient attention is not identical to extrapolative positional fidelity. In the reported zero-shot comparison for long code completion, PagedAttention often attains the best Exact Match, whereas ReRoPE is consistently strongest on Edit Similarity, suggesting that efficient KV management and positional extrapolation improve different aspects of long-range code behavior (Ghosh et al., 25 Feb 2026). A plausible implication is that hybrid designs combining positional extrapolation with efficient attention may be necessary in domains where structural validity depends on very long-range dependencies.

Several misconceptions are addressed explicitly by the literature. First, length extrapolation is not only a positional-encoding problem: DAPE V2, T5 attention alignment, UltraViCo, and the streaming sketching algorithm all intervene directly in score space, distribution shape, or operator approximation rather than merely redefining positions (Zheng et al., 2024, Chi et al., 2023, Zhao et al., 25 Nov 2025, Addanki et al., 2023). Second, perplexity is not a sufficient proxy for long-context use: BAM emphasizes long-range retrieval, ALiBi plus position interpolation reports that fine-grained line retrieval remains harder than topic retrieval, and code studies separate Exact Match from Edit Similarity (Bianchessi et al., 28 May 2025, Al-Khateeb et al., 2023, Ghosh et al., 25 Feb 2026). Third, global structure remains central: older extrapolation work in NLP and newer long-context attention papers alike support the view that robust extrapolation requires structural constraints that survive movement outside the training manifold, rather than merely smoother local fit (Mitchell et al., 2018).

Across these strands, attention-space extrapolation emerges as a general research program: preserve or redesign the internal geometry of attention so that longer contexts, larger grids, or longer sequences remain computationally tractable and semantically usable. The diversity of successful interventions—priors, entropy control, sparse/full alignment, score-tensor processing, sketching, transformed bases, and modality-specific concentration rules—indicates that no single mechanism has become canonical. What the literature does converge on is narrower: extrapolation succeeds when the attention operator, not just the input length, is kept within a regime the model can still exploit.