Positional Extrapolation and Interpolation

• Positional Extrapolation and Interpolation are techniques to estimate function outputs beyond and within observed data, crucial in numerical analysis and Transformer design.
• They integrate classical methods like Taylor expansion with modern kernel and Bayesian approaches to ensure reliable in-distribution and out-of-distribution predictions.
• These methods underpin advanced sequence modeling in large language models by improving error bounds, scalability, and context adaptability across varying input lengths.

Positional extrapolation and interpolation are fundamental concepts in numerical analysis, classical signal processing, and, most recently, the design and evaluation of positional encoding schemes in modern neural architectures, especially Transformers. Extrapolation refers to predicting function values or model behavior outside the range of observed positions, while interpolation restricts predictions to within, or as continuous re-parametrization of, the known positional domain. These concepts are now central to sequence modeling and LLM research, as context length limits and positional out-of-distribution (OOD) issues drive the need for effective methods for both robust interpolation and reliable extrapolation.

1. Formal Definitions of Positional Extrapolation and Interpolation

• Positional Interpolation: Given a sequence or function specified on a domain $\Omega$ (e.g., positions $p \in [0, L)$), interpolation constructs estimates at unsampled positions within or smoothly mapped to $\Omega$ based on observed data. In the Transformer context, position interpolation refers to remapping longer inference sequences $[0, L')$ into the training range $[0,L)$ by a continuous, invertible function $f_{\text{interp}}(p)$, such that the positional representation remains in-distribution (Zhao et al., 2023, Chen et al., 2023).
• Positional Extrapolation: Extrapolation extends the model or function to positions $p > L$, outside the range of observed or trained positions. Reliable positional extrapolation requires that the model's attention or function estimation does not degrade catastrophically when faced with OOD position values (Zhao et al., 2023, Al-Khateeb et al., 2023).

Transformers, signal processing, and classical regression all instantiate these concepts, but the mechanisms, guarantees, and empirical outcomes differ by domain.

2. Classical and Theoretical Approaches

Classical Taylor-Based Interpolation and Extrapolation

Analytic interpolation and extrapolation are classically unified by Taylor or polynomial expansion techniques. Given $N=2\alpha+1$ equispaced points $(x_k, y_k)$, the unique polynomial of degree $N-1$ matching these points is

$$f^{\rm UIE}(x) = \sum_{n=0}^{N-1} \frac{D_n}{n!}(x - x_0)^n\,,$$

with $x_0$ the central point and $D$ the solution to the Vandermonde system enforcing exact agreement at the observed data points (Shukurov, 2020). Interpolation is guaranteed at the data points within $[x_{-\alpha}, x_{\alpha}]$; extrapolation continues this polynomial beyond the sampled interval. Error grows rapidly outside the interval, scaling as $|x-x_0|^N/N!$.

Anchor-Based Feasibility and Projection Framework

A general model-agnostic framework recasts extrapolation as a feasibility problem: for domains $\Omega$ (train/interpolation) and $\Xi$ (test/extrapolation), one constructs "anchor" functions $a$ with certificates $\delta$ ensuring $\|f-a\|_\Xi \leq \delta$, and defines feasible sets $S = \cap S(a_i, \delta_i)$ containing the true function on $\Xi$. Projecting a baseline $g$ onto $S$ produces a corrected extrapolation $h$ with guaranteed non-increasing $\Xi$-error, and proven upper/lower improvement bounds (Hay et al., 10 Mar 2026). Certification relies on spectral, classical, and inner-domain stability constants, and can be probabilistic or deterministic.

Kernel and Bandlimited Reconstruction

For multidimensional or bandlimited signals, extrapolation or interpolation proceeds by alternating region-limiting (masking) and bandlimiting (Fourier projection) operators. Iterative correction and projection converges strongly (firm nonexpansiveness), with numerical stability and Tikhonov regularization parameterizing convergence and noise resistance (Frankenbach et al., 2020).

3. Positional Encoding for Interpolation and Extrapolation in Neural Models

Absolute and Relative Positional Encodings

• Absolute Positional Encodings (APEs): Classical schemes (e.g., sinusoidal [Vaswani et al.], learnable embeddings) are additive to token embeddings and poorly extrapolate, as positions $p > L$ yield OOD behavior; variance augmentation methods (SHAPE, CAPE) aim for some shift-invariance (Zhao et al., 2023).
• Relative Positional Encodings (RPEs): Shift-invariant, often using a bias $p_{\text{rel}}(j-i)$ or rotations (RoPE). RPEs such as ALiBi (linear bias), Kerple (kernelized log/power bias), and T5-style bucketing dramatically improve extrapolation by encoding only differences; their inductive bias generalizes to unseen positional ranges (Zhao et al., 2023, Chi et al., 2022, Chi et al., 2023).

Kernelized/Bayesian and Multiple-Kernel Approaches

• Kernelized RPEs: Kerple introduces shift-invariant conditionally positive definite (CPD) kernels (e.g., log, power) as biases, enabling theoretically principled and empirically robust length extrapolation through slow decay at long distances (Chi et al., 2022).
• Bayesian Attention Mechanism (BAM): Formulates self-attention as a product of content and an explicit positional prior $p(\tau)$; special cases recover NoPE and ALiBi, while a Generalized Gaussian prior with learned shape $\beta$ allows much slower decay or even "retrieval heads" that attend only to distant tokens. BAM yields >80% retrieval accuracy at 500$\times$ training length and provides error guarantees (Bianchessi et al., 28 May 2025).
• Multiple-Kernel Learning (MEP): Composes post-softmax biases from exponential, Gaussian, and polynomial-log kernels with fixed or learned mixture weights to achieve slower, smoother decay and improved extrapolation; both parameter-free and parameterized variants outperform prior methods on perplexity at all tested lengths (Gao, 2024).

Bilevel and Data-Adaptive Encodings

• BiPE: Explicitly disentangles intra-segment (absolute, bounded-range) and inter-segment (relative, unbounded range) positional encoding, matching the hierarchical structure in natural language and providing both theoretical efficiency and superior extrapolation (e.g., perplexity improvements and improved length generalization) (He et al., 2024).
• DAPE: Data-Adaptive Positional Encoding introduces a learned, context-dependent bias via a per-attention MLP correction applied to both the semantic similarity $QK^T$ and the fixed relative bias. This enables the model to learn both local and non-local ("anti-local") patterns, yielding substantial perplexity reductions in both interpolation and extrapolation regimes, especially at much longer evaluation lengths than those used in training (Zheng et al., 2024).

4. Position Interpolation, Scaling, and Plug-in Methods

Position interpolation methods remap positions at test time to the original training range to avoid attention disaster from extrapolation, especially in rotary and bias-based PE schemes:

Method	Mechanism	Key Results	Reference
Position Interpolation (PI)	$p' = \alpha p,\, \alpha=L/L'$	$\sim600\times$ lower error bound than extrapolation; stable up to $32$K tokens	(Chen et al., 2023, Al-Khateeb et al., 2023)
RoPE+Interpolation	Down-scaling indices in RoPE	Maintains in-dist PPL, enables up to $2\times$ window extension	(Chen et al., 2023, Al-Khateeb et al., 2023)
Greedy Attention Logit Interpolation (GALI)	Greedy chunking, logit interpolation between trained intervals	Stable up to $32$K with no tuning; outperforms chunk-based and scaling approaches	(Li et al., 4 Feb 2025)
Position Interpolation for ALiBi	Directly scales slope: $m' = m \cdot (L/L')$	Doubles the reliable context for ALiBi	(Al-Khateeb et al., 2023)
NTK-aware, NTK-by-Parts, YaRN	Scaling, partial scaling, softmax temperature	Combined with interpolation for further resolution gains	(Zhao et al., 2023)
T5 Bucket Interpolation	Log-bucket indexing	Robust to zero-shot lengths $\gg L$; limited by bucket granularity	(Chi et al., 2023)

PI and direct scaling methods achieve near in-distribution performance at up to $2\times$ (RoPE/ALiBi) or up to $32$K tokens with fine-tuning (RoPE-based LLaMA models) with only minimal degradation on short-input tasks (Chen et al., 2023, Al-Khateeb et al., 2023).

5. Empirical and Theoretical Comparisons

Empirical studies across language modeling, retrieval, summarization, arithmetic reasoning, and code tasks support the following conclusions:

• RPEs with slowly decaying kernels (log, heavy-tailed power) or multiple-kernel mixtures maintain high effective attention and low perplexity in extrapolation, significantly outperforming APEs and naive RoPE at long contexts (Chi et al., 2022, Gao, 2024).
• Interpolation strictly improves attention-score deviation bounds: PI's upper error bound is $\sim600\times$ smaller than naive extrapolation in rotary-based encodings (Chen et al., 2023).
• Segment-level encoding (BiPE) provides sharp increases in long-context performance with no loss on short tasks, and matches the hierarchical structure of real data, yielding both theoretical and practical efficiency (He et al., 2024).
• Context-adaptive and learnable-bias methods (DAPE, BAM) further improve over static or monotonic approaches, enabling both local and anti-local attention patterns and dramatically boost scalability and generalization (Zheng et al., 2024, Bianchessi et al., 28 May 2025).
• Plug-in and training-free interpolation strategies (GALI, PI, position-bucket interpolation) permit efficient, backward-compatible extension to huge context windows, routinely outperforming both naive length extrapolation and chunking approaches (Chen et al., 2023, Li et al., 4 Feb 2025).

6. Outstanding Challenges and Future Directions

Three major technical challenges and active research areas arise:

• Theoretical explanation of optimal attention decay and positional encoding shape: While slow logarithmic or generalized Gaussian priors empirically outperform sharp linear or bounded Gaussian decay, a unified theory for the relationship between positional decay, model expressivity, and context-awareness remains an open problem (Zhao et al., 2023, Bianchessi et al., 28 May 2025).
• Automatic and data-adaptive tuning of interpolation functions and kernel parameters: Data-adaptive PEs such as DAPE suggest learnable, input-dependent correction is superior to fixed priors; future work may integrate continuous, neural, or probabilistic mechanisms into baseline PEs (Zheng et al., 2024).
• Certified extrapolation guarantees and spectral risk measurement: Spectral or probabilistic condition numbers can be used to furnish certified error bounds in the extrapolation regime, providing a new axis for model selection and risk assessment beyond perplexity/accuracy (Hay et al., 10 Mar 2026).

Novel combinations—hierarchical encodings (BiPE), content-based adaptation (DAPE), plug-in scaling (PI, GALI), and probabilistic priors (BAM)—are now central tools for robust interrogation, evaluation, and scaling of positional encodings in both classical and modern machine learning models. The field continues to deepen the connections between analytic signal processing, theoretical learning guarantees, and scalable practical architectures for extrapolation in high-capacity sequence models.