---
title: Positional Extrapolation and Interpolation
url: https://www.emergentmind.com/topics/positional-extrapolation-and-interpolation
type: topic
---

# Positional Extrapolation and Interpolation

Positional extrapolation and interpolation are fundamental concepts in numerical analysis, classical signal processing, and, most recently, the design and evaluation of positional encoding schemes in modern neural architectures, especially Transformers. Extrapolation refers to predicting function values or model behavior outside the range of observed positions, while interpolation restricts predictions to within, or as continuous re-parametrization of, the known positional domain. These concepts are now central to sequence modeling and large language model (LLM) research, as context length limits and positional out-of-distribution (OOD) issues drive the need for effective methods for both robust interpolation and reliable extrapolation.

## 1. Formal Definitions of Positional Extrapolation and Interpolation

- **Positional Interpolation:** Given a sequence or function specified on a domain $\Omega$ (e.g., positions $p \in [0, L)$), interpolation constructs estimates at unsampled positions within or smoothly mapped to $\Omega$ based on observed data. In the Transformer context, position interpolation refers to remapping longer inference sequences $[0, L')$ into the training range $[0,L)$ by a continuous, invertible function $f_{\text{interp}}(p)$, such that the positional representation remains in-distribution [2312.17044, 2306.15595].

- **Positional Extrapolation:** Extrapolation extends the model or function to positions $p > L$, outside the range of observed or trained positions. Reliable positional extrapolation requires that the model's attention or function estimation does not degrade catastrophically when faced with OOD position values [2312.17044, 2310.13017].

Transformers, signal processing, and classical regression all instantiate these concepts, but the mechanisms, guarantees, and empirical outcomes differ by domain.

## 2. Classical and Theoretical Approaches

### Classical Taylor-Based Interpolation and Extrapolation

Analytic interpolation and extrapolation are classically unified by Taylor or polynomial expansion techniques. Given $N=2\alpha+1$ equispaced points $(x_k, y_k)$, the unique polynomial of degree $N-1$ matching these points is
$$
f^{\rm UIE}(x) = \sum_{n=0}^{N-1} \frac{D_n}{n!}(x - x_0)^n\,,
$$
with $x_0$ the central point and $D$ the solution to the Vandermonde system enforcing exact agreement at the observed data points [2002.11438]. Interpolation is guaranteed at the data points within $[x_{-\alpha}, x_{\alpha}]$; extrapolation continues this polynomial beyond the sampled interval. Error grows rapidly outside the interval, scaling as $|x-x_0|^N/N!$.

### Anchor-Based Feasibility and Projection Framework

A general model-agnostic framework recasts extrapolation as a feasibility problem: for domains $\Omega$ (train/interpolation) and $\Xi$ (test/extrapolation), one constructs "anchor" functions $a$ with certificates $\delta$ ensuring $\|f-a\|_\Xi \leq \delta$, and defines feasible sets $S = \cap S(a_i, \delta_i)$ containing the true function on $\Xi$. Projecting a baseline $g$ onto $S$ produces a corrected extrapolation $h$ with guaranteed non-increasing $\Xi$-error, and proven upper/lower improvement bounds [2603.09937]. Certification relies on spectral, classical, and inner-domain stability constants, and can be probabilistic or deterministic.

### Kernel and Bandlimited Reconstruction

For multidimensional or bandlimited signals, extrapolation or interpolation proceeds by alternating region-limiting (masking) and bandlimiting (Fourier projection) operators. Iterative correction and projection converges strongly (firm nonexpansiveness), with numerical stability and Tikhonov regularization parameterizing convergence and noise resistance [2008.00026].

## 3. Positional Encoding for Interpolation and Extrapolation in Neural Models

### Absolute and Relative Positional Encodings

- **Absolute Positional Encodings (APEs):** Classical schemes (e.g., sinusoidal [Vaswani et al.], learnable embeddings) are additive to token embeddings and poorly extrapolate, as positions $p > L$ yield OOD behavior; variance augmentation methods (SHAPE, CAPE) aim for some shift-invariance [2312.17044].
- **Relative Positional Encodings (RPEs):** Shift-invariant, often using a bias $p_{\text{rel}}(j-i)$ or rotations (RoPE). RPEs such as ALiBi (linear bias), Kerple (kernelized log/power bias), and T5-style bucketing dramatically improve extrapolation by encoding only differences; their inductive bias generalizes to unseen positional ranges [2312.17044, 2205.09921, 2311.00684].

### Kernelized/Bayesian and Multiple-Kernel Approaches

- **Kernelized RPEs:** Kerple introduces shift-invariant conditionally positive definite (CPD) kernels (e.g., log, power) as biases, enabling theoretically principled and empirically robust length extrapolation through slow decay at long distances [2205.09921].
- **Bayesian Attention Mechanism (BAM):** Formulates self-attention as a product of content and an explicit positional prior $p(\tau)$; special cases recover NoPE and ALiBi, while a Generalized Gaussian prior with learned shape $\beta$ allows much slower decay or even "retrieval heads" that attend only to distant tokens. BAM yields >80% retrieval accuracy at 500$\times$ training length and provides error guarantees [2505.22842].
- **Multiple-Kernel Learning (MEP):** Composes post-softmax biases from exponential, Gaussian, and polynomial-log kernels with fixed or learned mixture weights to achieve slower, smoother decay and improved extrapolation; both parameter-free and parameterized variants outperform prior methods on perplexity at all tested lengths [2403.17698].

### Bilevel and Data-Adaptive Encodings

- **BiPE:** Explicitly disentangles intra-segment (absolute, bounded-range) and inter-segment (relative, unbounded range) positional encoding, matching the hierarchical structure in natural language and providing both theoretical efficiency and superior extrapolation (e.g., perplexity improvements and improved length generalization) [2401.16421].
- **DAPE:** Data-Adaptive Positional Encoding introduces a learned, context-dependent bias via a per-attention MLP correction applied to both the semantic similarity $QK^T$ and the fixed relative bias. This enables the model to learn both local and non-local ("anti-local") patterns, yielding substantial perplexity reductions in both interpolation and extrapolation regimes, especially at much longer evaluation lengths than those used in training [2405.14722].

## 4. Position Interpolation, Scaling, and Plug-in Methods

Position interpolation methods remap positions at test time to the original training range to avoid attention disaster from extrapolation, especially in rotary and bias-based PE schemes:

| Method               | Mechanism                             | Key Results                         | Reference         |
|----------------------|---------------------------------------|-------------------------------------|-------------------|
| Position Interpolation (PI) | $p' = \alpha p,\, \alpha=L/L'$  | $\sim600\times$ lower error bound than extrapolation; stable up to $32$K tokens | [2306.15595, 2310.13017] |
| RoPE+Interpolation   | Down-scaling indices in RoPE           | Maintains in-dist PPL, enables up to $2\times$ window extension | [2306.15595, 2310.13017] |
| Greedy Attention Logit Interpolation (GALI)        | Greedy chunking, logit interpolation between trained intervals | Stable up to $32$K with no tuning; outperforms chunk-based and scaling approaches | [2502.02659]  |
| Position Interpolation for ALiBi          | Directly scales slope: $m' = m \cdot (L/L')$ | Doubles the reliable context for ALiBi | [2310.13017]    |
| NTK-aware, NTK-by-Parts, YaRN             | Scaling, partial scaling, softmax temperature | Combined with interpolation for further resolution gains | [2312.17044]   |
| T5 Bucket Interpolation                   | Log-bucket indexing                       | Robust to zero-shot lengths $\gg L$; limited by bucket granularity | [2311.00684]   |

PI and direct scaling methods achieve near in-distribution performance at up to $2\times$ (RoPE/ALiBi) or up to $32$K tokens with fine-tuning (RoPE-based LLaMA models) with only minimal degradation on short-input tasks [2306.15595, 2310.13017].

## 5. Empirical and Theoretical Comparisons

Empirical studies across language modeling, retrieval, summarization, arithmetic reasoning, and code tasks support the following conclusions:

- **RPEs with slowly decaying kernels (log, heavy-tailed power) or multiple-kernel mixtures** maintain high effective attention and low perplexity in extrapolation, significantly outperforming APEs and naive RoPE at long contexts [2205.09921, 2403.17698].
- **Interpolation strictly improves attention-score deviation bounds**: PI's upper error bound is $\sim600\times$ smaller than naive extrapolation in rotary-based encodings [2306.15595].
- **Segment-level encoding (BiPE) provides sharp increases in long-context performance** with no loss on short tasks, and matches the hierarchical structure of real data, yielding both theoretical and practical efficiency [2401.16421].
- **Context-adaptive and learnable-bias methods (DAPE, BAM)** further improve over static or monotonic approaches, enabling both local and anti-local attention patterns and dramatically boost scalability and generalization [2405.14722, 2505.22842].
- **Plug-in and training-free interpolation strategies (GALI, PI, position-bucket interpolation)** permit efficient, backward-compatible extension to huge context windows, routinely outperforming both naive length extrapolation and chunking approaches [2306.15595, 2502.02659].

## 6. Outstanding Challenges and Future Directions

Three major technical challenges and active research areas arise:

- **Theoretical explanation of optimal attention decay and positional encoding shape**: While slow logarithmic or generalized Gaussian priors empirically outperform sharp linear or bounded Gaussian decay, a unified theory for the relationship between positional decay, model expressivity, and context-awareness remains an open problem [2312.17044, 2505.22842].
- **Automatic and data-adaptive tuning of interpolation functions and kernel parameters**: Data-adaptive PEs such as DAPE suggest learnable, input-dependent correction is superior to fixed priors; future work may integrate continuous, neural, or probabilistic mechanisms into baseline PEs [2405.14722].
- **Certified extrapolation guarantees and spectral risk measurement**: Spectral or probabilistic condition numbers can be used to furnish certified error bounds in the extrapolation regime, providing a new axis for model selection and risk assessment beyond perplexity/accuracy [2603.09937].

Novel combinations—hierarchical encodings (BiPE), content-based adaptation (DAPE), plug-in scaling (PI, GALI), and probabilistic priors (BAM)—are now central tools for robust interrogation, evaluation, and scaling of positional encodings in both classical and modern machine learning models. The field continues to deepen the connections between analytic signal processing, theoretical learning guarantees, and scalable practical architectures for extrapolation in high-capacity sequence models.

Source: https://www.emergentmind.com/topics/positional-extrapolation-and-interpolation