PI-RoPE: Scalable Rotary Positional Encoding

• The paper demonstrates that PI-RoPE linearly interpolates positional indices to stabilize Transformer attention beyond training context lengths.
• PI-RoPE modifies the rotary mechanism by replacing the original position with a scaled index, mitigating phase aliasing and unstable attention scores.
• Empirical results show that PI-RoPE improves long-context performance, reducing perplexity and enhancing retrieval and summarization tasks.

Searching arXiv for the cited papers to ground the article and verify identifiers. Position-Interpolated Rotary Positional Encoding (PI-RoPE) is an inference-time method for extending the effective context length of Transformer models that use rotary positional embeddings (RoPE) by linearly compressing position indices before applying the rotary transformation. In the form summarized in "Position Interpolation Improves ALiBi Extrapolation" (Al-Khateeb et al., 2023), PI-RoPE keeps the model in a positional regime closer to that seen during training by replacing the original position index $p$ with an interpolated index $p' = p \cdot \frac{L}{L'}$ when the inference context length $L'$ exceeds the training context length $L$. More recent long-context work treats PI-RoPE as a family of RoPE scaling methods, including linear and frequency-aware variants, unified by modifying the effective phase trajectory without retraining (Qiao et al., 17 Sep 2025).

1. Conceptual origin and problem setting

RoPE encodes token position by rotating queries and keys in paired two-dimensional subspaces. For a query or key vector at position $p$, each 2D pair is rotated by an angle proportional to $p\theta_i$, where $\theta_i$ is the frequency assigned to that pair. A central property of this construction is that the dot product between a query at position $i$ and a key at position $j$ depends on the relative position $i-j$, which is why RoPE is widely treated as a relative positional mechanism (Al-Khateeb et al., 2023).

The operational difficulty addressed by PI-RoPE is not that RoPE is mathematically undefined beyond the training context, but that the trained model has only experienced positions up to a maximum length $p' = p \cdot \frac{L}{L'}$0. The supplied literature describes two linked failure modes outside that range. First, high-frequency components produce many phase cycles as $p' = p \cdot \frac{L}{L'}$1 grows, leading to aliasing or phase wrap-around. Second, extrapolation can induce unstable attention behavior; the background summary associated with (Al-Khateeb et al., 2023) states that Chen et al. reported that attention score magnitudes can “blow up” or otherwise behave badly when RoPE is applied beyond the trained regime. PI-RoPE emerged as a training-free response to that mismatch.

Within the supplied sources, PI-RoPE is presented as prior art rather than as the main contribution of (Al-Khateeb et al., 2023). That paper uses it as the prototype for a broader “compress distances at inference” principle, then adapts the same principle to ALiBi. This placement is historically important: PI-RoPE is treated not as an isolated trick, but as the canonical instance of position interpolation for long-context extrapolation.

2. Mathematical construction

In the standard RoPE formulation summarized in the supplied material, query and key vectors $p' = p \cdot \frac{L}{L'}$2 are partitioned into 2D pairs, and each pair is rotated by a position-dependent matrix. For a given frequency $p' = p \cdot \frac{L}{L'}$3, RoPE replaces a 2D pair of $p' = p \cdot \frac{L}{L'}$4 by

$p' = p \cdot \frac{L}{L'}$5

and analogously for $p' = p \cdot \frac{L}{L'}$6. The same rotation can be written in complex form as multiplication by $p' = p \cdot \frac{L}{L'}$7 (Al-Khateeb et al., 2023).

PI-RoPE modifies only the positional argument. Let $p' = p \cdot \frac{L}{L'}$8 be the maximum context length during training and $p' = p \cdot \frac{L}{L'}$9 the context length used at inference. For target position $L'$0, PI-RoPE defines

$L'$1

The model then uses $L'$2 instead of $L'$3 in the rotary angle, so the phase becomes

$L'$4

This globally compresses the interval $L'$5 into $L'$6, so the model behaves as if it were operating in a stretched coordinate system whose internal position grows more slowly than the actual token index (Al-Khateeb et al., 2023).

The supplied material emphasizes several practical properties of this construction. The interpolation is global and linear. The interpolated index $L'$7 is typically real-valued rather than integer-valued, and is passed directly into $L'$8 and $L'$9. The scaling is applied only when $L$0; for $L$1, the original position $L$2 is retained to preserve training-time behavior. The only effective hyperparameter is therefore the scale factor $L$3, which is fixed by the training and target context lengths.

3. Relation to ALiBi and the general position-interpolation principle

The importance of (Al-Khateeb et al., 2023) for PI-RoPE lies partly in how it generalizes the same idea to a different positional mechanism. ALiBi adds a linear positional bias to attention scores, using a head-specific slope $L$4. In that paper, the extrapolation analogue of PI-RoPE is obtained by scaling the slope during inference: $L$5 so that the bias $L$6 becomes $L$7. The authors explicitly interpret this as the ALiBi counterpart of RoPE position interpolation, because it compresses effective positional distances into the training range (Al-Khateeb et al., 2023).

The comparison is instructive because the reported failure modes differ. For RoPE, the supplied summary states that interpolation is motivated by unstable or inflated attention behavior during extrapolation. For ALiBi, (Al-Khateeb et al., 2023) reports the opposite qualitative issue: “Contrary to RoPE, we observe ALiBi introduces lower magnitude attention scores for tokens in the extrapolation regime than are seen in the interpolation regime.” The common remedy is still distance compression by $L$8. This suggests that PI-RoPE is best understood as one instance of a broader positional rescaling doctrine rather than as a RoPE-specific anomaly.

The same paper also reports empirical patterns that are repeatedly used in later discussions of PI-RoPE. For 8K-trained ALiBi models, baseline perplexity remains low only to about 9K–10K tokens, whereas ALiBi with interpolation maintains low perplexity up to 16K. For 2K-trained models, baseline extrapolation reaches roughly 2.5K–3K before perplexity blows up, while interpolation extends it to about 4K. On document summarization with BTLM-3B-8K at 16K context, the QMSum ROUGE scores increase from $L$9 to $p$0, and long-range retrieval also improves substantially (Al-Khateeb et al., 2023). Although these results are not RoPE experiments, the paper presents them as evidence for the general mechanism behind position interpolation.

4. Spectral and mechanistic interpretations

Several later analyses in the supplied literature recast standard RoPE in a spectral language that clarifies why PI-RoPE works. "Beyond Position: the emergence of wavelet-like properties in Transformers" writes the RoPE-modulated attention contribution of frequency $p$1 as a cosine of the relative offset,

$p$2

and identifies three regimes: near, where the cosine is close to $p$3; intermediate, where it crosses $p$4; and far, where it becomes negative and eventually oscillatory. That paper explicitly states that a solution is to bound $p$5 so that cosine remains in a decaying regime, which is exactly what PI-RoPE achieves by compressing effective distances (Ruscio et al., 2024).

The same source describes RoPE’s memory function as a Fourier-like sum

$p$6

so interpolation can be viewed as reparameterizing the lag variable in a learned superposition of cosines. This makes PI-RoPE more than a geometric rescaling: it is also a modification of the model’s learned memory kernel. A plausible implication is that successful interpolation should preserve the shape of that kernel over the lag range most heavily used during training.

"Rotary Outliers and Rotary Offset Features in LLMs" adds a complementary frequency-space picture. It defines low-frequency “rotary offset features” whose periods exceed the trained context length and derives an upper bound for such features,

$p$7

together with a lower bound on the mean query–key angle required for offset behavior,

$p$8

The paper reports that large RoPE outliers cluster in these low-frequency, partial-cycle bands across Phi-1, Llama-2-7B, and DeepSeek-V2-Lite (Jonasson, 3 Mar 2025). For PI-RoPE, this means that interpolation changes not only ordinary relative phases but also the geometry of the low-frequency channels that enforce long-range suppression and attention sinks.

A third line of interpretation comes from "Unpacking Positional Encoding in Transformers: A Spectral Analysis of Content-Position Coupling", which formalizes RoPE as multiplicative content–position coupling: the logit matrix is a Hadamard product of a content-related Gram matrix and a relative-position Toeplitz signal. That paper argues that the Hadamard modulation induces spectral contraction and associates RoPE with localized “single-head deposit” patterns in early layers (Gu et al., 19 May 2025). In that framework, PI-RoPE is best seen as preserving the multiplicative Toeplitz structure while changing the effective phase law. This suggests that interpolation methods should maintain relative-position Toeplitz behavior as closely as possible if they are to preserve RoPE’s optimization and specialization properties.

5. Critiques, variants, and alternative generalizations

The supplied literature also contains several critiques of RoPE that bear directly on PI-RoPE. "PoPE: Legendre Orthogonal Polynomials Based Position Encoding for LLMs" argues that sinusoidal bases become highly correlated in high dimensions and that this limitation extends beyond additive sinusoidal embeddings to RoPE, which, in the authors’ view, remains constrained by the same underlying basis even if it avoids an explicit additive bias term (Aggarwal, 2024). Within that perspective, PI-RoPE may improve extrapolation while leaving the representational redundancy of the sinusoidal basis intact.

A different critique appears in "Decoupling the 'What' and 'Where' With Polar Coordinate Positional Embeddings", which derives the RoPE attention term in polar form as

$p$9

That paper argues that RoPE entangles content and position through the content-dependent phase difference $p\theta_i$0, and introduces a different PoPE in which magnitudes encode content and phases encode position only. It reports that this alternative outperforms RoPE on an indirect indexing diagnostic, language modeling, music, genomics, and zero-shot extrapolation, and that its gains persist against YaRN despite YaRN’s use of additional fine-tuning and frequency interpolation (Gopalakrishnan et al., 5 Sep 2025). For PI-RoPE, the implication is that frequency interpolation alone may not resolve deeper structural issues in the RoPE score function.

The wavelet line of work extends this critique in another direction. "Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding" shows that the RoPE phase factor reappears exactly as the phase of a more general Morlet positional encoding, while the Morlet construction adds a Gaussian locality envelope. In that formulation the effective positional kernel factorizes approximately as

$p\theta_i$1

so RoPE is the $p\theta_i$2 limit in which amplitude locality disappears (Zeris, 31 May 2026). That paper explicitly suggests that PI-RoPE-style schemes might eventually interpolate not only phase but also locality bandwidth.

RoPE’s geometric generality is illustrated by "Spherical Position Encoding for Transformers", which replaces 2D SO(2) blocks with 3D SO(3) rotation blocks parameterized by latitude and longitude for “geotokens” (Unlu, 2023). Although that work does not address long-context extrapolation, it shows that the rotary principle extends to continuous, non-sequential coordinate systems. This broadens the conceptual meaning of PI-RoPE: interpolation need not be restricted to one-dimensional token indices, but can be understood more generally as smooth rescaling within a chosen positional geometry.

6. Deployment, quantization, and current limitations

The most explicit engineering treatment of PI-RoPE in the supplied corpus is "Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs" (Qiao et al., 17 Sep 2025). That paper formalizes a broad class of PI-RoPE methods by writing the scaled phase as

$p\theta_i$3

where $p\theta_i$4 is a position-warping function and $p\theta_i$5 is a per-dimension or per-band scaling factor. In that formulation, linear interpolation, frequency-aware scaling, YaRN, and NTK-aware long-context variants become instances of a common phase-remapping template.

Q-ROAR argues that combining PI-RoPE with post-training quantization degrades performance because of four coupled effects: long-context aliasing, dynamic range dilation, axis grid anisotropy, and outlier shifting, which together induce position-dependent logit noise. It introduces two diagnostics. The first is Interpolation Pressure,

$p\theta_i$6

which measures phase sensitivity to scaling. The second is the Tail Inflation Ratio, defined for both pre-activations and rotated activations to quantify the growth of extreme-value tails from short to PI-long contexts (Qiao et al., 17 Sep 2025). These definitions make explicit that PI-RoPE is not merely a geometric transformation; in quantized systems it is also a source of frequency-dependent numerical stress.

To mitigate those effects, Q-ROAR groups RoPE dimensions into 6 or 8 log-spaced frequency bands and rescales $p\theta_i$7 and $p\theta_i$8 per band, optionally in a symmetric mode that approximately preserves the logit scale. The paper reports that this weight-only stabilization recovers up to $p\theta_i$9 accuracy on standard tasks and reduces GovReport perplexity by more than $\theta_i$0, while preserving short-context performance and requiring no fine-tuning, kernel change, or architecture change (Qiao et al., 17 Sep 2025). Those results indicate that PI-RoPE remains practical for deployment, but that its interaction with quantization is sufficiently structured to require RoPE-aware compensation.

Across the supplied sources, the principal limitations of PI-RoPE are consistent. Interpolation eventually degrades at sufficiently large context multiples. Compression of positions can blur long-range distinctions even when it prevents catastrophic phase behavior. It does not by itself address critiques of the sinusoidal basis, content–position entanglement, or the concentration of positional processing into a small subset of heads and frequencies. The literature therefore portrays PI-RoPE as a strong baseline and a durable design principle for context extension, but not as the final form of long-context positional encoding.