Relative Positional Encoding Overview

• Relative positional encoding is a technique that encodes token relationships using offset functions rather than fixed indices, enabling invariance to input shifts and refined dependency modeling.
• It modulates the attention mechanism by incorporating learnable or deterministic transformations of query–key offsets, which decouple content and position for enhanced training stability.
• Applications of RPE span language, speech, vision, and graph domains, offering efficient computation and improved handling of both local and global dependencies.

Relative positional encoding (RPE) is a class of techniques for injecting sequence or relational structure into transformer models by encoding relationships between pairs of positions (or tokens, nodes, or elements) rather than their absolute indices. RPE methods decouple content and position, enabling greater invariance to input shifts, enhanced generalization to unseen lengths, and improved modeling of local and global dependencies. Unlike absolute positional encoding (APE), which conditions only on the index of each individual token, RPE modulates the attention mechanism using functions of the query–key positional offset, usually in the form of learnable or deterministic transformations. In modern transformers and their variants—including for text, speech, vision, and graph learning—RPE has become foundational for robust sequence modeling and advanced reasoning.

1. Mathematical Formulation and Core Variants

The principal defining feature of relative positional encoding is its parameterization of a function $f(i, j)$ or $f(i-j)$ that depends on the positional relationship between two elements $i$ and $j$. In the multi-head self-attention mechanism, this manifests as an extra bias, transformation, or kernel applied to the attention score $A_{ij}$ between positions $i$ and $j$: $A_{ij} = \langle q_i, k_j \rangle + b_{i-j}$ where $b_{i-j}$ is typically a learned embedding or a fixed function of the offset $i-j$ (Qin et al., 2023, Likhomanenko et al., 2021, Pham et al., 2020).

Several canonical designs exist:

• Additive Bias RPE (Shaw et al., T5): A bias term $b_{i-j}$ is looked up or computed for each relative offset and directly added to the $QK^T$ pre-softmax matrix (Pham et al., 2020, Zhang et al., 2024).
• Multiplicative RPE (RoPE/Rotary): Queries and keys are transformed via position-dependent orthogonal or unitary matrices (planar rotations for each frequency bin), such that the attention score depends only on the relative offset $i-j$. RoPE’s advantage is its spectral contraction and stability properties (Gu et al., 19 May 2025, Qin et al., 2023, Veisi et al., 30 Jul 2025).
• Kernel or Convolutional RPE: RPE is parameterized as a shift-invariant kernel $f(i-j)$; attention can be efficiently implemented via Toeplitz structure and FFT for sub-quadratic complexity (Luo et al., 2021, Zubkov et al., 2022).
• Stochastic RPE (SPE): Relative kernels are recovered in expectation by coupling content vectors to random-feature expansions or Gaussian process cross-covariances (Liutkus et al., 2021).
• Graph RPE: For graphs, RPE is a function $U_G(i, j)$ of node pairs (e.g., shortest-path or resistance distance), used as a pairwise bias or gating in the attention (Park et al., 2022, Black et al., 2024).

2. Theoretical Foundations and Spectral Structure

Relative positional encodings, especially those inducing Toeplitz or circulant structures, provide translation invariance along the sequence and grant architectural bias to modulate dependencies by distance rather than index (Gu et al., 19 May 2025). Mathematically, the RPE bias $b_{i-j}$ forms a Toeplitz matrix, which enables efficient convolutions and admits spectral analysis. For example, in RoPE, the Hadamard product with a Toeplitz kernel modulates the Gram matrix of content similarities, achieving "spectral contraction" and provably enhancing optimization stability: $L_{i, j} = \langle q_i, k_j \rangle \odot R_{i-j}$ where $R_{i-j}$ is a Toeplitz kernel, typically sinusoidal or exponential (Gu et al., 19 May 2025). Spectral contraction tightens the condition number of the attention matrix, accelerating training dynamics and robust generalization on position-sensitive tasks.

LRPE (Linearized RPE) formalizes RPE as transformations $W_{t-s}$ (e.g., rotations, permutations) satisfying $W_{t-s} = W_s^\dagger W_t$ for unitary or orthogonal $W$; this secures compatibility with linear/time-efficient attention (Qin et al., 2023).

3. Applications and Domains

Text and Language Modeling: Relative PE is standard in transformers for autoregressive and bidirectional LMs, such as DeBERTa, T5-style bucketing, RoPE, KERPLE, and their extensions. RPE unlocks stable length extrapolation and stronger inductive biases for hierarchically structured text (e.g., dialogue, code) (Zhang et al., 2024, Gao, 2024, Angelotti, 2023).

Speech and Sequence Modeling: For speech, where absolute time indices are often uninformative, RPE improves robustness to variable segmentation, silent spans, and long-range dependencies; models trained with RPE achieve state-of-the-art performance in ASR and speech translation (Pham et al., 2020). In spiking transformers, RPEs constructed via Gray code or logarithmic distance enable position awareness compatible with binary architectures (Lv et al., 28 Jan 2025).

Vision and Multi-View Perception: RPE generalizes to 2D/3D data using domain-specific relations (e.g., 2D patch grids, camera SE(3) extrinsics, or projective frustum relationships in multi-view models). PRoPE and GTA encode both camera intrinsics and extrinsics at the attention level, enabling transformers to reason over global geometric relationships invariant to viewpoint (Li et al., 14 Jul 2025).

Graphs: RPE in graph transformers leverages node pairwise structure (shortest-path, resistance, spectral kernels). GRPE integrates node-topology and edge-type interactions into self-attention, achieving fine control over both local and global graph structure. Theoretical work demonstrates that, for graph transformers, RPE and APE are equivalent in representational power, with practical biases for choosing RPE when pairwise relations are fundamental (Park et al., 2022, Black et al., 2024).

4. Algorithmic Implementations and Computational Complexity

Traditional naive RPE implementation is quadratic in sequence length due to the formation of a full $L \times L$ bias matrix. Recent advances circumvent this:

• Toeplitz/FFT Acceleration: Where $b_{i-j}$ is parameterized for $|i-j|\in\{-L+1,...,L-1\}$ (e.g., FastRPB), matrix-vector products reduce to convolutions, computable in $O(L \log L)$ by FFT (Zubkov et al., 2022, Luo et al., 2021).
• Linearizable Transformations: Methods such as LRPE and PermuteFormer reparameterize RPE in terms of position-dependent transformations on queries and keys, enabling $O(L)$–$O(Ld^2)$ complexity, no explicit pairwise computation, and drop-in compatibility for kernelized/linear attention (Qin et al., 2023, Chen, 2021).
• Parameter Sharing and Multi-Kernel Biases: Multi-kernel RPEs (MEP) fuse several decay profiles (exponential, Gaussian, log-polynomial), smoothing the decline of attention weights for improved length extrapolation with zero or few new parameters and efficient computation (Gao, 2024).
• FlashAttention-Compatible RPE: Methods such as HyPE introduce relative biases via hyperbolic function expansion (e.g., $-\tau \sinh(\mu (j-i))$), encoded by concatenation in query/key projections, fully compatible with fused FusAttention/FlashAttention-2 and subquadratic memory (Angelotti, 2023).

5. Practical Performance and Empirical Results

Relative positional encoding routinely outperforms absolute variants across domains:

• Language Modeling: LRPE and PermuteFormer yield perplexity reductions of 1–2 PPL points over RoPE and kernelized APEs on tasks such as WikiText-103 and Books, and substantial accuracy gains on the Long-Range Arena benchmark (2–5% increase in absolute performance) (Qin et al., 2023, Chen, 2021).
• Speech Enhancement and ASR: In noncausal speech enhancement, RPE methods such as T5-RPE and KERPLE deliver consistent improvements (e.g., up to +0.07 CSIG and +3.4% ESTOI over absolute PE) (Zhang et al., 2024). For end-to-end ASR and speech translation, RPE models set benchmarks for WER and BLEU on standard datasets (Pham et al., 2020).
• Vision and Multi-view Reasoning: PRoPE shows gains in PSNR, LPIPS, and SSIM for feed-forward novel view synthesis, outperforming both absolute raymap and SE(3)-only RPE; scaling studies confirm its robustness for variable camera configurations and large models (Li et al., 14 Jul 2025).
• Graph Learning: GRPE with node-topology and node-edge bias integration achieves lower MAE and higher classification accuracy than bias-only or linearization approaches on ZINC, MolHIV, PATTERN, and CLUSTER; depth-structured attention mappings are observed (Park et al., 2022).
• Long Context Extrapolation: MEP consistently outperforms ALiBi and Kerple in length extrapolation (perplexity at $L=8192$ drops from 2.360 to 2.318 on code tasks) while incurring negligible cost (Gao, 2024).
• Training Efficiency: PoPE demonstrates 2–3× accelerated convergence and boosts BLEU score by ~4–5 points over sinusoidal PE in EN-DE translation (BLEU 40.7 vs 35.6), attributed to improved high-dimensional decorrelation (Aggarwal, 2024).

6. Extensions, Limitations, and Theoretical Equivalence

RPE methods extend naturally beyond standard sequences:

• Graphs: Any RPE (as a function $U_G(i, j)$) and APE (as node labels) are mutually convertible in expressive power, via universal function approximation (DeepSets, equivariant GNNs) (Black et al., 2024).
• Multi-Dimensional and Heterogeneous Data: CAPE introduces augmentation-induced, coordinate-free RPE for continuous data (vision, speech), generalizing to arbitrary-length or resolution (Likhomanenko et al., 2021).
• Spiking Neural Networks: RPEs that preserve binary compatibility and time-translation invariance (Gray-PE, Log-PE) are critical in spiking transformer models (Lv et al., 28 Jan 2025).

Chief limitations include: increased computational and memory cost for naive $O(L^2)$ formulations; potential instability in special-purpose random-feature or kernel approximations unless properly regularized; and the challenge of designing generalizable kernel functions for non-Euclidean or complex domains.

7. Future Directions and Design Principles

Emerging lines of research highlight:

• Orthogonal Bases and Recurrence: PoPE and related work advocate using orthogonal polynomial bases with internal recurrence (e.g., Legendre, Chebyshev) to preserve high-dimensional decorrelation, robust relative embedding, and efficient computation (Aggarwal, 2024).
• Spectral and Toeplitz Analysis: Effective RPE leverages explicit multiplicative coupling and Toeplitz kernels to modulate content–position interactions, with spectral contraction as a core guiding metric (Gu et al., 19 May 2025).
• Parameter Efficiency: Consolidation of parameter-free and parameterized RPE, e.g., via multi-kernel weighting, minimal per-head hyperparameters, and architectural compatibility (FlashAttention, linear kernels) (Gao, 2024, Angelotti, 2023).
• Context Sensitivity: Dynamic, input-aware RPEs (e.g., CARoPE) adapt rotary frequencies per head and token, yielding substantial improvements in long-context perplexity and throughput over static RoPE (Veisi et al., 30 Jul 2025).
• Generalization to New Modalities: Modular RPE definitions support cross-modal, cross-sequence, and graph transformation by redefining the relational function per application, with guidelines for diagonal- and combinatorial-awareness (Black et al., 2024, Park et al., 2022).
• Hybridization: Future architectures may combine orthogonal-polynomial-based absolute encoding with lightweight explicit or kernelized RPE for further improvements in length extrapolation, generalization, and learning speed (Aggarwal, 2024, Gao, 2024).

Relative positional encoding now constitutes a central paradigm for structure-aware, length-robust, and generalizable transformer models across a wide spectrum of data modalities and computational regimes.