Relative Positional Encoding (RPE)

• Relative Positional Encoding (RPE) defines positions by pairwise relations rather than fixed absolute indices, enabling context-sensitive attention.
• Different implementations, including additive biases and rotary phase differences, tailor RPE to sequence, graph, and image domains.
• Empirical studies indicate that RPE improves model performance on long-context and structure-dependent tasks, with benefits varying by architecture.

Relative Positional Encoding (RPE) is a family of positional mechanisms in which positional information is represented primarily through relations between positions rather than by assigning each token or node an independent absolute index embedding. In the sequence setting, RPE is typically injected into self-attention so that the interaction between positions $i$ and $j$ depends on their offset or other pairwise geometry; in the graph setting, it is formalized as a feature on ordered node pairs; and in rotary formulations it is realized through relative phase differences in transformed queries and keys (Black et al., 2024, Veisi et al., 30 Jul 2025). Across recent work, the term has expanded from 1D token offsets to 2D image displacements, graph distances, camera relations, and other pairwise geometric structures, while retaining the core idea that attention should depend on relative configuration rather than only on content or absolute coordinates.

1. Definition and architectural role

The basic contrast between absolute and relative positional encoding is consistent across domains. Absolute positional encoding (APE) assigns a feature to each individual position, whereas RPE assigns a feature to each ordered pair of positions. In graph transformers this distinction is stated explicitly as

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$

with $\phi_G$ an APE and $\psi_G$ an RPE (Black et al., 2024). In sequence models, the same distinction appears operationally: APE is added to token embeddings before the transformer stack, while RPE modifies the pairwise attention score inside self-attention rather than the input representation itself (Zhang et al., 2024).

This architectural placement matters. In the speech-enhancement study, APE methods are applied at the input-embedding level as $Z' = Z + P$, whereas RPE methods alter the pre-softmax attention scores through a head-specific relative-position bias $P_h \in \mathbb{R}^{L\times L}$ (Zhang et al., 2024). In graph transformers, pairwise RPE enters both additively and multiplicatively in attention,

$$A(X)= f_1(V_G)\odot \mathrm{softmax}\!\left(\frac{XW^Q(XW^K)^\top}{\sqrt{d_h}} + f_2(V_G)\right),$$

so the positional signal directly modulates who attends to whom (Black et al., 2024).

A stricter distinction appears in the disentangled encoder of "Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders" (Lequeu et al., 28 May 2026). There, the authors separate semantic, absolute positional (AP), and relative positional (RP) streams, and define the RP component as a learned bucketed relative position bias added to attention logits rather than a vector embedding added to token states. In that formulation, AP is persistent token-wise hidden-state information, whereas RP is transient, attention-level information. This sharp separation has become important in later discussion because it isolates a recurrent property of many RPE schemes: they control attention effectively, but do not necessarily leave a persistent positional trace in hidden states.

2. Principal mathematical formulations

One major RPE family is the additive relative-bias formulation. In T5-style schemes and closely related variants, the attention logit is augmented by a learned scalar indexed by relative displacement. The kernelized-attention study writes this as

$$\alpha^{rel}_{ij}=\frac{1}{\sqrt{d}} (x_iW^{Q})(x_jW^{K})^{\top} + b_{j-i},$$

where $b_{j-i}$ depends only on the relative offset; the induced bias matrix $j$0 is Toeplitz because $j$1 (Luo et al., 2021). The disentangled encoder uses a bucketed learned bias $j$2, with special handling for $j$3 and $j$4, and adds it to semantic and AP dot products before softmax (Lequeu et al., 28 May 2026). Other additive RPEs replace bucketed tables with continuous functions of distance; the speech-enhancement paper’s KERPLE bias,

$j$5

is a representative monotone-decay formulation (Zhang et al., 2024).

A second major family is rotary or multiplicative relative encoding. Standard RoPE is written in phase form as

$j$6

so that attention after rotation depends on relative phase differences (Veisi et al., 30 Jul 2025). CARoPE generalizes this by replacing the fixed frequency schedule with token- and head-dependent frequencies,

$j$7

thereby keeping the rotary mechanism while making the frequency spectrum context-aware (Veisi et al., 30 Jul 2025). 3D-RPE adds another relative axis by separating within-chunk and between-chunk position: $j$8 so the score depends on both $j$9 and $$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$0 (Ma et al., 2024). Jordan-RoPE pushes the rotary view further by using a defective complex Jordan block and obtaining primitive lag functions such as

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$1

which couple phase and polynomial distance modulation in a single relative operator (Zhang, 5 May 2026).

A third line of work adapts RPE to efficient or linear attention. LRPE states the compatibility condition for linear transformers as

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$2

so that the relative interaction factorizes into separate position-dependent transforms on queries and keys (Qin et al., 2023). SPE recasts relative position as the cross-covariance of correlated stochastic processes, with

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$3

thereby approximating RPE inside linear-complexity attention without explicitly materializing the attention matrix (Liutkus et al., 2021). FLT instead learns the Fourier transform $$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$4 of the relative-position function $$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$5 and uses random Fourier features so that the relative mask is approximated by a low-rank product $$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$6, allowing Performer-style linear attention to incorporate general RPEs, including higher-dimensional geometric ones (Choromanski et al., 2023).

3. Representation, mechanism, and what RPE stores

A recurring question in recent work is whether RPE merely steers attention or whether it also serves as a durable structural memory. The disentangled encoder study offers a direct mechanistic answer: its RP stream is purely a learned relative bias over attention logits, is never injected into values, and does not appear as a persistent token representation after the layer (Lequeu et al., 28 May 2026). Their head-ablation analysis reports that there are no purely RP-oriented heads; instead, “the RP bias acts as a localized support mechanism for the semantic cluster” (Lequeu et al., 28 May 2026). In the same study, RoPE and RP are described as weak at preserving macroscopic structure in hidden states, while entangled AP can encode structure but loses much of it in the final layer under masked-language-modeling pressure (Lequeu et al., 28 May 2026).

This emphasis on transient control rather than persistent storage is echoed, from a different angle, by DAPE. That paper criticizes both APE and RPE for remaining fixed after training and proposes an adaptive extension in which the pre-softmax attention logits become

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$7

or in residual form,

$$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$8

with $$\phi_G : V_G \to \mathbb{R}^\ell,\qquad \psi_G : V_G \times V_G \to \mathbb{R}^k,$$9 implemented as a two-layer LeakyReLU MLP (Zheng et al., 2024). The core claim is that a fixed RPE may be “optimal on average” over training data but suboptimal for a particular instance, especially beyond the training length. This suggests that part of what is often attributed to “positional encoding” is actually attention control conditioned on content.

An additional perspective comes from AMR-to-text generation. There, standard T5/Shaw-style RPE substantially improves generation even when graph-aware structural adapters are present, and under graph corruption “using RPE with corrupted graph leads to similar performance than using graph information without RPE” (Montella et al., 2023). Link-prediction probes and attention-to-adjacency comparisons are interpreted as evidence that RPE may be partially encoding input-graph structure after AMR linearization (Montella et al., 2023). Taken together, these results do not imply that RPE is a complete graph or document-structure memory. They do indicate that RPE can carry structure-relevant information even when its formal implementation is only a bias in attention.

4. Domain-specific generalizations

In vision, RPE has evolved from 1D sequence offsets to explicitly 2D formulations. "Rethinking and Improving Relative Position Encoding for Vision Transformer" develops image RPE (iRPE), with 2D relative mappings based on Euclidean, Quantization, Cross, and Product encodings, and distinguishes bias mode from contextual mode in attention (Wu et al., 2021). Its central finding is that directed 2D relations and contextual query-relative interactions matter in images: relative encoding can replace absolute encoding for image classification, but absolute position remains necessary for object detection (Wu et al., 2021). A related development appears outside attention in PosMLP, where classical RPE formulations are repurposed to parameterize token-mixing relations in a vision MLP. There, LRPE reduces the $\phi_G$0 parameterization of the token-mixing matrix to $\phi_G$1, while GQPE reduces it to $\phi_G$2, and group-wise extensions provide multi-granular spatial context (Wang et al., 2022).

In graph and geometric settings, RPE becomes the natural way to encode structure. Graph transformers formalize it as a permutation-equivariant pairwise node encoding and prove that, under the paper’s framework, APE-based and RPE-based graph transformers are equivalent in distinguishing power (Black et al., 2024). The paper compares shortest path distance, resistance distance, heat kernels, adjacency- and Laplacian-based encodings, and spectral distances within that RPE framework (Black et al., 2024). In multiview vision, camera geometry is treated as a form of relative positional structure. PRoPE defines the relation between cameras $\phi_G$3 and $\phi_G$4 through the projective transform $\phi_G$5, incorporating both intrinsics and extrinsics as an attention-level RPE; when intrinsics are identity, it reduces to relative $\phi_G$6 transformations of prior camera-RPE methods (Li et al., 14 Jul 2025).

Speech and spiking systems have prompted additional variants. In direct speech recognition and translation, relative distance between acoustic states is injected into self-attention, replacing absolute positional terms with a distance-dependent embedding $\phi_G$7, and the resulting model is reported to adapt better to variable sentence segmentation quality and to synthetic data (Pham et al., 2020). In monaural speech enhancement, T5-RPE and KERPLE are compared with APE and No-Pos under causal and noncausal attention, showing that the impact of RPE depends strongly on whether causality already imposes an ordering bias (Zhang et al., 2024). In spiking transformers, Gray-PE and Log-PE approximate RPE while preserving the binary spike setting; Gray Code is used because its Hamming-distance behavior provides a structured approximation to relative distance, and Log-PE adds a logarithmic relative-distance matrix directly into the spiking attention map (Lv et al., 28 Jan 2025).

5. Empirical behavior across tasks

Empirically, RPE does not behave uniformly across architectures or domains. In the disentangled encoder study, standard benchmark performance remains close across positional schemes, but the disentangled model improves linguistic representation on 49 of the 65 phenomena of the Flash-Holmes probing benchmark (Lequeu et al., 28 May 2026). The same paper reports that RoPE and RP are weak at preserving macroscopic structure in hidden states, while the dedicated AP stream is the main structural store (Lequeu et al., 28 May 2026). This is a reminder that downstream parity on general benchmarks does not imply equivalent internal positional behavior.

On long-context language modeling, adaptive and context-aware RPE variants report large gains over static baselines. DAPE states that, when trained on sequence length 128 and evaluated at 8192 on Arxiv, DAPE-Kerple reaches perplexity 5.00 while vanilla Kerple is 31.93; on Books3 with training length 512, DAPE-Kerple reaches 17.88 at 8192 versus 39.31 for Kerple (Zheng et al., 2024). CARoPE reports that, in GPT-Small, perplexity at 1024 context is 21.39 for CARoPE versus 56.61 for RoPE, and in GPT-Tiny it is 36.74 versus 81.27 (Veisi et al., 30 Jul 2025). These results are presented as evidence that content-conditioned relative structure can improve longer-context behavior without abandoning the rotary framework.

Task-dependent effects are equally clear outside language modeling. In monaural speech enhancement, positional encoding is reported to be only marginally useful in causal configurations, but clearly beneficial in noncausal ones; among the tested encodings, RPE generally outperforms APE, and T5-RPE is usually slightly stronger than KERPLE (Zhang et al., 2024). In vision, iRPE yields up to 1.5% top-1 accuracy improvement on ImageNet and up to 1.3% mAP improvement on COCO for DeiT and DETR, without tuning extra hyperparameters such as learning rate and weight decay (Wu et al., 2021). In multiview vision, PRoPE improves over relative-pose baselines when intrinsics vary: on the intrinsics-augmented RealEstate10K setting, PRoPE reports 21.42 PSNR / 0.247 LPIPS / 0.678 SSIM, while GTA reports 15.77 / 0.512 / 0.641 (Li et al., 14 Jul 2025).

The evidence therefore supports neither a blanket superiority claim nor a blanket skepticism. Instead, it suggests that the effect of RPE depends on the inductive bias already present in the model, the geometry of the input domain, and whether the target task primarily needs local recency, long-range organization, or content-conditioned relational structure.

6. Theory, controversies, and research directions

Theoretical work has made clear that “having RPE” is not the same as having unrestricted positional expressivity. "Your Transformer May Not be as Powerful as You Expect" proves that standard RPE-based transformers with the relative term placed inside softmax are not universal approximators of continuous sequence-to-sequence functions on compact domains (Luo et al., 2022). The paper attributes the limitation to the fact that softmax attention produces a right-stochastic matrix, which constrains how positional information can survive the attention normalization. Its proposed remedy, URPE attention,

$\phi_G$8

adds a multiplicative Toeplitz matrix $\phi_G$9 and is proved to satisfy sufficient conditions for universal approximation (Luo et al., 2022).

A different controversy concerns whether the absolute-versus-relative distinction is itself fundamental. In graph transformers, the comparison paper proves that APE-GTs and RPE-GTs are equivalent in distinguishing power under the paper’s notion of indistinguishability, and gives constructive APE $\psi_G$0 RPE conversions (Black et al., 2024). This does not erase practical differences, because the same paper cautions that the conversions may require DeepSet or 2-EGN modules that are harder to optimize in practice (Black et al., 2024). A plausible implication is that theoretical equivalence does not settle architectural choice.

Another line of critique targets the basis functions used inside positional schemes. PoPE argues that sinusoidal basis functions exhibit high correlation in high dimensions and that this weakness may adversely affect some RPE methods, such as RoPE, because they still rely on sinusoidal factors (Aggarwal, 2024). The paper’s direct empirical evidence is centered on an absolute Legendre-polynomial encoding rather than a head-to-head modern RPE comparison, so its implications for RPE are largely inferential (Aggarwal, 2024). Even so, the argument has influenced later work that seeks richer phase bases, adaptive frequencies, or coupled distance-phase features.

Across these lines of work, several directions recur. One is explicit separation of persistent structural state from transient relative control, as in disentangled AP/RP/semantic streams (Lequeu et al., 28 May 2026). Another is context dependence: DAPE and CARoPE treat static RPE as a bottleneck and make the positional bias or phase generation depend on current content (Zheng et al., 2024, Veisi et al., 30 Jul 2025). A third is domain geometry: graph distance, camera projective relations, 2D image directionality, 3D coordinates, and spike-compatible Hamming structure all motivate specialized forms of “relative position” that extend far beyond 1D token lag (Li et al., 14 Jul 2025, Choromanski et al., 2023, Lv et al., 28 Jan 2025). This suggests that RPE is less a single mechanism than a design principle: pairwise structure should be represented in the same space in which attention compares elements.