---
title: Relative Position Representations
url: https://www.emergentmind.com/topics/relative-position-representations
type: topic
---

# Relative Position Representations

Relative position representations are methods for encoding the positional relationships between tokens, objects, or nodes in models based on self-attention or message-passing, such that pairwise or structural spatial information directly modulates the computation of model outputs. These representations allow models to be equivariant or sensitive to translational, sequential, or graph-theoretic relationships, conferring improved generalization, translation invariance, and structural awareness over purely absolute positional encodings. Relative positional encoding is foundational in advanced Transformers, graph neural networks, and spatial reasoning architectures across language, vision, and multimodal domains.

## 1. Foundations: Absolute vs. Relative Position Encodings

Classic Transformer architectures as introduced in Vaswani et al. (2017) encode positional information via absolute encodings—injecting fixed or learned vectors denoting sequence position into token embeddings. This mechanism breaks the permutation equivariance of attention but does not directly model pairwise distances. Relative position representations, in contrast, provide a mechanism by which model computations depend on (and are typically parametrized by) the distance or relational structure between elements, not just their absolute index [1803.02155].

Relative position encodings can be additive (e.g., biasing attention logits with learned vectors indexed by $j-i$) or more structurally embedded, as in rotary-encoded dot products, multiplicative gates, or deep polynomial interactions [2009.13658, 2406.10322]. These mechanisms have been shown empirically to improve generalization, especially to data outside the training domain, e.g., longer sequences, larger image patches, or greater spatial deformations [2109.02377, 2403.14852]. In graph transformers, relative position encodings are required to capture the underlying relational topology [2402.14202].

## 2. Canonical Relative Position Mechanisms

The primary mechanisms for relative position representations include:

- **Shaw-style additive RPE**: The seminal method of [1803.02155], which incorporates learned relative embeddings into the self-attention computation. For sequence positions $i, j$, a clipped signed offset $r = \text{clip}(j-i, -K, K)$ indexes a learned embedding $a_{ij}=w_r$. For each attention head, the score is:
  $$
  e_{ij} = \frac{q_i \cdot (k_j + a_{ij})}{\sqrt{d_z}}
  $$
  and the aggregated output incorporates an analogous value-side bias.

- **Rotary Positional Embedding (RoPE)**: Introduced to encode position via phase rotations in 2D planes within the embedding space. For token position $m$, the block-diagonal matrix $R(m)$ rotates key and query vectors such that the inner product $\langle R(m)q, R(n)k \rangle$ depends only on $m-n$. RoPE's extension to spherical and high-dimensional rotations enables encoding positions on spheres or volumetric grids [2310.04454, 2406.10322].

- **Advanced multiplicative/gated RPE**: Recent methods generalize beyond purely additive forms, introducing multiplicative gates, triple-products, or full three-way dot products between $(q_i, k_j, w_{j-i})$, enabling more expressive interactions [2009.13658].

- **Permutation-based and anchor-based methods**: In long-sequence or graph settings, relative positions may be encoded by compositional permutations or via learned/sampled anchor points whose shortest-path distances to nodes form relative position descriptors [2109.02377, 2105.11346].

- **Pairwise continuous or off-grid representations**: Some visual and geometric models replace grid-based encodings with representations derived from continuous relative translations, scales, or landmark offsets, as in PART for images and KP-RPE for vision transformers [2506.03682, 2403.14852].

## 3. Mathematical Formalism and Varieties

Relative position representations span several domains, with formal instantiations including:

- **Sequences**: Positions are indexed by integer offsets $j-i$, with learned or parameterized tables per offset up to a clipping distance $K$ [1803.02155, 2009.13658].
- **Grids and Spheres**: 2D/3D inputs use either discretized displacement vectors or parametric functions mapping $(\Delta x, \Delta y)$ or spherical angles $(\phi, \theta)$ to rotation or embedding matrices [2310.04454, 2406.10322].
- **Graphs**: Pairwise node relationships are represented by shortest path distance, resistance distance, or spectral kernels. RPEs here are functions $U_G: V \times V \rightarrow \mathbb{R}^k$ preserved under graph isomorphism [2402.14202]. PSGNN uses learned anchors with relative distances factored by a non-linear embedding [2105.11346].
- **Structural (syntactic/code) positions**: In code summarization and NLP, relative positions derive from structural relations in the underlying tree (e.g., AST or dependency tree), with edge embeddings parameterized by structural path length [2202.06521, 1909.00383].

These mechanisms are formalized in multiple mathematical forms:
- Additive: $e_{ij} = q_i^T k_j + q_i^T a_{ij}^K$;
- Multiplicative: $e_{ij} = (q_i \cdot k_j) \cdot a_{ij}$;
- Rotational: $q_i \rightarrow R(p_i)q_i; \quad k_j \rightarrow R(p_j)k_j; \quad e_{ij} = (q_i^T R_{ij} k_j)$;
- Full interaction: $e_{ij} = q_i^T k_j + q_i^T w_{j-i} + k_j^T w_{j-i}$ [2009.13658, 2310.04454, 2406.10322].

## 4. Applications in Language, Vision, and Graphs

Relative position representations are adopted in a range of domains:

- **Language Modeling and Translation**: RPEs yield consistent BLEU gains in translation benchmarks. Predicting pairwise relative positions also enables dense self-supervised objectives, providing label-rich pretraining [1803.02155, 2202.01145].
- **Vision Transformers**: In ViT architectures, relative and keypoint-anchored RPEs impart robustness to misalignments and geometric transformations, improving unaligned face identification and gait recognition [2403.14852]. Off-grid or continuous relative encodings improve spatial precision in detection and temporal modeling [2506.03682].
- **Graph Transformers and GNNs**: Relative structural embeddings derived from graph distances, resistance, or spectral transforms are essential for breaking node-permutation symmetry and delivering expressive, topology-aware representations [2402.14202, 2105.11346].
- **3D Geometric Reasoning**: In 3D vision tasks, relative position-aware attention across object pairs enables accurate localization and relational reasoning, as in 3DRP-Net for 3D visual grounding [2307.13363].

## 5. Theoretical Properties and Inductive Generalization

Relative position encodings confer advantageous inductive properties:

- **Translation invariance and extrapolation**: RPEs, by construction, produce attention patterns invariant to global shifts, unlike absolute encodings, and support longer or shifted contexts without new parameters [2109.02377, 2009.13658].
- **Expressiveness in graphs**: On graphs, the theoretical power of RPE-augmented models equals that of APE-augmented transformers under mild conditions, though combinatorially-aware RPEs (such as shortest-path distance) can strictly refine the Weisfeiler–Leman test, surpassing ordinary message passing in graph isomorphism distinguishing [2402.14202].
- **Capacity and generality**: Generalizations to high-dimensional and manifold-aware rotations (LieRE, spherical RoPE) enable direct preservation of geodesic distances and support large-scale, modality-agnostic applications [2310.04454, 2406.10322]. Full three-way or polynomial interactions expand the model’s representational capacity [2009.13658, 2208.10247].

## 6. Empirical Performance and Comparative Results

Relative position representations demonstrate consistent improvements:

- **NLP tasks**: On SQuAD1.1, method 4 (pairwise dot-product relative encoding) yields F1=90.53 over an absolute baseline of 88.59, also matching or exceeding performance on GLUE and machine translation [2009.13658, 1803.02155].
- **Vision and multimodal**: LieRE achieves top-1 accuracy 69.4%/68.8% on CIFAR-100/ImageNet, outperforming RoPE variants and absolute position baselines by 1.5+% [2406.10322]. KP-RPE improves face verification accuracy to 93.56% on CFPFP (vs. 72.81% for vanilla ViT) [2403.14852]. PART attains an improvement of 0.3–1.0 AP in COCO detection and >2% Cohen's κ in time-series classification over grid-based alternatives [2506.03682].
- **Graph and 3D grounding**: PSGNNs boost AUC by 10–20% in position-aware node/link tasks [2105.11346]. 3DRP-Net lifts 3D localization accuracy by 2.45–2.47 points relative to prior methods [2307.13363].

Performance generally correlates positively with the expressiveness of the relative encoding and the extent to which the downstream task rewards structural or pairwise awareness.

## 7. Extensions, Limitations, and Future Directions

Relative position encoding research continues to advance:

- **Geometric generalization**: Rotational encodings on Lie groups (LieRE), spherical and hyperbolic parametrizations, and keypoint-anchored variants extend RPEs beyond simple translational offsets to arbitrary manifolds and structured datasets [2310.04454, 2406.10322, 2403.14852].
- **Scalability and complexity**: Methods such as LieRE and PermuteFormer achieve O(N) scaling with respect to sequence or node count, enabling large-scale deployment without quadratic cost [2406.10322, 2109.02377].
- **Universal applicability**: Off-grid, pairwise, or conditional relative encodings (e.g., PART, KP-RPE) are being adapted to video, medical imaging, and non-visual time series [2506.03682, 2403.14852].
- **Expressivity vs. efficiency trade-offs**: Full pairwise parameterization increases model capacity at some computational cost; methods seek to balance expressiveness with head/parameter sharing and fast matrix algebra [2009.13658, 2208.10247].
- **Theoretical unification**: APEs and RPEs can be formally interconverted without loss of distinguishing power for finite graphs, so practical implementation often determines the optimal choice [2402.14202].

Limitations include increased memory/compute cost for fully pairwise encodings and challenges in encoding nonrigid, non-translational relationships outside traditional settings. Ongoing work focuses on expanding efficiency, applicability, and inductive robustness.

---

**References**:  
[1803.02155], [2009.13658], [2109.02377], [2310.04454], [2406.10322], [2403.14852], [2506.03682], [2105.11346], [2402.14202], [2202.01145], [2208.10247], [1909.00383], [2202.06521], [2307.13363]

Source: https://www.emergentmind.com/topics/relative-position-representations